:W&Bá¾eK^M rp6viianª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0-SŒ.Ä‚" 7yrns497z0r5 rp6viianŠ  rp6viianspatop" _wandb‚{}Bspatop_v6_blocks_outputjdp-spatop-train-v6-blocks-48nfkŠ ˜÷ÄÓØ‹×Àš1 (H¨¸h"3.12.10*0.28.1j linux-x86_64L €ÜcÒÝ -Linux-6.8.0-117-generic-x86_64-with-glibc2.36CPython 3.12.10 ˜÷ÄÓ€¢¹ü*-of*A/data/spatop/options_files/sweep_fixed_dp/config_gs8pex8v_v6.json*-l*/scratch/logs/*-n*spatop_v6_blocks_output2-m spanet.trainRdanielprimosch@gmail.comZ/scratch/logs/bdp-spatop-train-v6-blocks-48nfkr/opt/conda/bin/pythonº g6o11rbvwu2ki664mnd2u9dec6valbryjZi(ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ ŠàÖ&ª€ÀåÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-90988967-60ae-29dd-fe10-539d0dd12c25Ê13.2º g6o11rbvwu2ki664mnd2u9dec6valbryæPò2  requirements.txtØŠTP rp6viianZ? (H¨¸ (H¨¸h"3.12.10*0.28.1j linux-x86_64.8`R rp6viianZA (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64âluï* rp6viianb trainer/global_step2Ó U  rp6viianZD (H¨¸ (H¨¸8h"3.12.10*0.28.1j linux-x86_64bgþ/  rp6viianb*"trainer/global_step2ÔöˆK  rp6viianj: ™÷ÄÓ˜ë¹ó*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] .ïãA“  rp6viianj  ™÷ÄÓˆ©œ¥î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¾s,bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPwìVo-bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP†ækm.bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPѯw$n/bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÓ›lDl0bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP:žx¹l1bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP+ g2bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP?;òWp3bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP•–ðyp4bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPS{Xg5bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPh@Ãe6ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPSÍ#Dj7bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÅÀ­n8bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP¨ù v9br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP;bËpm:bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP‹7s;bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPŸ®nb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPØš°ðh?bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP žOe@ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP¦ù[÷kAbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPã'ÙrBbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP?]V^CbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPÔÆ=inDbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPøÓïlEbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP|ÕAˆ_Fb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPËwyjGbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPÊ_fhHbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPnc eoIbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP8îÑjJbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JP³hëìiKbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPhÛTqLbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP¹ædMb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPFìûLpNbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPðÜÁNqObm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPÐÿøxPbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP‚ZcwsQbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPXHÁ¦kRbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP^q×iSbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP€02hTbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP¶ÆãeUba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPï¸åÏoVbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP»Æ$Û^WbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP¥Ç ØpXbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP?ÐJ¯kYbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPêƒ7^gZbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPj¨4!g[bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPÒiÞ g\bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPg@Tm]bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPäÍ·]^bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPŸÖ a_b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPcžYn`bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPn=”eWabS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPºPücbb_ ntbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPŒ}Dßiube BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPkÅ’Áfvbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP| {awb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP²¡exba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP-i`)gybc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP=e˜Jbzb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPŽitk{bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP…±Lo|bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPÜ4r„v}br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPnS2`t~bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP½·"fbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPB#–s€bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPJXjfba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPU M@a‚b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPëÄV‰iƒbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPâE¾-r„bm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPV}q…bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPèÞšYu†bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP\Yãh‡bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPþDM¯qˆbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÞ°¶j‰be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP\ø‘sŠbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPL¸œj‹be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPôMkŒbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JPþ´_tbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP¡?’átŽbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPÝÌßØobj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPF-Heb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPÒÖw~o‘bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPi-+&k’bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JPb¢7g“bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPØw§wd”b_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11814 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 862863791104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 44920055808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚199247 O>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 906456278528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 906456278528 [JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 911253711360 % _timestamp‚1.7858057450103514e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 866180174848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 43679917056 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 43679917056 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 43571143052 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886704 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940454912 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚534307 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚524651 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 GDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚336727 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚199245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 841570332832 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚144329 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚861378 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250522112 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚144579 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11885 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250522112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 RADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 885038633400 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 841553293472 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚336729 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 862880830464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 866205078528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 43592487424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚49 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚871036 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚343574 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 23.305165811 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚336727 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚199245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 841570332832 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚144329 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚861378 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250522112 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚144579 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11885 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250522112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 RADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 885038633400 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 841553293472 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚336729 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 862880830464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 866205078528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 43592487424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚49 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚871036 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚343574 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 23.305165811 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11814 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 862863791104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 44920055808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚199247 O>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 906456278528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 906456278528 [JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 911253711360 % _timestamp‚1.7858057450103514e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 866180174848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 43679917056 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 43679917056 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 43571143052 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886704 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940454912 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚534307 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚524651 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 GDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP›ð5©^»bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP$æa‹m¼bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPXÙøf½ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP¶xr¾bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPíø’ i¿bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP€’£mkÀbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPuÕàUwÁbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPLx.^ÂbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP'Ä jÃbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP…))mÄbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPÉ´ÖîkÅbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPœÇÅ2dÆb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÈuéˆgËbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JP8ÑznÌbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPtôÄ–mÍbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP\^eÎb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPI˜dÏb_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÕ¬û÷eÚb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP„æÚÃkÛbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPFêBaÜb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPÁk/´_ÝbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPb<€ˆuÞbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP*ÄgVßbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP!‡ÒÁhàbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP—£1àoábj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP)”‰gâbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP'=÷dfãba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP>ëvzeäb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPŽ]8dåb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP3þ+éeïb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP0‰c"kðbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP¹.hñbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPÝËálòbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP§0fpóbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP]"5˜môbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPBó!Qqõbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPr6Ïaöb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPÞÖµod÷b_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPQÆ£}qübl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPÍ…é1sýbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPwúŒ^þbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPó¿>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP©l$òc‹b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPçeŒb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPí%#eb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPJÅbylŽbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP=qpbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPcŽí)]bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP\Ì"Oe‘b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPr2£l’bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP¹r›>.“) _wandbruntime‚23  _runtime‚23K1ÖT!”çL OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚2 Q@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 924607137280 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 880047593472 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 YHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 929364081664 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚888216 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 44472107008 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚888216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 TCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 924607137280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 P?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 924502661120 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚343370 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚ 23.65140467 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 880030554112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚544844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858057453594735e+09 MDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚350358 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚147431 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 45829607936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 44362813150 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚203184 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚203182 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚535174 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚514 UDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 929216899584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 858293465792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 902759121538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚878544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 880047593472 LADeviceStatsMonitor.on_train_batch_end/allæÙ{ocation.large_pool.freed‚343370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 45951886336 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 44472107008 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147182080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚343370 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚ 23.65140467 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 880030554112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚544844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858057453594735e+09 MDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚350358 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚147431 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 45829607936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 44362813150 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚203184 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚203182 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚535174 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚514 UDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 929216899584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 858293465792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 902759121538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚878544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 880047593472 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚343370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 45951886336 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 44472107008 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 147182080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚888216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 TCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 924607137280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392é/żA–b< loss/FRt1/assignment_loss"trainer/global_step2:JP..ÒdA—b< loss/FRt2/assignment_loss"trainer/global_step2:JPpaˤC˜b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPii{xA™b< loss/FBt2/assignment_loss"trainer/global_step2:JPþµf@šb; loss/FBt2/detection_loss"trainer/global_step2:JPPBêA›b< loss/FBt1/assignment_loss"trainer/global_step2:JPŒÌ K@œb; loss/FRt2/detection_loss"trainer/global_step2:JPÞ¼wÍBb= loss/SRqqt1/detection_loss"trainer/global_step2:JPÁ:¡ÝBžb= loss/SRqqt2/detection_loss"trainer/global_step2:JPz•ƒÆ7Ÿb2 loss/total_loss"trainer/global_step2:JP‰[x@ b; loss/FBt1/detection_loss"trainer/global_step2:JPœÎVC¡b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP[]ÉÌ@¢b; loss/FRt1/detection_loss"trainer/global_step2:JPhxoÿ-£b( epoch"trainer/global_step2:JP BΨ.¤) _wandbruntime‚23  _runtime‚23Lt¤ ÷¥ñ $ _timestamp‚1.785805745360828e+09 _step‚3 .loss/FRt1/assignment_loss‚5.68307165352695 /loss/FRt2/assignment_loss‚5.864719852805138 1loss/SRqqt2/assignment_loss‚4.486253484926726 /loss/FBt2/assignment_loss‚2.375892788171768 /loss/FBt2/detection_loss‚24.957605976611376 0loss/FBt1/assignment_loss‚2.3452710807323456 .loss/FRt2/detection_loss‚5.520205176901072 1loss/SRqqt1/detection_loss‚3.5922417598907077 1loss/SRqqt2/detection_loss‚3.4082127265240016 %loss/total_loss‚83.26175810159191 /loss/FBt1/detection_loss‚16.345535709212225 _runtime‚ 23.652298134 2loss/SRqqt1/assignment_loss‚4.5447008899439165 .loss/FRt1/detection_loss‚4.138047002345683 epoch‚0 trainer/global_step‚49¦¯û¦õ %loss/total_loss‚83.26175810159191 /loss/FBt1/detection_loss‚16.345535709212225 _runtime‚ 23.652298134 2loss/SRqqt1/assignment_loss‚4.5447008899439165 .loss/FRt1/detection_loss‚4.138047002345683 epoch‚0 trainer/global_step‚49 $ _timestamp‚1.785805745360828e+09 _step‚3 .loss/FRt1/assignment_loss‚5.68307165352695 /loss/FRt2/assignment_loss‚5.864719852805138 1loss/SRqqt2/assignment_loss‚4.486253484926726 /loss/FBt2/assignment_loss‚2.375892788171768 /loss/FBt2/detection_loss‚24.957605976611376 0loss/FBt1/assignment_loss‚2.3452710807323456 .loss/FRt2/detection_loss‚5.520205176901072 1loss/SRqqt1/detection_loss‚3.5922417598907077 1loss/SRqqt2/detection_loss‚3.40821272652400161âû¤œ§‚(:‘ ·÷ÄÓÜí•Ó gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.80666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ýR4¨‚(:© ·÷ÄÓ®¤ÜÖ disk.dm-1.in‚0 disk.rbd1.out‚0$ proc.memory.rssMB‚11267.71484375 network.recv‚47408 disk.nvme0c0n1.in‚0 disk.rbd5.in‚0 disk.rbd4.out‚0* proc.memory.availableMB‚38534.87890625 cpu‚14.378211472243397 proc.cpu.threads‚29) disk./.usagePercent‚83.06225583025414 disk.sda.out‚ 1.44140625 disk.drbd1743.out‚0.046875 disk.dm-0.out‚ 3.5380859375 network.sent‚55474 disk.sda2.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1237.6876373291016# disk.nvme0c0n1.out‚ 3.5380859375 disk.rbd2.out‚0 disk.rbd3.in‚0 disk.rbd5.out‚0 disk.rbd0.in‚0% memory_percent‚41.200441122055054 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 3.5380859375 disk.dm-1.out‚ 1.44140625 disk.rbd3.out‚0 disk.nvme0n1.in‚0 disk.rbd4.in‚0) proc.memory.percent‚17.19316840171814 disk.sda3.out‚ 1.44140625€<ó?.©) _wandbruntime‚43  _runtime‚43æ Íýª¼ _runtime‚ 43.70735077 _step‚4 ' lr-AdamW/pg1‚2.5514464620453036e-06 ' lr-AdamW/pg2‚2.5514464620453036e-06 trainer/global_step‚99 % _timestamp‚1.7858057654156904e+09Zµ gÆ«À _runtime‚ 43.70735077 _step‚4 ' lr-AdamW/pg1‚2.5514464620453036e-06 ' lr-AdamW/pg2‚2.5514464620453036e-06 trainer/global_step‚99 % _timestamp‚1.7858057654156904e+09?¿®.¬) _wandbruntime‚43  _runtime‚43ý13|'­öN XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚668879 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1721201941504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 87459606560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 % _timestamp‚1.7858057654166813e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1808790564352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1721218980864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚5 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚668879 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 87676052480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90116208640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 87676052480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1678578948832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1052653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚668877 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1808895033344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1816652223488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1765935712908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1678561909472 _runtime‚ 43.711959908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1721201941504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚683141 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚396095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1808790564352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1816811995648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1052653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1721218980864 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 87588622848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1731188 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚287303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 87373803436 HÆ~÷®úN WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 1808895033344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1062309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1766038555392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 90116208640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 87676052480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1678578948832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1052653 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚512 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚668877 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1808895033344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1816652223488 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1765935712908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1678561909472 _runtime‚ 43.711959908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1721201941504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚683141 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚396095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 1808790564352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1816811995648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1052653 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1721218980864 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 87588622848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1731188 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚287303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 87373803436 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 87459606560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 % _timestamp‚1.7858057654166813e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1808790564352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1721218980864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚5 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚668879 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 87676052480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1738696 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚689957 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚400032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚99 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚675520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 88251276658 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚690200 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 88555679232 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚675520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1695302081792 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1748368 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚400034 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1783553358450 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1826836946944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858057657623477e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 91151332352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1738385743872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1695319121152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1072846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚289925 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1743768035328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1783656201046 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚6 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1748368 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1063176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1738368704512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 88468242432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚290166 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 91018568192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1834761699840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1063176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1072846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1743743131648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 88337079894 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640[n»Š&±„M I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚689957 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚400032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚99 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚675520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 88251276658 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚690200 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 88555679232 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚675520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1695302081792 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1748368 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚400034 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1783553358450 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1826836946944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858057657623477e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 91151332352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1738385743872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1695319121152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1072846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚289925 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1743768035328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1783656201046 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚6 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1748368 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1063176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 1738368704512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 88468242432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚290166 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 91018568192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1834761699840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1063176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1072846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1743743131648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 88337079894 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚675522 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1738696 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚592947 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 131672187904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1590311 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2711229319168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2591340 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.785805785478432e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 131584758272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ICDeviceStatsMonitor.on_train_baÍØÚ—\tch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _step‚9 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2579557131264 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 131584758272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1023067 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2515570525472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1001027 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 131262266944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2579540091904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1001029 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2586916758528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1001029 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1580655 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 135338538496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 63.773969425 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2579540091904 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2711124850176 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2711124850176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1580655 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚592947 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 131672187904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1590311 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2711229319168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2591340 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.785805785478432e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 131584758272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _step‚9 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2579557131264 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 131584758272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1023067 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2515570525472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1001027 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 131262266944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 2579540091904 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1001029 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 2586916758528 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1001029 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1580655 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚522 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 135338538496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 63.773969425 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 2579540091904 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2711124850176 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2711124850176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1580655 HDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1007670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1007672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2664553280554 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2532327737152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2729275708928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2596723894272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2596706854912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1591178 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 132464377856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1029887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2729275708928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2596723894272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1591178 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2598848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚596882 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚522 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 136245420032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2608520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2532310697792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2596706854912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚596884 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2664450437958 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 132225543402 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚10 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚433261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680«H9H0Á“M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1007672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2664553280554 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2532327737152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2729275708928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2596723894272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2596706854912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1591178 NDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 132464377856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1029887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2729275708928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2596723894272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1591178 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2598848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚596882 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚522 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 136245420032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2608520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2532310697792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2596706854912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚596884 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2664450437958 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 MGDeviceStatsMonitor.on_train_batch_#±Ý×i end/inactive_split.large_pool.current‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 132225543402 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚10 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚433261 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2608520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1007670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ¢#½±.Â) _wandbruntime‚64  _runtime‚64Ût|ñÃë .loss/FRt2/detection_loss‚4.315545496421975 .loss/FBt1/detection_loss‚19.30980885028839 /loss/FBt2/detection_loss‚20.588312310607808 /loss/FRt2/assignment_loss‚5.708894680072735 /loss/SRqqt2/assignment_loss‚4.5153989570657 0loss/SRqqt2/detection_loss‚3.363598612626803 $loss/total_loss‚80.7202176309243 trainer/global_step‚149 /loss/FRt1/assignment_loss‚5.623162602123461 0loss/SRqqt1/detection_loss‚3.401163389001574 _runtime‚ 64.121183949 _step‚11 epoch‚0 1loss/SRqqt1/assignment_loss‚4.530628118060884 /loss/FBt1/assignment_loss‚2.321742558479309 /loss/FBt2/assignment_loss‚2.362058488946212 .loss/FRt1/detection_loss‚4.679903567229447 # _timestamp‚1.78580578582972e+09 ë}õÄï .loss/FRt1/detection_loss‚4.679903567229447 # _timestamp‚1.78580578582972e+09 .loss/FRt2/detection_loss‚4.315545496421975 .loss/FBt1/detection_loss‚19.30980885028839 /loss/FBt2/detection_loss‚20.588312310607808 /loss/FRt2/assignment_loss‚5.708894680072735 /loss/SRqqt2/assignment_loss‚4.5153989570657 0loss/SRqqt2/detection_loss‚3.363598612626803 $loss/total_loss‚80.7202176309243 trainer/global_step‚149 /loss/FRt1/assignment_loss‚5.623162602123461 0loss/SRqqt1/detection_loss‚3.401163389001574 _runtime‚ 64.121183949 _step‚11 epoch‚0 1loss/SRqqt1/assignment_loss‚4.530628118060884 /loss/FBt1/assignment_loss‚2.321742558479309 /loss/FBt2/assignment_loss‚2.362058488946212 ÝW×Å‚(:… ä÷ÄÓºìÙÒ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚154.332' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚85.74 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó£6Æ‚(:« ä÷ÄÓõÈÖ disk.sda.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 38534.3828125 cpu‚14.278839032589746 disk.sda1.in‚0 disk.dm-0.out‚ 4.43896484375 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd2.out‚0 disk.rbd1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 3.58203125 disk.dm-0.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚83.06226786260825 disk.drbd1743.in‚0" disk.nvme0n1.out‚ 4.43896484375 disk.rbd0.out‚0 disk.sda.out‚ 3.58203125 disk.rbd2.in‚0 disk.rbd1.in‚0# memory_percent‚41.2011981010437 disk.drbd1743.out‚ 0.10546875 disk.rbd5.out‚0$ disk.nvme0c0n1.out‚ 4.43896484375 disk.sda3.in‚0 disk.dm-1.out‚ 3.58203125 network.recv‚52984 disk.rbd3.in‚0 disk.nvme0n1.in‚0 disk.rbd5.in‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 11267.7265625* proc.memory.percent‚17.193186283111572 proc.cpu.threads‚29 network.sent‚147190$ disk./.usageGB‚1237.687816619873 disk.rbd4.out‚0ÂT„.Ç) _wandbruntime‚83  _runtime‚83b\ÛŽÁÈ» _runtime‚ 83.96834034 _step‚12 & lr-AdamW/pg1‚5.128665110575913e-06 & lr-AdamW/pg2‚5.128665110575913e-06 trainer/global_step‚199 $ _timestamp‚1.785805805676627e+09[‰›ÅÉ¿ & lr-AdamW/pg1‚5.128665110575913e-06 & lr-AdamW/pg2‚5.128665110575913e-06 trainer/global_step‚199 $ _timestamp‚1.785805805676627e+09 _runtime‚ 83.96834034 _step‚12 ÚÖïØ.Ê) _wandbruntime‚83  _runtime‚83Û?¹:‹'Ë…O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 175150730452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3613563604992 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108657 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3627730790912 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚526 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3437895281664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 175580893696 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108657 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚573582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3352579141472 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 175236533576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2118313 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3613459136000 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1333179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3613563604992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1333179 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3451492 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3352596180832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2118313 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚789795 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3627890563072 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3437895281664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 180617964544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 175668323328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1333177 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚573322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3447247694848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 83.972963455 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3437878242304í•é>'̉O YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3451492 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3352596180832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2118313 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚789795 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3627890563072 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3437895281664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 180617964544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 175668323328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1333177 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚573322 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3447247694848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 83.972963455 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3437878242304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 175150730452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3613563604992 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2108657 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3627730790912 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚526 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3437895281664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 175580893696 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2108657 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚573582 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3352579141472 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 175236533576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2118313 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3613459136000 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1333179 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3613563604992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1333179 {œ¥{.Í) _wandbruntime‚84  _runtime‚84±ÀöP1ÎM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3464479715328 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3468672 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3631609994752 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1339822 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1339820 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚793734 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚516 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3464454811648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚14 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2128850 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3369319313792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1339820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3545347517466 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3455062044672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3455045005312 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1370131 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 176114006910 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3645845946368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1339822 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 176460513280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576397 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1369888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2119180 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3645995225600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3369336353152 % _timestamp‚1.7858058060371034e+09 F:DeviceStatsMonitor.on_train_batch_eñéðend/allocation.all.freed‚3459000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3631505518592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 176547950080 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 176547950080 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2128850 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚793732 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚576156 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3545450360062 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3455062044672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 176460513280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚526 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 181515510272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 181391134720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 84.329085658 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3455045005312 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3468672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2119180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291yÆ¿–š&Ï”M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3455045005312 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3468672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2119180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3464479715328 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3468672 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3631609994752 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1339822 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1339820 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚793734 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚516 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3464454811648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚14 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2128850 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3369319313792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1339820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 3545347517466 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3455062044672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3455045005312 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1370131 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 176114006910 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3645845946368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1339822 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 176460513280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576397 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1369888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2119180 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3645995225600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3369336353152 % _timestamp‚1.7858058060371034e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3459000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3631505518592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 176547950080 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 176547950080 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2128850 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚793732 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚576156 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3545450360062 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 3455062044672 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 176460513280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚526 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 181515510272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 181391134720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 84.329085658-íÒ.Ð) _wandbruntime‚84  _runtime‚84UÈyËúÑô /loss/FRt2/assignment_loss‚5.797928887744282 0loss/FBt1/assignment_loss‚2.3030858834584556 epoch‚0 .loss/FRt1/assignment_loss‚5.70598755067992 1loss/SRqqt1/assignment_loss‚4.564329117536545 0loss/FBt2/assignment_loss‚2.3109893004099527 .loss/FRt2/detection_loss‚3.870992266161497 1loss/SRqqt2/detection_loss‚3.2696716041893374 /loss/FBt2/detection_loss‚18.447128876334144 %loss/total_loss‚78.42841948030147 trainer/global_step‚199 /loss/FRt1/detection_loss‚3.3116033505467537 _runtime‚ 84.329767035 1loss/SRqqt2/assignment_loss‚4.505873548741243 1loss/SRqqt1/detection_loss‚2.8897849174837273 .loss/FBt1/detection_loss‚21.45104417701562 % _timestamp‚1.7858058060382748e+09 _step‚15Äd¶/þÒø % _timestamp‚1.7858058060382748e+09 _step‚15 /loss/FRt2/assignment_loss‚5.797928887744282 0loss/FBt1/assignment_loss‚2.3030858834584556 epoch‚0 .loss/FRt1/assignment_loss‚5.70598755067992 1loss/SRqqt1/assignment_loss‚4.564329117536545 0loss/FBt2/assignment_loss‚2.3109893004099527 .loss/FRt2/detection_loss‚3.870992266161497 1loss/SRqqt2/detection_loss‚3.2696716041893374 /loss/FBt2/detection_loss‚18.447128876334144 %loss/total_loss‚78.42841948030147 trainer/global_step‚199 /loss/FRt1/detection_loss‚3.3116033505467537 _runtime‚ 84.329767035 1loss/SRqqt2/assignment_loss‚4.505873548741243 1loss/SRqqt1/detection_loss‚2.8897849174837273 .loss/FBt1/detection_loss‚21.45104417701562iK&lœÓ‚(:‘ ó÷ÄÓÕë€Ó gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.80666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÁ8ù%Ô‚(:š ó÷ÄÓ¨ÖÜÖ disk.nvme0c0n1.in‚0 disk.sda.out‚ 4.2109375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚0.125 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 11267.7265625 cpu‚14.257882299293653 disk.dm-0.out‚ 5.0234375 disk.rbd3.in‚0 disk.nvme0n1.out‚ 5.0234375 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.out‚ 5.0234375* proc.memory.percent‚17.193186283111572 disk.sda1.out‚0 disk.dm-0.in‚0 disk.sda1.in‚0 disk.rbd4.out‚0 disk.rbd2.out‚0 disk.rbd1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚29 network.sent‚191916 network.recv‚55393 disk.rbd5.out‚0 disk.sda3.out‚ 4.2109375 disk.dm-1.out‚ 4.2109375* proc.memory.availableMB‚38534.02734375% disk./.usageGB‚1237.6878395080566 disk.nvme0n1.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚41.20174050331116) disk./.usagePercent‚83.06226939865346 disk.rbd5.in‚0YËÓà0Õ+ _wandbruntime‚104  _runtime‚104UѨ+ÃÖ½ & lr-AdamW/pg1‚6.417274434841219e-06 & lr-AdamW/pg2‚6.417274434841219e-06 trainer/global_step‚249 % _timestamp‚1.7858058259823494e+09 _runtime‚ 104.27593958 _step‚16»zºÇ×Á & lr-AdamW/pg1‚6.417274434841219e-06 & lr-AdamW/pg2‚6.417274434841219e-06 trainer/global_step‚249 % _timestamp‚1.7858058259823494e+09 _runtime‚ 104.27593958 _step‚16C¼ 0Ø+ _wandbruntime‚104  _runtime‚104› Eå'Ù‡O K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2636659 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4515897890816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4307603534848 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 225762959360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2646315 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1665329 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4311644 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 219577029120 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4515897890816 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1665329 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1703510 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 219039193960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4301986 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚986647 _step‚17 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4515793421824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4515793421824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4296233432064 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533235820032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚517 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7858058259830317e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1665327 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 4189587757472 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚716863 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 4189604796832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4296233432064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4296216392704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4408729793916 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚716623 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚986645 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2646315 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4533391397888 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4408626951432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4296216392704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚527 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚249 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2636659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4307628438528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4515793421824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4515793421824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4296233432064 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4533235820032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚517 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7858058259830317e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1665327 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 4189587757472 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚716863 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 4189604796832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4296233432064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4296216392704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4408729793916 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚716623 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚986645 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2646315 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4533391397888 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4408626951432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4296216392704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚527 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚249 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2636659 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4307628438528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚986647 _step‚17ì¡ž±0Û+ _wandbruntime‚104  _runtime‚104m°ÆŠ—&Ü‘M WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1671970 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 220002470418 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1671970 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2647182 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚990584 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4533839804416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 226530671616 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1671972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4319152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4551501088256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4324835555328 _step‚18 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 220456648704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4426347439570 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4426244596974 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 226665532928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4313383155712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1710292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710047 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 220544085504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4206327929792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4206344969152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1671972 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4328824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 220456648704 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 219916667182 % _timestamp‚1.7858058263315775e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2647182 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚719708 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚719465 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚517 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4313383155712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4313400195072 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2656852 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 104.623483358 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4319152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2656852 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 220544085504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4313400195072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4328824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4533944280576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4533839804416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 226530671616 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1671972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4319152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4551501088256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4324835555328 _step‚18 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 220456648704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4426347439570 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4426244596974 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 226665532928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4313383155712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1710292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1710047 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 220544085504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4206327929792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 4206344969152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1671972 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4328824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 220456648704 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 219916667182 % _timestamp‚1.7858058263315775e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2647182 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚719708 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚719465 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚517 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4313383155712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4313400195072 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2656852 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 104.623483358 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4319152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2656852 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 220544085504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4313400195072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4328824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4533944280576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245\r¦0Þ+ _wandbruntime‚104  _runtime‚104šZö÷ßñ /loss/FRt2/detection_loss‚3.8081604526801542 $loss/total_loss‚69.9528723027139 0loss/FBt2/assignment_loss‚2.3125559034801664 -loss/FRt1/detection_loss‚3.99794215968875 0loss/SRqqt1/detection_loss‚3.313414515483947 0loss/SRqqt2/detection_loss‚4.155432208404912 .loss/FBt2/detection_loss‚18.28999995901471 trainer/global_step‚249 _runtime‚ 104.624174353 _step‚19 /loss/FRt2/assignment_loss‚5.801190387118947 1loss/SRqqt1/assignment_loss‚4.528765755607968 .loss/FBt1/detection_loss‚11.07200566942201 epoch‚0 % _timestamp‚1.7858058263326972e+09 /loss/FRt1/assignment_loss‚5.800463116446207 1loss/SRqqt2/assignment_loss‚4.531225380959449 0loss/FBt1/assignment_loss‚2.3417167944066666DædVûàõ $loss/total_loss‚69.9528723027139 0loss/FBt2/assignment_loss‚2.3125559034801664 -loss/FRt1/detection_loss‚3.99794215968875 0loss/SRqqt1/detection_loss‚3.313414515483947 0loss/SRqqt2/detection_loss‚4.155432208404912 .loss/FBt2/detection_loss‚18.28999995901471 trainer/global_step‚249 _runtime‚ 104.624174353 _step‚19 /loss/FRt2/assignment_loss‚5.801190387118947 1loss/SRqqt1/assignment_loss‚4.528765755607968 .loss/FBt1/detection_loss‚11.07200566942201 epoch‚0 % _timestamp‚1.7858058263326972e+09 /loss/FRt1/assignment_loss‚5.800463116446207 1loss/SRqqt2/assignment_loss‚4.531225380959449 0loss/FBt1/assignment_loss‚2.3417167944066666 /loss/FRt2/detection_loss‚3.80816045268015422+/Yœá‚(:‘ ‚øÄÓþ¬ÓÒ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚151.198' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.99888888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯¨ž©,â‚(:¡ ‚øÄÓËÖÕ! disk.nvme0c0n1.out‚ 5.25390625 disk.rbd2.out‚0 disk.dm-1.in‚0% disk./.usageGB‚1237.6878242492676 disk.rbd4.in‚0 disk.rbd4.out‚0 disk.dm-0.in‚0 network.sent‚236459 disk.sda2.out‚0 disk.drbd1743.out‚ 0.14453125 disk.rbd3.in‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.drbd1743.in‚0 disk.dm-0.out‚ 5.25390625% memory_percent‚41.202521324157715 disk.rbd1.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚0* proc.memory.percent‚17.193186283111572 proc.cpu.threads‚29) disk./.usagePercent‚83.06226837462332 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 11267.7265625 cpu‚14.238504337276813 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 5.25390625 disk.sda.out‚ 4.9140625 disk.sda3.out‚ 4.9140625( proc.memory.availableMB‚ 38533.515625 disk.dm-1.out‚ 4.9140625 disk.rbd1.in‚0 network.recv‚57249 disk.rbd5.out‚0F™ˆCœã‚(:‘ ‘øÄÓ€‰ÇÒ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚144.992' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚80.55111111111111 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î#CÕ2ä‚(:§ ‘øÄÓœÒõÕ) disk./.usagePercent‚83.09595973415195 disk.rbd5.out‚0 disk.rbd4.in‚0* proc.memory.percent‚17.193186283111572 network.sent‚281100 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 5.572265625 disk.sda.in‚0% disk./.usageGB‚1238.1898498535156 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0" disk.nvme0c0n1.out‚ 5.572265625 disk.rbd2.out‚0 network.recv‚59446 disk.drbd1743.out‚ 0.1640625 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 5.51171875% memory_percent‚41.202348470687866 proc.cpu.threads‚29 disk.dm-0.out‚ 5.572265625 disk.sda3.in‚0 cpu‚14.22374236550505 disk.sda1.in‚0 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.rbd1.out‚0 disk.sda1.out‚0 disk.sda.out‚ 5.51171875 disk.sda3.out‚ 5.51171875# proc.memory.rssMB‚ 11267.7265625 disk.rbd1.in‚0 disk.dm-0.in‚0 disk.rbd5.in‚0* proc.memory.availableMB‚38533.62890625h$E{0å+ _wandbruntime‚124  _runtime‚124˜- Äæ¾ & lr-AdamW/pg1‚7.705883759106523e-06 & lr-AdamW/pg2‚7.705883759106523e-06 trainer/global_step‚299 % _timestamp‚1.7858058466629148e+09 _runtime‚ 124.954568682 _step‚202‡¯ Èç trainer/global_step‚299 % _timestamp‚1.7858058466629148e+09 _runtime‚ 124.954568682 _step‚20 & lr-AdamW/pg1‚7.705883759106523e-06 & lr-AdamW/pg2‚7.705883759106523e-06rSƒ‘0è+ _wandbruntime‚124  _runtime‚124,d»©A é‰O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 270799512064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5154554543104 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5438758886912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1997479 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1997477 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2044019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚860524 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5418127707648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 26366059 »ƒN4176 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚860748 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1183497 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5438914464768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3164661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5167984278528 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5289524030940 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5026596373472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 124.958984128 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 263573164544 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚299 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 263013460592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5154571582464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3174317 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5026613412832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3174317 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 % _timestamp‚1.7858058466638536e+09 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚299 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 263013460592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5154571582464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3174317 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5026613412832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3174317 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 % _timestamp‚1.7858058466638536e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5418127707648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 263660594176 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚860748 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1183497 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5438914464768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3164661 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5167984278528 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5289524030940 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5026596373472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚ 124.958984128 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 263573164544 HDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2050790 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5185191395328 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5179304 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 264452784128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1187432 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5456886680576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚863611 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3184854 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2051045 % _timestamp‚1.7858058470287538e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5188976 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5457035959808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5307244519078 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5171721306112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5307141676482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5043353585152 _step‚22 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2004120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3175184 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2004122 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1187434 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5179304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 264540220928 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 271720188928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5185166491648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5436278566400 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5436174090240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0lŽV£3í—M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5188976 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5457035959808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5307244519078 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5171721306112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5307141676482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5043353585152 _step‚22 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2004120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3175184 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2004122 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1187434 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5179304 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 264540220928 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 271720188928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5185166491648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5436278566400 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5436174090240 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 263805130690 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2004122 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2050790 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5185191395328 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5179304 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 264452784128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1187432 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5456886680576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚863611 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3184854 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2051045 % _timestamp‚1.7858058470287538e+09£Þ0î+ _wandbruntime‚125  _runtime‚125üë:Œøïò .loss/FBt1/detection_loss‚19.97254450776075 /loss/FRt2/assignment_loss‚5.921983999013901 0loss/SRqqt2/detection_loss‚4.407059165271553 0loss/FRt1/assignment_loss‚5.6629178698469955 /loss/FBt2/detection_loss‚21.446131587028503 trainer/global_step‚299 _step‚23 .loss/FRt1/detection_loss‚4.155175332434293 .loss/FRt2/detection_loss‚4.192025565728545 % _timestamp‚1.7858058470299046e+09 _runtime‚ 125.32136398 %loss/total_loss‚82.19112429106923 epoch‚0 1loss/SRqqt1/assignment_loss‚4.489719493048532 1loss/SRqqt2/assignment_loss‚4.514058241973052 /loss/FBt1/assignment_loss‚2.271648231305574 /loss/FBt2/assignment_loss‚2.297801415125529 1loss/SRqqt1/detection_loss‚2.8600588825319995÷‘'@üðö /loss/FBt2/detection_loss‚21.446131587028503 trainer/global_step‚299 _step‚23 .loss/FRt1/detection_loss‚4.155175332434293 .loss/FRt2/detection_loss‚4.192025565728545 % _timestamp‚1.7858058470299046e+09 _runtime‚ 125.32136398 %loss/total_loss‚82.19112429106923 epoch‚0 1loss/SRqqt1/assignment_loss‚4.489719493048532 1loss/SRqqt2/assignment_loss‚4.514058241973052 /loss/FBt1/assignment_loss‚2.271648231305574 /loss/FBt2/assignment_loss‚2.297801415125529 1loss/SRqqt1/detection_loss‚2.8600588825319995 .loss/FBt1/detection_loss‚19.97254450776075 /loss/FRt2/assignment_loss‚5.921983999013901 0loss/SRqqt2/detection_loss‚4.407059165271553 0loss/FRt1/assignment_loss‚5.6629178698469955ZŽþ¬œñ‚(:‘  øÄÓ¾â×Ò gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚156.874' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚87.15222222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢» 2ò‚(:§  øÄÓ°“€Ö disk.sda1.out‚0 disk.sda3.out‚ 6.37890625 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 cpu‚14.223277635828987 proc.cpu.threads‚29 disk.dm-0.out‚ 603.466796875 disk.rbd3.out‚0 disk.rbd5.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 11267.7265625) disk./.usagePercent‚83.10426410654763 disk.sda.out‚ 6.37890625 disk.rbd2.in‚0 disk.dm-1.out‚ 6.37890625 disk.rbd1.in‚0 disk.sda2.out‚0$ memory_percent‚41.20321273803711$ disk./.usageGB‚1238.313591003418 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd1.out‚0 network.sent‚283754" disk.nvme0n1.out‚ 603.466796875 disk.dm-1.in‚0 network.recv‚60688 disk.dm-0.in‚0 disk.nvme0n1.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚0$ disk.nvme0c0n1.out‚ 603.466796875 disk.rbd2.out‚0 disk.sda2.in‚0 disk.drbd1743.out‚0.171875& proc.memory.availableMB‚ 38533.0625* proc.memory.percent‚17.1931862831115721Š~$0ó+ _wandbruntime‚144  _runtime‚144”;WÃô½ trainer/global_step‚349 $ _timestamp‚1.785805866087459e+09 _runtime‚ 144.379129784 _step‚24 & lr-AdamW/pg1‚8.994493083371828e-06 & lr-AdamW/pg2‚8.994493083371828e-06VŒ°CÇõÁ & lr-AdamW/pg1‚8.994493083371828e-06 & lr-AdamW/pg2‚8.994493083371828e-06 trainer/global_step‚349 $ _timestamp‚1.785805866087459e+09 _runtime‚ 144.379129784 _step‚24ª’30ö+ _wandbruntime‚144  _runtime‚144»”4‘'÷‹O % _timestamp‚1.7858058660884037e+09 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 307569299968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6012909732864 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2329627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6028315214848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3702319 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2384678 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6170523952932 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3692663 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6028340118528 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 306816120976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2329629 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6344296672768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6320461993472 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2329629 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6320461993472 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 315981457920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6012909732864 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6320566462464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5863604989472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 316120520704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6012892693504 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 307656729600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6320566462464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚ 144.384157789 _step‚25 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3702319 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1380345 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2384930 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6344460639232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6012892693504 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 316120520704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 6012892693504 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 307656729600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6320566462464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚ 144.384157789 _step‚25 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3702319 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1380345 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2384930 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6344460639232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6012892693504 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6028315214848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3702319 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2384678 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 6170523952932 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3692663 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6028340118528 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 306816120976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2329629 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6344296672768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚349 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6320461993472 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2329629 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6320461993472 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 315981457920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6012909732864 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 6320566462464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5863604989472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121Q˦ 0ù+ _wandbruntime‚144  _runtime‚144{9Âú•M REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2391465 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 308448919552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6030059456512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3712856 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3703186 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6030059456512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858058664420333e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2336270 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6039456 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6338508376064 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2336270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6045522331648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6338612852224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1007424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 308536356352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1007183 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 307693594198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6188038755990 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1384282 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 316883440640 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6188141598586 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6362405772288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6039456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6030076495872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 308448919552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1384284 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 317020399104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6049128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3712856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6049128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2336272 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 144.734179121 _step‚26 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6030076495872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3703186 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5880345161792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2391708 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 307779397434 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6362567634432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936£ Ÿ&û™M YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6030076495872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3703186 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5880345161792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2391708 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 307779397434 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6362567634432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚349 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2391465 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 308448919552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6030059456512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3712856 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3703186 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6030059456512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858058664420333e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2336270 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6039456 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6338508376064 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2336270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6045522331648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6338612852224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1007424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 308536356352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1007183 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 307693594198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 6188038755990 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1384282 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 316883440640 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 6188141598586 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6362405772288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6039456 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6030076495872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 308448919552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1384284 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 317020399104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6049128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3712856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6049128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2336272 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 144.734179121 _step‚26áëJ0ü+ _wandbruntime‚144  _runtime‚144¯tuR÷ýñ .loss/FBt2/detection_loss‚13.81209755992448 epoch‚0 _step‚27 /loss/FBt1/assignment_loss‚2.160181665420532 _runtime‚ 144.734858144 0loss/SRqqt1/assignment_loss‚4.52270582318306 1loss/SRqqt2/assignment_loss‚4.494992932276939 /loss/FBt2/assignment_loss‚2.241465135856911 0loss/SRqqt2/detection_loss‚4.839657929851048 %loss/total_loss‚80.13893590324071 trainer/global_step‚349 % _timestamp‚1.7858058664433913e+09 1loss/SRqqt1/detection_loss‚2.6556043596938252 .loss/FBt1/detection_loss‚25.51312290628751 /loss/FRt1/assignment_loss‚5.797207911809285 /loss/FRt2/assignment_loss‚5.749745491074353 .loss/FRt1/detection_loss‚4.309907723313723 /loss/FRt2/detection_loss‚4.0422464645490415nµ‹ûþõ /loss/FRt1/assignment_loss‚5.797207911809285 /loss/FRt2/assignment_loss‚5.749745491074353 .loss/FRt1/detection_loss‚4.309907723313723 /loss/FRt2/detection_loss‚4.0422464645490415 .loss/FBt2/detection_loss‚13.81209755992448 epoch‚0 _step‚27 /loss/FBt1/assignment_loss‚2.160181665420532 _runtime‚ 144.734858144 0loss/SRqqt1/assignment_loss‚4.52270582318306 1loss/SRqqt2/assignment_loss‚4.494992932276939 /loss/FBt2/assignment_loss‚2.241465135856911 0loss/SRqqt2/detection_loss‚4.839657929851048 %loss/total_loss‚80.13893590324071 trainer/global_step‚349 % _timestamp‚1.7858058664433913e+09 1loss/SRqqt1/detection_loss‚2.6556043596938252 .loss/FBt1/detection_loss‚25.51312290628751Q†é—œÿ‚(:‘ ¯øÄÓäžïÒ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚132.573' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚73.65166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©‚Û#6€‚(:« ¯øÄÓÈÓ¤Ö disk.rbd4.in‚0 disk.nvme0c0n1.in‚0 disk.sda.in‚0 disk.dm-1.out‚6.953125 disk.rbd3.in‚0# disk.nvme0n1.out‚728.6845703125 disk.rbd5.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 11267.7265625% disk.nvme0c0n1.out‚728.6845703125 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚0 disk.rbd2.out‚0 disk.sda3.out‚6.953125* proc.memory.availableMB‚38532.87109375 disk.dm-0.in‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚6.953125 disk.dm-1.in‚0 cpu‚14.213290560510291 proc.cpu.threads‚29 network.sent‚328521 network.recv‚63075) disk./.usagePercent‚83.10422980153804% disk./.usageGB‚1238.3130798339844 disk.rbd1.out‚0 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.19140625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd4.out‚0* proc.memory.percent‚17.193186283111572 disk.dm-0.out‚728.6845703125$ memory_percent‚41.20350480079651 LE–0+ _wandbruntime‚164  _runtime‚164TÊ÷âÆ‚À ' lr-AdamW/pg1‚1.0283102407637132e-05 ' lr-AdamW/pg2‚1.0283102407637132e-05 trainer/global_step‚399 % _timestamp‚1.7858058858145201e+09 _runtime‚ 164.106203679 _step‚28ÌšÀÊƒÄ ' lr-AdamW/pg1‚1.0283102407637132e-05 ' lr-AdamW/pg2‚1.0283102407637132e-05 trainer/global_step‚399 % _timestamp‚1.7858058858145201e+09 _runtime‚ 164.106203679 _step‚28½l©0„+ _wandbruntime‚164  _runtime‚164 d%‘'…‹O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6700613605472 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2661779 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1577195 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6882442 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4230321 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6700630644832 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 350790387608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 trainer/global_step‚399 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351565435392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2725603 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7222796279296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4230321 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7249990581248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 361294622720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7222900748288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6871247883264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858058858155077e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 351652865024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 361161851392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7222900748288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6892100 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1148408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6888695958528 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7051318189956 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2725860 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7249832906240 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 351652865024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4220665 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6871247883264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1148663 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚29 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2661777 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7222796279296 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 351565435392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6888671054848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 164.110980673ÿä:íø†O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2661779 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4220665 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7222796279296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4230321 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7249990581248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 361294622720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7222900748288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6871247883264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858058858155077e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 351652865024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 361161851392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7222900748288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6892100 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1148408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6888695958528 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7051318189956 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2725860 QIDeviceStatsMonitor.on_train_batcÿ(=‚h_start/inactive_split.small_pool.current‚255 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7249832906240 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 351652865024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4220665 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6871247883264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1148663 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚29 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2661777 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7222796279296 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 351565435392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6888671054848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 164.110980673 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6700613605472 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2661779 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1577195 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6882442 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4230321 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 6700630644832 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 350790387608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 trainer/global_step‚399 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 351565435392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2725603 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2732382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151509 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4231188 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 352445054976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6888414646272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6899608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6909280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2668422 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2732643 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7267940817920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4240858 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6905878171648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 351667860942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4240858 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6899608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 362062646272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7240947138048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6888397606912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚399 _runtime‚ 164.446620333 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7240842661888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6905903075328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6888414646272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2668422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1581134 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858058861546211e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1581132 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6909280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 352532491776 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7068935835498 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7240947138048 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 362195410432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392~®&ÕŸ&‰™M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1581132 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6909280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 352532491776 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7068935835498 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7240947138048 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 362195410432 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2668420 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6717370817152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4231188 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 352532491776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2732382 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151509 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4231188 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 352445054976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 6888414646272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6899608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6909280 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2668422 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2732643 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7267940817920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4240858 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6905878171648 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 351667860942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4240858 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6899608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 362062646272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7240947138048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6888397606912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚399 _runtime‚ 164.446620333 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7240842661888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6905903075328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6888414646272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2668422 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1581134 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858058861546211e+09äÐ0Š+ _wandbruntime‚164  _runtime‚164‘C û‹õ /loss/FRt1/assignment_loss‚5.629063240317411 _runtime‚ 164.447395576 0loss/FBt1/assignment_loss‚2.0325393676757812 0loss/FBt2/assignment_loss‚2.1408408246934414 .loss/FRt2/detection_loss‚4.067435924285724 0loss/SRqqt2/detection_loss‚5.224917635085091 epoch‚0 trainer/global_step‚399 % _timestamp‚1.7858058861559036e+09 _step‚31 /loss/FRt2/assignment_loss‚5.839712154718093 /loss/FBt1/detection_loss‚26.959406116179057 /loss/FBt2/detection_loss‚11.471935727167875 %loss/total_loss‚78.74775098976846 1loss/SRqqt1/assignment_loss‚4.511530815249812 1loss/SRqqt2/assignment_loss‚4.595008433811248 .loss/FRt1/detection_loss‚3.932847962129948 1loss/SRqqt1/detection_loss‚2.34251278845498157~‰ïÿŒù /loss/FRt2/assignment_loss‚5.839712154718093 /loss/FBt1/detection_loss‚26.959406116179057 /loss/FBt2/detection_loss‚11.471935727167875 %loss/total_loss‚78.74775098976846 1loss/SRqqt1/assignment_loss‚4.511530815249812 1loss/SRqqt2/assignment_loss‚4.595008433811248 .loss/FRt1/detection_loss‚3.932847962129948 1loss/SRqqt1/detection_loss‚2.3425127884549815 /loss/FRt1/assignment_loss‚5.629063240317411 _runtime‚ 164.447395576 0loss/FBt1/assignment_loss‚2.0325393676757812 0loss/FBt2/assignment_loss‚2.1408408246934414 .loss/FRt2/detection_loss‚4.067435924285724 0loss/SRqqt2/detection_loss‚5.224917635085091 epoch‚0 trainer/global_step‚399 % _timestamp‚1.7858058861559036e+09 _step‚31Ÿ±B‚(:… ¾øÄÓãàÅÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚145.764' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚80.98 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08QDñ'Ž‚(:œ ¾øÄÓߎûÔ disk.rbd5.out‚0% disk./.usageGB‚1238.3130683898926 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda3.out‚7.5% disk.nvme0c0n1.out‚728.9462890625 disk.sda.out‚7.5 disk.dm-1.in‚0 disk.rbd4.out‚0 network.recv‚65245 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd2.out‚0$ memory_percent‚41.20461344718933 cpu‚14.205570294635526 disk.rbd1.out‚0 disk.dm-0.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚83.10422903351544 disk.nvme0n1.in‚0 disk.dm-0.out‚728.9462890625* proc.memory.availableMB‚38532.14453125# proc.memory.rssMB‚ 11267.7265625* proc.memory.percent‚17.193186283111572# disk.nvme0n1.out‚728.9462890625 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.dm-1.out‚7.5 proc.cpu.threads‚29 disk.sda2.in‚0 network.sent‚373055 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.22265625 disk.rbd3.in‚0 disk.rbd3.out‚0¾\Ö›‚(: ÍøÄÓ§þñÓ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚155.51' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.39444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?x5‚(:ª ÍøÄÓ²æ¨×% disk.nvme0c0n1.out‚729.3056640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚83.10422928952298 disk.rbd0.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚29 disk.dm-1.out‚8.140625 disk.drbd1743.out‚ 0.2265625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚0 disk.rbd5.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-0.out‚729.3056640625$ memory_percent‚41.20427966117859 disk.rbd0.in‚0 disk.dm-0.in‚0# disk.nvme0n1.out‚729.3056640625 disk.sda1.in‚0 cpu‚14.201429250302011% disk./.usageGB‚1238.3130722045898 disk.sda.out‚8.140625 disk.sda3.out‚8.140625 disk.sda2.out‚0* proc.memory.availableMB‚38532.36328125# proc.memory.rssMB‚ 11267.7265625* proc.memory.percent‚17.193186283111572 network.sent‚417985 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.drbd1743.in‚0 network.recv‚67733Aä0‘+ _wandbruntime‚183  _runtime‚183¢ù“ Æ’À _runtime‚ 183.496516275 _step‚32 ' lr-AdamW/pg1‚1.1571711731902438e-05 ' lr-AdamW/pg2‚1.1571711731902438e-05 trainer/global_step‚449 % _timestamp‚1.7858059052048576e+09ÆùºÊ“Ä _runtime‚ 183.496516275 _step‚32 ' lr-AdamW/pg1‚1.1571711731902438e-05 ' lr-AdamW/pg2‚1.1571711731902438e-05 trainer/global_step‚449 % _timestamp‚1.7858059052048576e+09 ©½­ó0”+ _wandbruntime‚183  _runtime‚183>”eÇè•‹O OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4758323 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7729568994304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2993927 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4748667 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8125235034112 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8125130565120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7537622221472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1774045 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4758323 ZLDeviceStatsMonitor.o9´šÑ©!n_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7729586033664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 394678851116 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8155544859136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7749051798528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 395649000448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4748667 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1291356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 406362386432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 _runtime‚ 183.498001248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7932215269464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8125235034112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 395649000448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚33 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8125130565120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 395561570816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 406493060608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3065401 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7932318111948 % _timestamp‚1.7858059052058094e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2993929 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7749026894848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 395561570816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7729568994304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3065656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1774047 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7742594 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2993927 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7752252 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7729586033664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1291609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 394593047992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3065656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1774047 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7742594 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2993927 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7752252 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7729586033664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1291609 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 394593047992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4748667 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8125235034112 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8125130565120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7537622221472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1774045 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4758323 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7729586033664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 394678851116 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8155544859136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7749051798528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 395649000448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4748667 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1291356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 406362386432 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 _runtime‚ 183.498001248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7932215269464 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8125235034112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 395649000448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚33 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8125130565120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 395561570816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 406493060608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3065401 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7932318111948 % _timestamp‚1.7858059052058094e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2993929 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7749026894848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 395561570816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7729568994304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚449! &H0—+ _wandbruntime‚183  _runtime‚183€›&˜•M LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3000570 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7746752796672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4768860 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3072184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858059055382373e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4759190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8143176947712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 183.830223317 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 396441190400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8143281423872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 396528627200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4768860 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7769432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1777984 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 407266155008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8143281423872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7766258915328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 395470521214 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3000570 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7554362393792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8173657834496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8173500166656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4759190 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3072435 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7759760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7746735757312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 396528627200 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949832915006 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ô)§QÔ™™M Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8143176947712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 183.830223317 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 396441190400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8143281423872 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 396528627200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4768860 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7769432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1777984 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 407266155008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8143281423872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7766258915328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 395470521214 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3000570 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7554362393792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8173657834496 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8173500166656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batc®—Ì1Ëh_end/requested_bytes.small_pool.peak‚ 248886712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4759190 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3072435 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7759760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7746735757312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 396528627200 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949832915006 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3000570 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7746752796672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4768860 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3072184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858059055382373e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4759190 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800"×à¢0š+ _wandbruntime‚183  _runtime‚183\…Zçù›ó .loss/FRt1/detection_loss‚4.426990848276522 trainer/global_step‚449 % _timestamp‚1.7858059055398192e+09 0loss/FBt2/assignment_loss‚1.9934260447820027 .loss/FRt2/detection_loss‚4.022415734551571 0loss/SRqqt1/detection_loss‚2.064638735850652 .loss/FBt1/detection_loss‚31.32853260760506 .loss/FBt2/detection_loss‚17.88914292270229 %loss/total_loss‚87.95493719901322 _runtime‚ 183.831291639 _step‚35 /loss/FRt1/assignment_loss‚5.606400879946622 /loss/FRt2/assignment_loss‚5.984388475064878 1loss/SRqqt1/assignment_loss‚4.515329802831014 1loss/SRqqt2/assignment_loss‚4.490459125182208 0loss/FBt1/assignment_loss‚1.9057154059410095 1loss/SRqqt2/detection_loss‚3.7274966162793777 epoch‚0––C»ýœ÷ /loss/FRt1/assignment_loss‚5.606400879946622 /loss/FRt2/assignment_loss‚5.984388475064878 1loss/SRqqt1/assignment_loss‚4.515329802831014 1loss/SRqqt2/assignment_loss‚4.490459125182208 0loss/FBt1/assignment_loss‚1.9057154059410095 1loss/SRqqt2/detection_loss‚3.7274966162793777 epoch‚0 .loss/FRt1/detection_loss‚4.426990848276522 trainer/global_step‚449 % _timestamp‚1.7858059055398192e+09 0loss/FBt2/assignment_loss‚1.9934260447820027 .loss/FRt2/detection_loss‚4.022415734551571 0loss/SRqqt1/detection_loss‚2.064638735850652 .loss/FBt1/detection_loss‚31.32853260760506 .loss/FBt2/detection_loss‚17.88914292270229 %loss/total_loss‚87.95493719901322 _runtime‚ 183.831291639 _step‚35#^‚Ó’œ‚(:‘ ÜøÄÓÛªñÒ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚133.507' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚74.17055555555557 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹°U 8ž‚(:­ ÜøÄÓÔÏìÕ disk.drbd1743.out‚ 0.24609375 disk.dm-0.in‚0 network.recv‚68914 disk.dm-0.out‚729.5517578125 disk.rbd5.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 8.8515625 disk.rbd2.in‚0 disk.sda3.out‚ 8.8515625 disk.rbd1.in‚0% disk./.usageGB‚1238.3131294250488% disk.nvme0c0n1.out‚729.5517578125 disk.sda.in‚0 disk.dm-1.out‚ 8.8515625 disk.rbd1.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.193186283111572) disk./.usagePercent‚83.10423312963599 disk.rbd3.out‚0 disk.rbd4.out‚0 disk.rbd5.in‚0 disk.sda2.in‚0# memory_percent‚41.2040650844574 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚729.5517578125 disk.rbd2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚420641 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0* proc.memory.availableMB‚38532.50390625# proc.memory.rssMB‚ 11267.7265625 cpu‚14.193416210276807 disk.drbd1743.in‚0 disk.rbd4.in‚0¬—V™0Ÿ+ _wandbruntime‚203  _runtime‚203&9€/Æ À trainer/global_step‚499 % _timestamp‚1.7858059249796655e+09 _runtime‚ 203.271303552 _step‚36 ' lr-AdamW/pg1‚1.2860321056167744e-05 ' lr-AdamW/pg2‚1.2860321056167744e-05'¨í#Ê¡Ä ' lr-AdamW/pg1‚1.2860321056167744e-05 ' lr-AdamW/pg2‚1.2860321056167744e-05 trainer/global_step‚499 % _timestamp‚1.7858059249796655e+09 _runtime‚ 203.271303552 _step‚36$‰þ0¢+ _wandbruntime‚203  _runtime‚203³Dö²'£ŠO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 451655597568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8587924184064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1434138 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8587907144704 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8612404 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9027569319936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8587924184064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3326079 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 451516534784 trainer/global_step‚499 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 439645135872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5286325 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1970897 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5276669 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9027464850944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8609407638528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8813215191456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 438567314624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ì#–<”'¤ŽO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3405035 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9061063236096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 $ _timestamp‚1.785805924980589e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚37 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9027464850944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3404751 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _runtime‚ 203.275957533 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3326077 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 439557706240 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 438481511500 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8813112348972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3326079 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9027569319936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8609382734848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9027464850944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8609407638528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8813215191456 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 438567314624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 451655597568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8587924184064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1434138 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8587907144704 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8612404 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9027569319936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8587924184064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3326079 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 451516534784 trainer/global_step‚499 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 439645135872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5286325 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1970897 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5276669 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 439358984722 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8619912 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 452418435584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9045511233536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8629584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8605073907712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8830832837110 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8619912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MAQ®ƒ DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5287192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 440437325824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8391388049152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8605090947072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 452553296896 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 439444787958 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8391371009792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858059253127751e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8626589851648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3411552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9079168052224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800º‹N Ÿ&§™M KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8391388049152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8605090947072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 452553296896 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 439444787958 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8391371009792 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858059253127751e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8626589851648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3411552 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9079168052224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚499 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3332722 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3332720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1436962 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 439358984722 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8619912 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 452418435584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9045511233536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8629584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8605073907712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8830832837110 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8619912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5287192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 440437325824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360&éZ&?0¨+ _wandbruntime‚203  _runtime‚203Îà5Sû©õ 1loss/SRqqt1/assignment_loss‚4.514997288477507 2loss/SRqqt2/assignment_loss‚4.5658850064918175 0loss/FBt2/assignment_loss‚1.7057710509551198 /loss/FBt2/detection_loss‚19.799288012087345 trainer/global_step‚499 _runtime‚ 203.605977392 /loss/FRt1/assignment_loss‚5.586867028474808 /loss/FBt1/assignment_loss‚1.554342395067215 %loss/total_loss‚76.88746974959524 epoch‚0 % _timestamp‚1.7858059253140569e+09 _step‚39 /loss/FRt2/assignment_loss‚5.938033207603123 0loss/SRqqt1/detection_loss‚2.295419115707171 .loss/FRt1/detection_loss‚4.182213471084833 /loss/FRt2/detection_loss‚3.4924338594402955 0loss/SRqqt2/detection_loss‚4.773467897701619 /loss/FBt1/detection_loss‚18.4787514165043846K–ÿªù 0loss/FBt2/assignment_loss‚1.7057710509551198 /loss/FBt2/detection_loss‚19.799288012087345 trainer/global_step‚499 _runtime‚ 203.605977392 /loss/FRt1/assignment_loss‚5.586867028474808 /loss/FBt1/assignment_loss‚1.554342395067215 %loss/total_loss‚76.88746974959524 epoch‚0 % _timestamp‚1.7858059253140569e+09 _step‚39 /loss/FRt2/assignment_loss‚5.938033207603123 0loss/SRqqt1/detection_loss‚2.295419115707171 .loss/FRt1/detection_loss‚4.182213471084833 /loss/FRt2/detection_loss‚3.4924338594402955 0loss/SRqqt2/detection_loss‚4.773467897701619 /loss/FBt1/detection_loss‚18.478751416504384 1loss/SRqqt1/assignment_loss‚4.514997288477507 2loss/SRqqt2/assignment_loss‚4.5658850064918175':äLòœ«‚(:‘ ëøÄÓì·ÃÒ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚154.153' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.64055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tçÌ8¬‚(:­ ëøÄÓýóŠÕ disk.rbd1.in‚0 disk.rbd5.in‚0 disk.sda3.out‚9.515625) disk./.usagePercent‚83.10426052244215 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚729.92041015625 disk.sda.in‚0 disk.sda.out‚9.515625* proc.memory.percent‚17.193186283111572 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0# proc.memory.rssMB‚ 11267.7265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd4.out‚0& disk.nvme0c0n1.out‚729.92041015625 disk.rbd2.in‚0 proc.cpu.threads‚29 network.recv‚71440% disk./.usageGB‚1238.3135375976562 disk.sda2.out‚0 disk.drbd1743.out‚ 0.2734375! disk.dm-0.out‚729.92041015625 disk.rbd3.out‚0 disk.sda3.in‚0* proc.memory.availableMB‚38531.95703125 disk.rbd1.out‚0 disk.drbd1743.in‚0$ memory_percent‚41.20489954948425 network.sent‚465451 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd2.out‚0 disk.dm-1.out‚9.515625 cpu‚14.187327956948682 disk.dm-0.in‚0 disk.rbd5.out‚0 disk.dm-1.in‚0™Þ×p0­+ _wandbruntime‚222  _runtime‚222½™Å®¿ ' lr-AdamW/pg1‚1.4148930380433047e-05 ' lr-AdamW/pg2‚1.4148930380433047e-05 trainer/global_step‚549 $ _timestamp‚1.785805944393283e+09 _runtime‚ 222.684964819 _step‚40^ÿùɯà ' lr-AdamW/pg2‚1.4148930380433047e-05 trainer/global_step‚549 $ _timestamp‚1.785805944393283e+09 _runtime‚ 222.684964819 _step‚40 ' lr-AdamW/pg1‚1.4148930380433047e-05(–4»0°+ _wandbruntime‚222  _runtime‚222>¦X‡'±ŠO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9469738574848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9211656492832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 496873106432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9211639453472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚549 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9694009428480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 483553841664 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 483641271296 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3658229 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 482369975008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3658229 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 496742432256 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9446245295104 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5804671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9446262334464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9472556 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5814327 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5804671 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5814327 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 482455778132 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9929903605760 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9929799136768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9966636584960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9929799136768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2167747 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 222.689967425 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9469763478528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3745043 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1577561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2167745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483641271296 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9966481007104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 $ _timestamp‚1.785805944394265e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 483553841664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3745308U®ZN„²ŽO P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9929799136768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2167747 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 222.689967425 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9469763478528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3745043 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1577561 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2167745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483641271296 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9966481007104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 $ _timestamp‚1.785805944394265e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 483553841664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3745308 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9469738574848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9211656492832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 496873106432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9211639453472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚549 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9694009428480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 483553841664 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 483641271296 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3658229 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 482369975008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3658229 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 496742432256 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9446245295104 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5804671 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9446262334464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9472556 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5814327 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5804671 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5814327 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 482455778132 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9929903605760 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9929799136768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9966636584960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9489736 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9984592678912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9486970595328 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9489736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 497646987264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3751815 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 483333251466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9228379625792 _step‚42 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 483247448230 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9480064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 # _timestamp‚1.78580594473416e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5815194 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3664872 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9984748249600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3664870 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9947949995520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9947845519360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9463429097472 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 484520898048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚549 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9463412058112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5824864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9486945691648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 484520898048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5824864 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9947949995520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚ 223.026166698 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2171684 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1580385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9480064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2171682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9711729916618 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9463429097472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9463412058112i÷Ñh&µ—M \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9228396665152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3664872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9489736 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9984592678912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9486970595328 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9489736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 497646987264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3751815 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 483333251466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9228379625792 _step‚42 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 483247448230 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9480064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 # _timestamp‚1.78580594473416e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5815194 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3664872 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9984748249600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3664870 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9947949995520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9947845519360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9463429097472 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 484520898048 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚549 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9463412058112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5824864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9486945691648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 484520898048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5824864 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9947949995520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚ 223.026166698 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2171684 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1580385 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9480064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2171682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9711729916618 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9463429097472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9463412058112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497777654272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E¼‚(:³ ‰ùÄÓ‘‘Õ disk.rbd4.out‚0 disk.sda.out‚ 10.89453125 disk.dm-1.in‚0# proc.memory.rssMB‚ 11267.7265625 network.sent‚554780 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.dm-1.out‚ 10.89453125) proc.memory.availableMB‚ 38531.4453125 disk.rbd1.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚730.8759765625 disk.sda1.out‚0 disk.sda3.out‚ 10.89453125 disk.rbd1.in‚0 disk.drbd1743.in‚0 network.recv‚75955 disk.rbd5.out‚0 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚14.177002460572373 proc.cpu.threads‚29) disk./.usagePercent‚83.10425796236682 disk.drbd1743.out‚ 0.30078125 disk.dm-0.in‚0 disk.dm-0.out‚730.8759765625 disk.rbd3.out‚0 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0$ memory_percent‚41.20568037033081* proc.memory.percent‚17.193186283111572% disk.nvme0c0n1.out‚730.8759765625 disk.sda.in‚0 disk.rbd2.in‚0% disk./.usageGB‚1238.3134994506836ßM‹ƒ0½+ _wandbruntime‚242  _runtime‚242 ÒRrƾÀ _step‚44 ' lr-AdamW/pg1‚1.5437539704698353e-05 ' lr-AdamW/pg2‚1.5437539704698353e-05 trainer/global_step‚599 % _timestamp‚1.7858059638370447e+09 _runtime‚ 242.128679571ØÕd¿Ä _step‚44 ' lr-AdamW/pg1‚1.5437539704698353e-05 ' lr-AdamW/pg2‚1.5437539704698353e-Û®Ðf05 trainer/global_step‚599 % _timestamp‚1.7858059638370447e+09 _runtime‚ 242.128679571,®eµŽ0À+ _wandbruntime‚242  _runtime‚242ûUœj¥'ÁŸO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10304600484864 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10832237891584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526258438516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10304600484864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 % _timestamp‚1.7858059638381445e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10832133422592 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4085894 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 542055262208 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10832237891584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10575009350472 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10574906507988 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10048648069472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2364595 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6342329 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4085643 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6332673 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10832133422592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10332708 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 541926685184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527549977088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10872174580736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚45 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚599 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6342329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10330119318528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10330094414848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 242.130074195 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527637406720 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3990377 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2364597 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10304583445504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816‚“SÕ©'£O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10872021100032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10048665108832 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3990377 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 527637406720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 527549977088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 541926685184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 527549977088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10872174580736 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚45 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚599 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6342329 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10330119318528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10330094414848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 242.130074195 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 527637406720 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3990377 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2364597 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚10304583445504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10304600484864 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10832237891584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 526258438516 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚10304600484864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 % _timestamp‚1.7858059638381445e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10832133422592 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4085894 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 542055262208 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10832237891584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚10575009350472 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚10574906507988 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10048648069472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2364595 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6342329 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4085643 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6332673 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10832133422592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10332708 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B50Ã+ _wandbruntime‚242  _runtime‚242ôjX¯&Ä©M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10340216 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6343196 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3997020 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 527135911738 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 528517033472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527221714974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4092409 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚46 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10321750208512 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10592626996126 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10349888 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6343196 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10349888 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 542826775552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10592524153530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528517033472 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚10321750208512 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3997022 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6352866 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10890128307200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 542961636864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10340216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 242.470033095 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6352866 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10850284281344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10890288072192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10065405281152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10347301531648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3997022 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528429596672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10850284281344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10321767247872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3997020 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4092663 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251ˆ;v¾ Å­M VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6352866 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10850284281344 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10890288072192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚10065405281152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_traAΈ§in_batch_end/allocation.small_pool.current‚9670 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10347301531648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3997022 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 528429596672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚10850284281344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10321767247872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3997020 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4092663 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10340216 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6343196 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3997020 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 527135911738 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 528517033472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 527221714974 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4092409 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚46 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10321750208512 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚10592626996126 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10349888 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6343196 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10349888 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 542826775552 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10592524153530 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 528517033472 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚10321750208512 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3997022 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6352866 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10890128307200 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 542961636864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10340216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 242.470033095 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0.±XÒ"0Æ+ _wandbruntime‚242  _runtime‚242èÅ~ÿúÇô 0loss/FRt2/assignment_loss‚5.9077183135012366 0loss/FBt1/assignment_loss‚0.9397740704672677 .loss/FRt2/detection_loss‚3.394469735153178 0loss/SRqqt2/detection_loss‚5.178014503883534 epoch‚0 trainer/global_step‚599 _step‚47 2loss/SRqqt1/assignment_loss‚4.1132654220827165 0loss/FBt2/assignment_loss‚1.0310841909161321 .loss/FRt1/detection_loss‚3.750982343473218 .loss/FBt1/detection_loss‚26.51549871478762 $loss/total_loss‚76.6445509617819 /loss/FRt1/assignment_loss‚5.483632878823713 1loss/SRqqt2/assignment_loss‚4.220670680530736 1loss/SRqqt1/detection_loss‚2.6689173940929676 /loss/FBt2/detection_loss‚13.440522714069596 % _timestamp‚1.7858059641793013e+09 _runtime‚ 242.47108143ßrÁþÈø 0loss/FRt2/assignment_loss‚5.9077183135012366 0loss/FBt1/assignment_loss‚0.9397740704672677 .loss/FRt2/detection_loss‚3.394469735153178 0loss/SRqqt2/detection_loss‚5.178014503883534 epoch‚0 trainer/global_step‚599 _step‚47 2loss/SRqqt1/assignment_loss‚4.1132654220827165 0loss/FBt2/assignment_loss‚1.0310841909161321 .loss/FRt1/detection_loss‚3.750982343473218 .loss/FBt1/detection_loss‚26.51549871478762 $loss/total_loss‚76.6445509617819 /loss/FRt1/assignment_loss‚5.483632878823713 1loss/SRqqt2/assignment_loss‚4.220670680530736 1loss/SRqqt1/detection_loss‚2.6689173940929676 /loss/FBt2/detection_loss‚13.440522714069596 % _timestamp‚1.7858059641793013e+09 _runtime‚ 242.47108143/1£\®œÉ‚(:‘ ˜ùÄÓŒ¾ðÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.046' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.91444444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çW-X=Ê‚(:² ˜ùÄÓÀ„Õ# proc.memory.rssMB‚ 11267.7265625 disk.rbd4.out‚0 network.recv‚77552 disk.nvme0n1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0 disk.rbd2.out‚0 disk.drbd1743.out‚ 0.3203125 network.sent‚557640$ disk.nvme0n1.out‚736.17724609375 disk.rbd5.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚736.17724609375& proc.memory.availableMB‚ 38531.3125 disk.rbd4.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 11.67578125 disk.dm-0.in‚0$ memory_percent‚41.20588302612305) disk./.usagePercent‚83.10427434684901& disk.nvme0c0n1.out‚736.17724609375 disk.rbd3.in‚0 cpu‚14.173390013740233% disk./.usageGB‚1238.3137435913086 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.193186283111572 disk.sda2.out‚0 proc.cpu.threads‚29 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 11.67578125 disk.rbd3.out‚0 disk.sda.out‚ 11.67578125 disk.rbd1.in‚0 disk.drbd1743.in‚0¶ì³ 0Ë+ _wandbruntime‚261  _runtime‚261݉wÅÌ¿ ' lr-AdamW/pg1‚1.6726149028963658e-05 ' lr-AdamW/pg2‚1.6726149028963658e-05 trainer/global_step‚649 $ _timestamp‚1.785805983401851e+09 _runtime‚ 261.694966456 _step‚48;@8ÉÍà trainer/global_step‚649 $ _timestamp‚1.785805983401851e+09 _runtime‚ 261.694966456 _step‚48 ' lr-AdamW/pg1‚1.6726149028963658e-05 ' lr-AdamW/pg2‚1.6726149028963658e-050Ì_0Î+ _wandbruntime‚261  _runtime‚261Pðí¤'ÏžO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11162921595904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 261.695080469 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6860675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11455803587496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11190475158528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11734467708416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4322527 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 570146902024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚49 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6860675 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10885673724832 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6870331 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11190450254848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322527 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1865121 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 587060417024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4426566 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.785805983402605e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322529 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11734572177408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11734467708416 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 571633542144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2561445 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11455906429980 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 571633542144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 571546112512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1865366 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4426813 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11777668346880 trainer/global_step‚649 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11734572177408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 571546112512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162938635264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322529 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11192860 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 587193188352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11162921595904 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2561447 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11162921595904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 261.695080469 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6860675 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11455803587496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11190475158528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11734467708416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4322527 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 570146902024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚49 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6860675 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10885673724832 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6870331 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11190450254848 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4322527 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1865121 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 587060417024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4426566 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.785805983402605e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322529 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11734572177408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11734467708416 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 571633542144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2561445 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11455906429980 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 571633542144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 571546112512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1865366 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4426813 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11777668346880 trainer/global_step‚649 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11734572177408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 571546112512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11162938635264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4322529 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11192860 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 587193188352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11162921595904 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2561447 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 I0Ñ+ _wandbruntime‚262  _runtime‚262¾ˆ–(ŒÒ©M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2565384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11752618567168 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6880868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11795779601920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11180088358912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11180088358912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1867914 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640ÁÂ0ì#" SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 572425732096 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 571024375246 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11200368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587968756736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11210040 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 588097326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚ 262.032033029 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11752618567168 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 572513168896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11207682275328 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4433296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1868154 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚50 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329172 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ÛE/³&Ó­M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4433296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1868154 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚50 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329172 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2565384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11752618567168 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6880868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚11795779601920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11180088358912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11180088358912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1867914 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 572425732096 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 571024375246 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11200368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587968756736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11210040 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 588097326592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚ 262.032033029 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚11752618567168 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 572513168896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11207682275328 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚2512`óý)0Ô+ _wandbruntime‚262  _runtime‚262ÿ)–ûÕõ # _timestamp‚1.78580598374124e+09 1loss/SRqqt2/assignment_loss‚3.740814267464404 0loss/FBt2/assignment_loss‚0.8326333824432257 1loss/SRqqt1/detection_loss‚2.4449271607891956 epoch‚0 _runtime‚ 262.033019238 _step‚51 2loss/SRqqt1/assignment_loss‚3.6464205576961204 0loss/SRqqt2/detection_loss‚5.998715728521347 %loss/total_loss‚78.31186952392528 trainer/global_step‚649 /loss/FRt1/assignment_loss‚5.475036169687907 /loss/FRt2/assignment_loss‚5.949711037718731 .loss/FRt1/detection_loss‚4.476678036451339 0loss/FBt1/assignment_loss‚0.7647552117705345 /loss/FRt2/detection_loss‚3.5437197126448154 /loss/FBt1/detection_loss‚30.807326016326744 /loss/FBt2/detection_loss‚10.6311322424109243”DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12021276785664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4654677 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12336803509488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7398333 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614035365532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4765568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12336700667004 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 614121168656 _runtime‚ 282.311697707 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2007030 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4654679 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615629677568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2758297 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2007271 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 615542247936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11722665301472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12021276785664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 632376158208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12636906463232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 615542247936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4654677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7398333 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12683207156736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12050830998528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615629677568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2758295 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚699 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12636801994240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12636801994240¼'Âæ-Þ¢O K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7388677 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12053012 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚53 TGDeviceStatsMonitor.on_train_batcaXä–{&h_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12636906463232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12021259746304 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 632237095424 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11722682340832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12050830998528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615629677568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2758295 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚699 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12636801994240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12636801994240 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.785806004019562e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12043354 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12021259746304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12050806094848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4654679 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7388677 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12021276785664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4654677 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12336803509488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7398333 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614035365532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4765568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚12336700667004 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 614121168656 _runtime‚ 282.311697707 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2007030 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4654679 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 615629677568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2758297 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2007271 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 615542247936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11722665301472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12021276785664 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 632376158208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12636906463232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 615542247936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4654677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7398333 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12683207156736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12070192 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4661322 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _step‚54 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12070192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12038443548672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858060043978367e+09 _runtime‚ 282.689922037 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 614998641990 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616509304320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12701309826560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7408870 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4661320 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4772259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11739405473792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2009780 trainer/global_step‚699 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4661322 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12060520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4772012 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633145238528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 616421867520 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616509304320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12654848376832P¬¼:³&á­M VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12654952852992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12354318312546 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4661320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2762234 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12354421155142 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11739405473792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2009780 trainer/global_step‚699 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4661322 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12060520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4772012 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 633145238528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 616421867520 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 616509304320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12654848376832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7399200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12070192 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4661322 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _step‚54 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12070192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12038443548672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858060043978367e+09 _runtime‚ 282.689922037 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 614998641990 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616509304320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12701309826560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7408870 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4661320 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4772259 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Eýœå‚(:‘ ¶ùÄÓëõôÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚134.178' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚74.54333333333332 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Qðæ‚(: ¶ùÄÓ–£ÀÕ disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd2.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚0# proc.memory.rssMB‚ 11267.7265625 disk.rbd4.in‚0 disk.dm-1.out‚ 15.421875 disk.rbd3.out‚0* proc.memory.percent‚17.193186283111572 disk.nvme0n1.in‚ 9.73046875 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-0.out‚ 1316.24609375) proc.memory.availableMB‚ 38530.6328125 proc.cpu.threads‚29 network.sent‚647165) disk./.usagePercent‚83.06637345543486% disk./.usageGB‚1237.7489929199219" disk.nvme0n1.out‚ 1316.24609375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 15.421875 cpu‚14.15881548835429 network.recv‚81768 disk.rbd4.out‚0 disk.sda.out‚ 15.421875 disk.rbd2.in‚0 disk.dm-0.in‚ 9.73046875$ memory_percent‚41.20692014694214 disk.rbd0.out‚0 disk.drbd1743.in‚0 disk.sda2.out‚0 disk.rbd5.out‚0 disk.nv=¿©]me0c0n1.in‚ 9.73046875$ disk.nvme0c0n1.out‚ 1316.24609375 disk.drbd1743.out‚0.359375éðwœç‚(:‘ ÅùÄÓš·çÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.96777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÉ^è‚(:Ó ÅùÄÓö³²Õ( disk./.usagePercent‚83.0520549540448 disk.drbd1743.out‚ 0.38671875 disk.sda1.out‚0 disk.rbd5.out‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚1317.31591796875 disk.rbd2.in‚0* proc.memory.percent‚17.193186283111572 disk.rbd1.out‚0 disk.drbd1743.in‚0% disk.nvme0n1.out‚1317.31591796875 disk.sda.out‚ 16.46484375 disk.dm-1.out‚ 16.46484375 network.sent‚691920% disk./.usageGB‚1237.5356369018555 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda.in‚0* proc.memory.availableMB‚38530.82421875 cpu‚14.14939696016921 network.recv‚84078 disk.rbd3.in‚0 disk.rbd4.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 11267.7265625 disk.sda2.in‚0 disk.nvme0n1.in‚ 9.73046875 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.sda3.out‚ 16.46484375 proc.cpu.threads‚29" disk.dm-0.out‚1317.31591796875 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9.73046875 disk.nvme0c0n1.in‚ 9.73046875$ memory_percent‚41.20662808418274qÑL0é+ _wandbruntime‚303  _runtime‚3037/FÆêÀ ' lr-AdamW/pg1‚1.9303367677494267e-05 ' lr-AdamW/pg2‚1.9303367677494267e-05 trainer/global_step‚749 % _timestamp‚1.7858060254102452e+09 _runtime‚ 303.702935113 _step‚56ò¹•#ÊëÄ ' lr-AdamW/pg2‚1.9303367677494267e-05 trainer/global_step‚749 % _timestamp‚1.7858060254102452e+09 _runtime‚ 303.702935113 _step‚56 ' lr-AdamW/pg1‚1.9303367677494267e-058Á<=[0ì+ _wandbruntime‚303  _runtime‚303F¬Pœ£'íO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5105843 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13539240749056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4986829 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 659625812992 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 657923829040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _step‚57 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12879614936064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚749 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12903506 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12879597896704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13217700588996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4986829 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7926335 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785806025410995e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12911186838528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13217597746512 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7926335 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 659625812992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4986827 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2150965 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12879597896704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13588769737216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 677448030208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12559673917472 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7916679 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13539136280064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13539240749056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12911161934848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12913164 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 677582898688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 659538383360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936'á-¼§'î¡O dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚12911186838528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13217597746512 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7926335 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 659625812992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4986827 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2150965 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12879597896704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13588769737216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 677448030208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12559673917472 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7916679 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13539136280064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13539240749056 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12911161934848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12913164 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 677582898688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 659538383360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5105843 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13539240749056 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4986829 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 659625812992 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 657923829040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _step‚57 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12879614936064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚749 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12903506 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12879597896704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13217700588996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4986829 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7926335 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785806025410995e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚189†?ì40ï+ _wandbruntime‚304  _runtime‚304€ƒb‰¯&ð©M R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚13557182662656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2959084 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 660505439744 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5112896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2153571 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12930344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 trainer/global_step‚749 % _timestamp‚1.7858060257679508e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12896764659712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2153812 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 678483155968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 660418002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13606877111296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12928369051648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 660418002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12920672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 304.060159598 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4993472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5112653 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13606727832064 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 678358780416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12930344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4993470 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13557287138816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4993470 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13235215392054 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12576414089792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13557287138816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2959082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12576431129152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12896764659712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12896781699072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 660505439744 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5112896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2153571 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12930344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 trainer/global_step‚749 % _timestamp‚1.7858060257679508e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12896764659712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2153812 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 678483155968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 660418002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13606877111296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12928369051648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 660418002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12920672 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 304.060159598 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4993472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5112653 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13606727832064 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 678358780416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12930344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4993470 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚13557287138816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4993470 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13235215392054 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12576414089792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13557287138816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2959082 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12576431129152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12896764659712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12896781699072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5318979 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚799 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703534518784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 701898095672 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8444681 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14441470565888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13771517774848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5318977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 722674364416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13396682533472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8454337 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2294248 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8444681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14441470565888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13771517774848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5318977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 722674364416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13396682533472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8454337 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2294248 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8444681 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5318979 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚799 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 703534518784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 701898095672 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8444681=™$&Ÿ0ý+ _wandbruntime‚325  _runtime‚325ˆEJ¯&þ©M J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14512302049792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2297378 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14459621424640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14512461814784 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 702775569006 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 702689765770 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚799 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 704414138368 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5325622 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8464874 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13755102810112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14459621424640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14459516948480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 704501575168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚ 325.065661756 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2297139 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3155932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13788749795328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14116112471562 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13790496 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8464874 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13790496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858060467736475e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8455204 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13755119849472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723712019456 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5325620 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5453312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5453071 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13780824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 704501575168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5325620 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704414138368 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8455204kÀ¯¥³&ÿ­M TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 704414138368 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8455204 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14512302049792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2297378 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14459621424640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14512461814784 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 702775569006 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 702689765770 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚799 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 704414138368 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5325622 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8464874 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13755102810112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14459621424640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14459516948480 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 704501575168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _runtime‚ 325.065661756 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2297139 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3155932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13788749795328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14116112471562 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13790496 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8464874 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13790496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858060467736475e+09 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8455204 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13755119849472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 723712019456 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5325620 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5453312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5453071 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13780824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 704501575168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5325620>PÈ60€+ _wandbruntime‚325  _runtime‚325ÔÚ†Äûõ /loss/FRt1/assignment_loss‚5.543700732682881 2loss/SRqqt2/assignment_loss‚2.5475348606705666 .loss/FRt2/detection_loss‚4.190795035698475 0loss/SRqqt2/detection_loss‚4.821255884831771 _runtime‚ 325.066585057 _step‚63 1loss/SRqqt1/assignment_loss‚2.863899375646169 0loss/FBt1/assignment_loss‚0.7005377461512884 /loss/FBt1/detection_loss‚30.248137300213177 .loss/FBt2/detection_loss‚9.414399886319895 epoch‚0 trainer/global_step‚799 /loss/FRt2/assignment_loss‚5.948649901610154 0loss/FBt2/assignment_loss‚0.6579750757664442 1loss/SRqqt1/detection_loss‚2.2437244930965754 %loss/total_loss‚73.52935880839057 .loss/FRt1/detection_loss‚4.348748515703176 % _timestamp‚1.7858060467750368e+09+5fDÿ‚ù /loss/FRt1/assignment_loss‚5.543700732682881 2loss/SRqqt2/assignment_loss‚2.5475348606705666 .loss/FRt2/detection_loss‚4.190795035698475 0loss/SRqqt2/detection_loss‚4.821255884831771 _runtime‚ 325.066585057 _step‚63 1loss/SRqqt1/assignment_loss‚2.863899375646169 0loss/FBt1/assignment_loss‚0.7005377461512884 /loss/FBt1/detection_loss‚30.248137300213177 .loss/FBt2/detection_loss‚9.414399886319895 epoch‚0 trainer/global_step‚799 /loss/FRt2/assignment_loss‚5.948649901610154 0loss/FBt2/assignment_loss‚0.6579750757664442 1loss/SRqqt1/detection_loss‚2.2437244930965754 %loss/total_loss‚73.52935880839057 .loss/FRt1/detection_loss‚4.348748515703176 % _timestamp‚1.7858060467750368e+09?Dð“Ùœƒ‚(:‘ ãùÄÓÊÅÆÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚117.584' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.32444444444445 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LAZ†e„‚(:Ú ãùÄÓÕ·Ö disk.rbd2.out‚ 0.0859375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd4.out‚0 network.sent‚739344) disk./.usagePercent‚83.05888037091519 disk.sda1.in‚0 disk.rbd5.out‚0! disk.nvme0c0n1.in‚ 10.29296875 disk.sda.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚1382.08251953125 disk.rbd3.in‚0$ memory_percent‚38.60068321228027" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 network.recv‚87859 disk.rbd4.in‚0 disk.sda.out‚17.90625 disk.drbd1743.in‚0' proc.memory.availableMB‚ 40238.65625 cpu‚14.136272665634198 proc.cpu.threads‚29 disk.sda2.out‚0 disk.dm-1.out‚17.90625% disk./.usageGB‚1237.6373405456543% disk.nvme0n1.out‚1382.08251953125' disk.nvme0c0n1.out‚1382.08251953125 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 0.0859375 disk.dm-0.in‚ 10.29296875 disk.rbd3.out‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.drbd1743.out‚ 0.4140625 disk.nvme0n1.in‚ 10.29296875 disk.rbd0.out‚0 disk.sda3.out‚17.90625¶,’¢›…‚(: òùÄÓØ–âÒ gpu.0.gpu‚16.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚116.36' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚64.64444444444445 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kä›W†‚(:Ì òùÄÓ£Ô•Ö network.sent‚784315 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 0.0859375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.in‚10.5625% disk.nvme0n1.out‚1382.28271484375 cpu‚14.129681196499568 proc.cpu.threads‚29 disk.sda1.out‚0 disk.rbd4.out‚0 disk.nvme0c0n1.in‚10.5625' disk.nvme0c0n1.out‚1382.28271484375 disk.sda.out‚18.5625" disk.dm-0.out‚1382.28271484375$ memory_percent‚38.59978914260864) disk./.usagePercent‚83.05888088293025 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-0.in‚10.5625 disk.rbd3.in‚0 network.recv‚89991 disk.rbd1.out‚ 0.0859375 disk.drbd1743.out‚0.421875" proc.memory.rssMB‚ 11267.796875) proc.memory.availableMB‚ 40239.2421875 disk.rbd5.in‚0 disk.rbd5.out‚0 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda3.out‚18.5625 disk.dm-1.out‚18.5625 disk.sda2.out‚0% disk./.usageGB‚1237.6373481750488 disk.sda2.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.193293571472168j-0‡+ _wandbruntime‚346  _runtime‚346jÎÏ=Ĉ¾ _step‚64 ' lr-AdamW/pg1‚2.1880586326024876e-05 ' lr-AdamW/pg2‚2.1880586326024876e-05 trainer/global_step‚849 # _timestamp‚1.78580606797846e+09 _runtime‚ 346.270157103Û=¤Îȉ _step‚64 ' lr-AdamW/pg1‚2.1880586326024876e-05 ' lr-AdamW/pg2‚2.1880586326024876e-05 trainer/global_step‚849 # _timestamp‚1.78580606797846e+09 _runtime‚ 346.270157103@pdëñ0Š+ _wandbruntime‚346  _runtime‚346¢ðÙåÕ‹ŸO VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 747530654208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 745786559180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3348847 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3348845 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5651129 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14233708188832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15343804851712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747618083840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14596274197504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 745700756056 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15399916882432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15343909320704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15343909320704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5651127 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5651129 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8972683 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14633468 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5787619 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 trainer/global_step‚849 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5651127 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8982339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚65 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14596291236864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14623810 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15343804851712 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747618083840 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 768018363904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14596291236864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚'6\uÐ 9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14631898518528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14596274197504 _runtime‚ 346.275147156 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 767881398272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14979494748012 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14233691149472 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 747530654208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5787875 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2439028 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15343804851712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 747618083840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14596274197504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 745700756056 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15399916882432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15343909320704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15343909320704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5651127 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5651129 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8972683 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14633468 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5787619 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 trainer/global_step‚849 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5651127 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8982339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚65 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14596291236864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14623810 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15343804851712 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 747618083840 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 768018363904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14596291236864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14631898518528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14596274197504 _runtime‚ 346.275147156 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392A Rõ40+ _wandbruntime‚346  _runtime‚346ÖÙít®&Ž¨M NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5794628 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15361851234304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748497710592 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14613440960512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14650648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2441844 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15418022624256 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 746664032514 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14613457999872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15417864956416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14997112393666 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14650648 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8983206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 746578229278 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14250448361152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858060683323262e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748497710592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14640976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14649105635328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5657772 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768916988928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14250431321792 _runtime‚ 346.62435473 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14613440960512 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5657770 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8983206 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 748410273792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8992876 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚66 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5657770 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794383 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3352784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14640976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5657772 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0w1úç†$¬M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14640976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5657772 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5794628 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15361851234304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748497710592 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14613440960512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14650648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2441844 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15418022624256 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 746664032514 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14613457999872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15417864956416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14997112393666 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14650648 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8983206 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 746578229278 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14250448361152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858060683323262e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748497710592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14640976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14649105635328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5657772 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 768916988928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14250431321792 _runtime‚ 346.62435473 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14613440960512 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5657770 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8983206 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 748410273792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8992876 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMoni€U2Ÿ,tor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚66 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5657770 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794383 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3352784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160BŒÿ0+ _wandbruntime‚346  _runtime‚346pèú‘ô /loss/FBt1/detection_loss‚23.944694546858468 $ _timestamp‚1.785806068333902e+09 /loss/FRt2/assignment_loss‚5.924914305723166 2loss/SRqqt2/assignment_loss‚2.5431697315044617 .loss/FRt2/detection_loss‚4.036683255735832 trainer/global_step‚849 2loss/SRqqt1/assignment_loss‚2.8716583251953125 .loss/FRt1/detection_loss‚3.527192866348702 1loss/SRqqt1/detection_loss‚2.1239911236677522 %loss/total_loss‚61.88409133105392 /loss/FRt1/assignment_loss‚5.660361813462299 0loss/FBt1/assignment_loss‚0.6545087635517121 1loss/SRqqt2/detection_loss‚3.3001051987992245 -loss/FBt2/detection_loss‚6.77706130014465 epoch‚0 _runtime‚ 346.625398336 _step‚67 /loss/FBt2/assignment_loss‚0.519750100062337®¢^áþ’ø 1loss/SRqqt2/detection_loss‚3.3001051987992245 -loss/FBt2/detection_loss‚6.77706130014465 epoch‚0 _runtime‚ 346.625398336 _step‚67 /loss/FBt2/assignment_loss‚0.519750100062337 /loss/FBt1/detection_loss‚23.944694546858468 $ _timestamp‚1.785806068333902e+09 /loss/FRt2/assignment_loss‚5.924914305723166 2loss/SRqqt2/assignment_loss‚2.5431697315044617 .loss/FRt2/detection_loss‚4.036683255735832 trainer/global_step‚849 2loss/SRqqt1/assignment_loss‚2.8716583251953125 .loss/FRt1/detection_loss‚3.527192866348702 1loss/SRqqt1/detection_loss‚2.1239911236677522 %loss/total_loss‚61.88409133105392 /loss/FRt1/assignment_loss‚5.660361813462299 0loss/FBt1/assignment_loss‚0.6545087635517121C§VKœ“‚(:‘ úÄÓ»†êÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚147.417' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.89833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yì@,e”‚(:Ú úÄÓ¨ÝÖ% disk./.usageGB‚1237.6373176574707 disk.rbd4.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚1383.13037109375) proc.memory.availableMB‚ 40406.4140625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 11267.796875 network.sent‚786896 network.recv‚91311% disk.nvme0n1.out‚1383.13037109375 disk.nvme0c0n1.in‚10.5625' disk.nvme0c0n1.out‚1383.13037109375 disk.dm-1.out‚ 19.28515625 disk.rbd1.in‚0 disk.dm-0.in‚10.5625 disk.sda1.in‚0 disk.sda.out‚ 19.28515625 disk.rbd2.in‚0 disk.rbd3.out‚0$ memory_percent‚38.34470510482788* proc.memory.percent‚17.193293571472168 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1743.in‚0) disk./.usagePercent‚83.05887883486997 proc.cpu.threads‚29 disk.nvme0n1.in‚10.5625 disk.rbd5.out‚0 disk.rbd4.in‚0 disk.rbd2.out‚ 0.1484375 disk.sda3.in‚0 disk.drbd1743.out‚ 0.44921875 cpu‚14.124624316558862 disk.sda1.out‚0 disk.sda3.out‚ 19.28515625 disk.rbd1.out‚ 0.1484375uç°0•+ _wandbruntime‚367  _runtime‚367ˆHîÿÄ–¾ _runtime‚ 367.16095772 _step‚68 ' lr-AdamW/pg1‚2.3169195650290183e-05 ' lr-AdamW/pg2‚2.3169195650290183e-05 trainer/global_step‚899 $ _timestamp‚1.785806088867688e+09p ÷®È— ' lr-AdamW/pg1‚2.3169195650290183e-05 ' lr-AdamW/pg2‚2.3169195650290183e-05 trainer/global_step‚899 $ _timestamp‚1.785806088867688e+09 _runtime‚ 367.16095772 _step‚68Do!Z0˜+ _wandbruntime‚367  _runtime‚367iy`£¤'™žO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9510341 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15454629387264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15860391827520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15070699765472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 791526789632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15454629387264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 trainer/global_step‚899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15454612347904 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15493620 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 367.161047278 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5983277 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16246243606528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9500685 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3545697 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6128161 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15860288985036 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9510341 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9500685 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 789675022688 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15483962 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚69 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 791614219264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3545695 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16246243606528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5983279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16305351707648 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 813257121280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6128420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2582723 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5983279 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16246139137536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15492229454848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813122252800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16305511479808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0m´&ì¨'š¢O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15454612347904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15492254358528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.785806088868403e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 791614219264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3545695 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16246243606528 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5983279 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16305351707648 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 813257121280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6128420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2582723 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5983279 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16246139137536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15492229454848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 813122252800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16305511479808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9510341 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15454629387264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15860391827520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15070699765472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 791526789632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15454629387264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 trainer/global_step‚899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15454612347904 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15493620 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚ 367.161047278 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5983277 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16246243606528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9500685 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3545697 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6128161 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15860288985036 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9510341 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9500685 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 789675022688 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15483962 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚69 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 792493846016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15501128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2585570 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 814159238144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15471779110912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 367.516606443 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5989920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 792406409216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15510800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5989920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6134948 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16264185520128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15510800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16264289996288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9520878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16323467239936 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9511208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814030668288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 792406409216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16323620713472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15471779110912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5989922 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3549632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.785806089224556e+09 _step‚70 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15087456977152 XGDeviceStatsMonitor.Vï‚ÌÍon_train_batch_end/active_bytes.small_pool.allocated‚ 792493846016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15509461475328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15878009473174 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3549634 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15509436571648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5989922 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9511208 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16264289996288 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15471796150272 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 792406409216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15510800 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5989920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6134948 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16264185520128 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15510800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16264289996288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9520878 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16323467239936 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9511208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 814030668288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 792406409216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16323620713472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15471779110912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5989922 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3549632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.785806089224556e+09 _step‚70 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15087456977152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 792493846016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15501128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15509461475328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15878009473174 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3549634 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15509436571648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5989922 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9511208 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16264289996288 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15471796150272 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0FpBFö0ž+ _wandbruntime‚367  _runtime‚367†Í­íýŸ÷ 1loss/SRqqt2/assignment_loss‚2.471275935570399 0loss/FBt1/assignment_loss‚0.6731178343296051 1loss/FBt2/assignment_loss‚0.20041853041411975 .loss/FRt1/detection_loss‚4.268526256672646 _runtime‚ 367.517454893 /loss/FRt2/assignment_loss‚6.002036483199508 2loss/SRqqt1/assignment_loss‚2.9175786763718983 0loss/SRqqt2/detection_loss‚3.404420410682048 .loss/FBt2/detection_loss‚7.136023066659739 0loss/FRt1/assignment_loss‚5.4728030781996875 0loss/SRqqt1/detection_loss‚2.843221850060288 .loss/FBt1/detection_loss‚71.96059571802616 &loss/total_loss‚111.33534876880904 trainer/global_step‚899 % _timestamp‚1.7858060892259476e+09 _step‚71 /loss/FRt2/detection_loss‚3.9853309286229406 epoch‚0Adl` û /loss/FRt2/assignment_loss‚6.002036483199508 2loss/SRqqt1/assignment_loss‚2.9175786763718983 0loss/SRqqt2/detection_loss‚3.404420410682048 .loss/FBt2/detection_loss‚7.136023066659739 0loss/FRt1/assignment_loss‚5.4728030781996875 0loss/SRqqt1/detection_loss‚2.843221850060288 .loss/FBt1/detection_loss‚71.96059571802616 &loss/total_loss‚111.33534876880904 trainer/global_step‚899 % _timestamp‚1.7858060892259476e+09 _step‚71 /loss/FRt2/detection_loss‚3.9853309286229406 epoch‚0 1loss/SRqqt2/assignment_loss‚2.471275935570399 0loss/FBt1/assignment_loss‚0.6731178343296051 1loss/FBt2/assignment_loss‚0.20041853041411975 .loss/FRt1/detection_loss‚4.268526256672646 _runtime‚ 367.517454893G’}Iœ¡‚(:‘ úÄÓ¢ÄçÒ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.70055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÀmûo¢‚(:ä úÄÓäá× disk.rbd3.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 0.1484375 disk.dm-0.in‚10.5625 disk.rbd4.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 19.95703125* proc.memory.availableMB‚41455.89453125 disk.rbd3.out‚0 disk.sda.out‚ 19.95703125 disk.rbd1.out‚ 0.1484375% memory_percent‚36.743324995040894" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168) disk./.usagePercent‚83.06026460365283 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd4.in‚0 disk.sda.in‚0 cpu‚14.120094135496467 network.sent‚831737% disk./.usageGB‚1237.6579666137695 disk.rbd0.in‚0' disk.nvme0c0n1.out‚1445.81787109375 disk.sda3.in‚0 disk.sda2.out‚0% disk.nvme0n1.out‚1445.81787109375 disk.rbd5.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.44921875 disk.rbd5.out‚ 0.0859375 disk.nvme0c0n1.in‚10.5625 disk.sda3.out‚ 19.95703125 disk.nvme0n1.in‚10.5625 network.recv‚93580" disk.dm-0.out‚1445.81787109375-“P‹0£+ _wandbruntime‚389  _runtime‚389—X…ƤÀ _runtime‚ 389.582440991 _step‚72 ' lr-AdamW/pg1‚2.4457804974555488e-05 ' lr-AdamW/pg2‚2.4457804974555488e-05 trainer/global_step‚949 % _timestamp‚1.7858061112906873e+09¡êw洀 ' lr-AdamW/pg1‚2.4457804974555488e-05 ' lr-AdamW/pg2‚2.4457804974555488e-05 trainer/global_step‚949 % _timestamp‚1.7858061112906873e+09 _runtime‚ 389.582440991 _step‚72H³¼œ0¦+ _wandbruntime‚389  _runtime‚389Ï D©'§£O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835522925056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15907708381472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 389.586998063 _step‚73 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10028687 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6315427 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17148473423360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10038343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17148577892352 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16312967537664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6315427 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17148473423360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10028687 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16741288907028 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16353772 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17210949981184 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16312967537664 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6315429 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 858364686336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16352610198528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16344114 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835610354688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17211109753344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16741186064544 % _timestamp‚1.7858061112918916e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835610354688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 833477683072 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6315429 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16352585294848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16312950498304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚949 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835522925056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6470193 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10038343 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15907725420832 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17211109753344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16741186064544 % _timestamp‚1.7858061112918916e+09 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835610354688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 833477683072 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6315429 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16352585294848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16312950498304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚949 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 835522925056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6470193 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10038343 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15907725420832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10038343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17148577892352 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16312967537664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6315427 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17148473423360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10028687 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16741288907028 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16353772I¾‡ïÙœ©‚(:‘ ŸúÄÓàþøÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.56166666666667 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?õ™Jnª‚(:ã ŸúÄÓ毒Ö disk.rbd5.in‚0& disk.nvme0c0n1.out‚1508.5185546875 disk.sda3.out‚ 20.7578125* proc.memory.availableMB‚41455.62890625" proc.memory.rssMB‚ 11267.796875! disk.dm-0.out‚1508.5185546875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚10.5625 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd4.out‚0 disk.dm-1.in‚0 network.sent‚876489) disk./.usagePercent‚83.06435893214793 disk.rbd1.out‚ 0.1806640625 disk.sda.out‚ 20.7578125 proc.cpu.threads‚29 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd2.in‚0 cpu‚14.113390069376276% disk./.usageGB‚1237.7189750671387 disk.drbd1743.out‚ 0.4765625 disk.sda.in‚0 disk.rbd2.out‚ 0.1806640625 disk.dm-1.out‚ 20.7578125% memory_percent‚36.743730306625366 network.recv‚96071 disk.dm-0.in‚10.5625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚10.5625$ disk.nvme0n1.out‚1508.5185546875 disk.sda1.in‚0 disk.rbd5.out‚ 0.0859375 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168ùë™­0«+ _wandbruntime‚390  _runtime‚390’ܼ³&¬­M R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17166519805952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10048880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 859268962816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15924448553792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6476741 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 390.018155288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836402544640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16758803710086 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16330117261312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836489981440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2730506 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17166624282112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17166624282112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17229219042304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16758906552682 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6322070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2730259 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834355156294 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16370952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16330134300672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15924465593152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16361280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚74 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10039210 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16369792411648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836489981440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6476990 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17229061374464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3746482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6322072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16330117261312 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16370952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16330134300672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130ñ:ž·&­±M E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16758803710086 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16330117261312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836489981440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2730506 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17166624282112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17166624282112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17229219042304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16758906552682 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6322070 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2730259 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 834355156294 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16370952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16330134300672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15924465593152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16361280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚74 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10039210 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16369792411648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836489981440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6476990 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17229061374464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3746482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6322072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16330117261312 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16370952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16330134300672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17166519805952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10048880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚528 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 859268962816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15924448553792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6476741 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 390.018155288 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836402544640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144JIËuº0®+ _wandbruntime‚390  _runtime‚390v©3¼ú¯ô epoch‚0 -loss/FRt2/detection_loss‚3.59613797723592 1loss/SRqqt2/detection_loss‚4.2854682932991315 .loss/FBt1/detection_loss‚39.29269192285008 trainer/global_step‚949 _step‚75 0loss/FBt2/assignment_loss‚0.2702873669693182 0loss/FRt2/assignment_loss‚5.7428206244667805 1loss/SRqqt2/assignment_loss‚2.441475931335898 0loss/FBt1/assignment_loss‚0.6088006984856393 /loss/FRt1/detection_loss‚3.7655971733962788 1loss/SRqqt1/detection_loss‚2.0461912900689057 % _timestamp‚1.7858061117275345e+09 _runtime‚ 390.019118854 /loss/FRt1/assignment_loss‚5.605851465113023 1loss/SRqqt1/assignment_loss‚2.749318284034729 .loss/FBt2/detection_loss‚10.10398820713609 $loss/total_loss‚80.5086292343918ѾšÊþ°ø 0loss/FBt1/assignment_loss‚0.6088006984856393 /loss/FRt1/detection_loss‚3.7655971733962788 1loss/SRqqt1/detection_loss‚2.0461912900689057 % _timestamp‚1.7858061117275345e+09 _runtime‚ 390.019118854 /loss/FRt1/assignment_loss‚5.605851465113023 1loss/SRqqt1/assignment_loss‚2.749318284034729 .loss/FBt2/detection_loss‚10.10398820713609 $loss/total_loss‚80.5086292343918 epoch‚0 -loss/FRt2/detection_loss‚3.59613797723592 1loss/SRqqt2/detection_loss‚4.2854682932991315 .loss/FBt1/detection_loss‚39.29269192285008 trainer/global_step‚949 _step‚75 0loss/FBt2/assignment_loss‚0.2702873669693182 0loss/FRt2/assignment_loss‚5.7428206244667805 1loss/SRqqt2/assignment_loss‚2.441475931335898KbÇ㜱‚(:‘ ®úÄÓÐâóÒ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚147.292' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.82888888888888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OZqw²‚(:ì ®úÄÓ•ºÕ$ disk.nvme0n1.out‚1508.7880859375 disk.rbd4.in‚0* proc.memory.percent‚17.193293571472168) disk./.usagePercent‚83.06435765211026 disk.sda1.in‚0 disk.rbd5.in‚0" proc.memory.rssMB‚ 11267.796875 cpu‚14.10733533511943 network.sent‚921253 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.rbd4.out‚ 0.0859375 disk.sda2.in‚0 network.recv‚98173 disk.drbd1743.out‚ 0.4765625 disk.nvme0n1.in‚10.5625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd2.out‚ 0.1806640625 disk.dm-1.out‚ 21.27734375 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 21.27734375% disk./.usageGB‚1237.7189559936523 disk.dm-0.in‚10.5625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚10.5625& disk.nvme0c0n1.out‚1508.7880859375 disk.sda.out‚ 21.27734375! disk.dm-0.out‚1508.7880859375 disk.sda1.out‚0 disk.rbd5.out‚ 0.1484375 disk.rbd1.out‚ 0.1806640625 disk.sda2.out‚0) proc.memory.availableMB‚ 41456.0546875 disk.sda.in‚0 disk.dm-1.in‚0% memory_percent‚36.743080615997314 proc.cpu.threads‚292cr0³+ _wandbruntime‚413  _runtime‚413~ø&wÅ´¿ _step‚76 ' lr-AdamW/pg1‚2.5746414298820792e-05 ' lr-AdamW/pg2‚2.5746414298820792e-05 trainer/global_step‚999 $ _timestamp‚1.785806135425364e+09 _runtime‚ 413.717297914”ìbþɵà $ _timestamp‚1.785806135425364e+09 _runtime‚ 413.717297914 _step‚76 ' lr-AdamW/pg1‚2.5746414298820792e-05 ' lr-AdamW/pg2‚2.5746414298820792e-05 trainer/global_step‚999L‚.e0¶+ _wandbruntime‚413  _runtime‚413ÇPij·¢O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10556689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2870199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877366146580 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16744734036832 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18050807709184 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6647577 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18116653054464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17212941134848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2870460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16744716997472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚999 _runtime‚ 413.722136344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10566345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 879606490112 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18116491185152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17212966038528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10556689 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17171288648704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 877451949704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17171305688064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17204266 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _step‚77 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809594 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZID§V>eviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 879606490112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3939395 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785806135426724e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6647579 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18050912178176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17622083144052 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 879519060480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17171288648704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18050912178176 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6647579 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17622185986536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17213924 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18050807709184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Ôﲬ'¸¦O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18050912178176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17622083144052 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 879519060480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17171288648704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18050912178176 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6647579 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17622185986536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17213924 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18050807709184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10556689 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2870199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 877366146580 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16744734036832 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18050807709184 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6647577 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18116653054464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17212941134848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2870460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16744716997472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚999 _runtime‚ 413.722136344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10566345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 879606490112 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18116491185152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17212966038528 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10556689 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17171288648704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 877451949704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17171305688064 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17204266 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _step‚77 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6809594 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 879606490112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3939395 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 $ _timestamp‚1.785806135426724e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6647579 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0MºQá‰0¹+ _wandbruntime‚414  _runtime‚4146DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18068854091776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17188472451072 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 904456489472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 880486116864 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17221432 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 880398680064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816613 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 878243619802 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10576882 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6654222 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚78 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6654220 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 880486116864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 878329423038 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2873029 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17188455411712 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6654220 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 880398680064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17188472451072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3943332 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16761457169792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2873279 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17188455411712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18134604741120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10576882 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16761474209152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17231104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10567212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17230173155328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚ 414.166861329 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816361 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17231104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640~ú€Ë»±M WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17639803632190 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18068854091776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17188472451072 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 904456489472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 880486116864 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17221432 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 880398680064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6816613 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 878243619802 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10576882 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6654222 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚78 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6654220 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 880486116864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 878329423038 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2873029 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚17188455411712 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6654220 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 880398680064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17188472451072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3943332 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16761457169792 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2873279 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17188455411712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18134604741120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10576882 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16761474209152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17231104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚858032@—Îѧ 36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10567212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17230173155328 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _runtime‚ 414.166861329 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6816361 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17231104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360N q ž0¼+ _wandbruntime‚414  _runtime‚414±èµ"ù½ó 1loss/SRqqt1/assignment_loss‚2.673353546544125 %loss/total_loss‚61.37431407593033 _step‚79 .loss/FBt2/detection_loss‚4.898476904636823 .loss/FRt2/assignment_loss‚5.57793282920664 2loss/SRqqt2/assignment_loss‚2.5322797180901113 -loss/FRt2/detection_loss‚3.63925858756358 1loss/SRqqt2/detection_loss‚4.3564107279542466 /loss/FBt1/detection_loss‚24.534836853189127 trainer/global_step‚999 _runtime‚ 414.167732332 0loss/FBt1/assignment_loss‚0.5240306306098189 -loss/FRt1/detection_loss‚4.92484477115795 epoch‚0 /loss/FRt1/assignment_loss‚5.636594749987125 1loss/FBt2/assignment_loss‚0.16071931209462959 1loss/SRqqt1/detection_loss‚1.9155754448961522 $ _timestamp‚1.785806135875927e+09ÊXÉý¾÷ .loss/FRt2/assignment_loss‚5.57793282920664 2loss/SRqqt2/assignment_loss‚2.5322797180901113 -loss/FRt2/detection_loss‚3.63925858756358 1loss/SRqqt2/detection_loss‚4.3564107279542466 /loss/FBt1/detection_loss‚24.534836853189127 trainer/global_step‚999 _runtime‚ 414.167732332 0loss/FBt1/assignment_loss‚0.5240306306098189 -loss/FRt1/detection_loss‚4.92484477115795 epoch‚0 /loss/FRt1/assignment_loss‚5.636594749987125 1loss/FBt2/assignment_loss‚0.16071931209462959 1loss/SRqqt1/detection_loss‚1.9155754448961522 $ _timestamp‚1.785806135875927e+09 1loss/SRqqt1/assignment_loss‚2.673353546544125 %loss/total_loss‚61.37431407593033 _step‚79 .loss/FBt2/detection_loss‚4.898476904636823OFÑKœ¿‚(:‘ ½úÄÓÿüìÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.341' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.30055555555555 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êѪà|À‚(:ñ ½úÄÓ„ú·Ö disk.sda2.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 0.1484375 disk.rbd2.in‚0 disk.dm-1.in‚0# memory_percent‚36.7432177066803 cpu‚14.09741336538073 proc.cpu.threads‚29 disk.nvme0n1.in‚10.5625" proc.memory.rssMB‚ 11267.796875 disk.sda2.in‚0 disk.dm-0.in‚10.5625 disk.sda1.in‚0' disk.nvme0c0n1.out‚1509.23095703125 disk.rbd2.out‚ 0.24365234375* proc.memory.availableMB‚41455.96484375 network.sent‚923909% disk./.usageGB‚1237.7189559936523 disk.rbd0.in‚0 disk.nvme0c0n1.in‚10.5625 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd3.in‚0 network.recv‚99771 disk.rbd1.in‚0 disk.rbd1.out‚ 0.24365234375 disk.rbd5.out‚ 0.1484375 disk.sda3.out‚ 22.01171875 disk.rbd3.out‚0% disk.nvme0n1.out‚1509.23095703125 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd1743.out‚ 0.48828125" disk.dm-0.out‚1509.23095703125 disk.sda1.out‚0 disk.dm-1.out‚ 22.01171875) disk./.usagePercent‚83.06435765211026 disk.drbd1743.in‚0 disk.sda.out‚ 22.01171875Â5¾MÁ‚(:… ÌúÄÓ–áÙÒ gpu.0.gpu‚25.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚115.398' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚64.11 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ cã|‚(:ñ ÌúÄÓ»ÞÂÖ disk.rbd3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1743.out‚ 0.5078125) proc.memory.availableMB‚ 41455.6640625 proc.cpu.threads‚29 disk.sda.in‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 22.7265625 disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.193293571472168 network.recv‚102031 disk.rbd4.in‚0 disk.rbd5.in‚0 network.sent‚968883 disk.sda3.in‚0% disk.nvme0n1.out‚1509.43017578125% disk./.usageGB‚1237.7189559936523 disk.nvme0c0n1.in‚10.5625' disk.nvme0c0n1.out‚1509.43017578125 disk.sda.out‚ 22.7265625 disk.rbd0.in‚0 disk.rbd4.out‚ 0.1484375 disk.sda3.out‚ 22.7265625 disk.rbd1.out‚ 0.24365234375 disk.dm-0.in‚10.5625" proc.memory.rssMB‚ 11267.796875 cpu‚14.09326510243088) disk./.usagePercent‚83.06435765211026 disk.rbd2.out‚ 0.24365234375 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.in‚10.5625$ memory_percent‚36.74367666244507" disk.dm-0.out‚1509.43017578125 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd5.out‚ 0.1806640625¬Œfƒ0Ã+ _wandbruntime‚437  _runtime‚437…àÇÄÁ ' lr-AdamW/pg1‚2.7035023623086093e-05 ' lr-AdamW/pg2‚2.7035023623086093e-05 trainer/global_step‚1049 % _timestamp‚1.7858061594650073e+09 _runtime‚ 437.756704959 _step‚80÷1ÿfËÅÅ ' lr-AdamW/pg1‚2.7035023623086093e-05 ' lr-AdamW/pg2‚2.7035023623086093e-05 trainer/global_step‚1049 % _timestamp‚1.7858061594650073e+09 _runtime‚ 437.756704959 _step‚80P¬Š”0Æ+ _wandbruntime‚437  _runtime‚437%×ݪ'ǤO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18029643838464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3013478 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19022163265536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3013197 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18029643838464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 923602625536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6979729 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 948708615680 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 921340413212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 948841387008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6979727 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18074076 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4136247 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚17581742652832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 437.760961108 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18029626799104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18953141995008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 923515195904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6979727 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11084691 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18073296974848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17581725613472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚81 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7149442 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11084691 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18953246464000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18953141995008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4136245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 921254610088 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18953246464000 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 923602625536 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 923515195904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7149725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Q÷ò®'ȨO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7149725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18029643838464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3013478 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19022163265536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3013197 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18029643838464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 923602625536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6979729 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 948708615680 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 921340413212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 948841387008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6979727 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18074076 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4136247 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚17581742652832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 437.760961108 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18029626799104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18953141995008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 923515195904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6979727 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11084691 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18073296974848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚17581725613472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚81 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7149442 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11084691 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18953246464000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18953141995008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4136245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 921254610088 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18953246464000 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 923602625536 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 923515195904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Q“Fž0É+ _wandbruntime‚438  _runtime‚438*P,3%Ê­M [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19040271587840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 922217886546 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7156208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11104884 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18046793562112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3016026 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18090504091648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18520597869102 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18081584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3016278 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6986370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 438.122406173 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18091256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 924394815488 trainer/global_step‚1049 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18090528995328 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18971292853760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18046810601472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19040113920000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785806159830403e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121ÐFé·&˱M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949742592512 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18971292853760 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 924482252288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚82 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18091256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4140182 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18971188377600 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6986372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11095214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 924394815488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18081584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18046793562112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18046793562112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3016026 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18090504091648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18520597869102 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18081584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3016278 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6986370 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 438.122406173 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18091256 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 924394815488 trainer/global_step‚1049 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18090528995328 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18971292853760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18046810601472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19040113920000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785806159830403e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19040271587840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 922217886546 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7156208 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11104884 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7490874 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 993940961792 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7311879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19927593776640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18924570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 967598760960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 967598760960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18933652814848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19383980145552 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7311877 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11622349 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19855580749824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 967511331328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19855476280832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18887964949504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18418751268832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7311879 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚85 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7490621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18934228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11622349 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 994071635968 % _timestamp‚1.7858061811430628e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 965143073596 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18887981988864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18933677718528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18418734229472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18887981988864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7311877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3157526 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19855580749824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4333095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3157777 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 965228876720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19855476280832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1099 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18418734229472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18887981988864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7311877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 294045ÊP‹+5936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3157526 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19855580749824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4333095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3157777 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 965228876720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19855476280832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1099 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7490874 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 993940961792 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7311879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19927593776640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18924570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 967598760960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 967598760960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18933652814848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19383980145552 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7311877 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11622349 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19855580749824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 967511331328 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚19855476280832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18887964949504U®j}0×+ _wandbruntime‚459  _runtime‚459nÓ˜N´&Ø®M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19945858661888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1099 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19401494948610 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7318520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4337032 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968478387712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 968390950912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18435474401792 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 968478387712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 994973826560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18905131712512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3160375 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19401597791206 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18941736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18941736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7318522 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19873627139584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚ 459.797050594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11623216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚86 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18905131712512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18435491441152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 966020546818 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4337034 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19945703091200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11632886 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7497407 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19873627139584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11623216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18951408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3160619 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858061815049589e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18951408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7318522 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18950884835328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚05Ñ££¸&Ù²M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4337032 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968478387712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 968390950912 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18435474401792 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 968478387712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 994973826560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18905131712512 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3160375 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19401597791206 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18941736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18941736 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7318522 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚19873627139584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚ 459.797050594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11623216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚86 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18905131712512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18435491441152 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 966020546818 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4337034 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19945703091200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11632886 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7497407 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19873627139584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11623216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18951408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3160619 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858061815049589e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18951408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7318522 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18950884835328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19945858661888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1099 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19401494948610 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7318520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121V´x¬—0Ú+ _wandbruntime‚459  _runtime‚459LÙ ÕûÛõ /loss/FRt1/assignment_loss‚5.611380128633408 $loss/total_loss‚55.6771871485971 epoch‚0 _runtime‚ 459.797737732 /loss/FRt2/assignment_loss‚5.197289455227736 1loss/FBt2/assignment_loss‚0.24903687310870737 /loss/FRt1/detection_loss‚3.7961541822268847 /loss/FRt2/detection_loss‚3.8273236431726594 1loss/SRqqt1/assignment_loss‚2.431635957339714 0loss/SRqqt2/assignment_loss‚2.64332104069846 0loss/FBt1/assignment_loss‚0.5102098771991829 1loss/SRqqt1/detection_loss‚1.6492095166138134 /loss/FBt1/detection_loss‚18.865657934091157 .loss/FBt2/detection_loss‚6.382840597689205 trainer/global_step‚1099 $ _timestamp‚1.785806181506293e+09 0loss/SRqqt2/detection_loss‚4.513127942596163 _step‚87Wð(ÿÜù trainer/global_step‚1099 $ _timestamp‚1.785806181506293e+09 0loss/SRqqt2/detection_loss‚4.513127942596163 _step‚87 /loss/FRt1/assignment_loss‚5.611380128633408 $loss/total_loss‚55.6771871485971 epoch‚0 _runtime‚ 459.797737732 /loss/FRt2/assignment_loss‚5.197289455227736 1loss/FBt2/assignment_loss‚0.24903687310870737 /loss/FRt1/detection_loss‚3.7961541822268847 /loss/FRt2/detection_loss‚3.8273236431726594 1loss/SRqqt1/assignment_loss‚2.431635957339714 0loss/SRqqt2/assignment_loss‚2.64332104069846 0loss/FBt1/assignment_loss‚0.5102098771991829 1loss/SRqqt1/detection_loss‚1.6492095166138134 /loss/FBt1/detection_loss‚18.865657934091157 .loss/FBt2/detection_loss‚6.382840597689205W> C›Ý‚(: êúÄÓÔóïÒ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚121.08' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.26666666666667 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07U%i„Þ‚(:ù êúÄÓÌè¤Õ disk.rbd1.in‚0! disk.dm-0.out‚1510.0224609375 disk.rbd4.out‚ 0.1806640625 disk.sda.out‚ 23.98046875$ memory_percent‚36.74443960189819 cpu‚14.085702487248113 disk.dm-1.in‚0 disk.drbd1743.in‚0$ disk.nvme0n1.out‚1510.0224609375 disk.nvme0c0n1.in‚10.5625& disk.nvme0c0n1.out‚1510.0224609375 network.recv‚105409 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚10.5625 disk.sda1.out‚0) proc.memory.availableMB‚ 41455.1640625 network.sent‚ 1.016403e+06 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168) disk./.usagePercent‚83.06435765211026 disk.rbd2.out‚ 0.306640625 disk.dm-1.out‚ 23.98046875 disk.rbd4.in‚0 proc.cpu.threads‚29 disk.sda2.out‚0% disk./.usageGB‚1237.7189559936523 disk.rbd1.out‚ 0.306640625 disk.drbd1743.out‚ 0.53515625 disk.dm-0.in‚10.5625 disk.rbd5.in‚0 disk.rbd5.out‚ 0.24365234375 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.out‚ 23.98046875™!b›ß‚(: ùúÄÓŽžøÒ gpu.0.gpu‚16.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚112.97' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚62.76111111111111 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðe|«ˆà‚(:ý ùúÄÓœôÁÕ" disk.dm-0.out‚1510.32763671875 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚10.5625 disk.sda.out‚ 24.76171875 disk.dm-0.in‚10.5625 disk.rbd3.in‚0 proc.cpu.threads‚29 disk.dm-1.in‚0 disk.rbd5.in‚0' disk.nvme0c0n1.out‚1510.32763671875 disk.sda.in‚0 cpu‚14.079364538486503) disk./.usagePercent‚83.06435381199724 disk.drbd1743.out‚ 0.58984375 disk.sda1.out‚0 disk.rbd4.out‚ 0.24365234375 disk.sda2.out‚0" proc.memory.rssMB‚ 11267.796875 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.306640625% disk.nvme0n1.out‚1510.32763671875 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 41454.9140625 network.sent‚ 1.06117e+06 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚10.5625 disk.rbd5.out‚ 0.24365234375% memory_percent‚36.744821071624756 network.recv‚107580% disk./.usageGB‚1237.7188987731934 disk.rbd2.out‚ 0.306640625 disk.dm-1.out‚ 24.76171875* proc.memory.percent‚17.193293571472168 disk.sda3.out‚ 24.76171875 disk.sda2.in‚0 disk.drbd1743.in‚0Ù$æ¶0á+ _wandbruntime‚481  _runtime‚481äéüœÇâÁ trainer/global_step‚1149 % _timestamp‚1.7858062034126482e+09 _runtime‚ 481.706091476 _step‚88 ' lr-AdamW/pg1‚2.9612242271616706e-05 ' lr-AdamW/pg2‚2.9612242271616706e-05¡½ßËãÅ ' lr-AdamW/pg2‚2.9612242271616706e-05 trainer/global_step‚1149 % _timestamp‚1.7858062034126482e+09 _runtime‚ 481.706091476 _step‚88 ' lr-AdamW/pg1‚2.9612242271616706e-05Xi ¡0ä+ _wandbruntime‚481  _runtime‚481’$kå¬O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1011507466752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19794380 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7644027 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1009117340228 ZHDeviceStatsMonitor.on_train_bat¾F”æG$ch_start/allocated_bytes.small_pool.freed‚ 1011507466752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19746303099904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7831626 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7831365 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12140695 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20264774382576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19746320139264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20833113201664 % _timestamp‚1.7858062034134102e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7644027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20757810566656 xû{¶'æ°O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4529947 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19794008654848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20264774382576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19746320139264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20833113201664 % _timestamp‚1.7858062034134102e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7644027 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20757810566656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1011507466752 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19794380 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7644027 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1009117340228 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1011507466752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19746303099904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7831626 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7831365 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12140695 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19811560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19801888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20775961425408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19763486902272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1009994813562 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20282392028118 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20775961425408 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7838210 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1009909010326 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7838473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1012387086336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19811560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19801888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20775961425408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_<Ä4/Ìpool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19763486902272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1009994813562 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304589 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20282392028118 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20775961425408 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7838210 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1009909010326 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7838473 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1012387086336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21660144852480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1055591031808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055503602176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7976177 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20604658289664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7976177 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12668697 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20604641250304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21738845161984 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21738683292672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21145671462084 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8172728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20654389398528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20604658289664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1084318797824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20092768500832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7976179 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7976179 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4726797 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20092751461472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1055503602176 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚21660249321472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055591031808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚1199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 _step‚93 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21145774304568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1053005803736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20644874 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12668697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1084455763456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8172985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20604641250304 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21660144852480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1055591031808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1055503602176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7976177 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20604658289664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7976177 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12668697 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20604641250304 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21738845161984 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21738683292672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21145671462084 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8172728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20654389398528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20604658289664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1084318797824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20092768500832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7976179 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7976179 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4726797 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20092751461472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1055503602176 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚21660249321472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1055591031808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚1199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 _step‚93 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21145774304568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1053005803736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20644874 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12668697 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1084455763456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8172985 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20604641250304 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7982822 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3449036 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20671712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20662040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21678295711232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20671596515328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20621825052672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20671712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 trainer/global_step‚1199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12688890 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1053883277070 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1056470658560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8179770 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20671571611648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4730732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8179508 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4730734 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20662040 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7982820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20621808013312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056383221760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056383221760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20109508673152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚21678191235072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20621825052672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20621808013312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21756787247616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12688890 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12679220 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0H2 $À&÷ºM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12679220 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7982822 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3449036 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20671712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20662040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚21678295711232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20671596515328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20621825052672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20671712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 trainer/global_step‚1199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12688890 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1053883277070 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1056470658560 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8179770 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20671571611648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4730732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8179508 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4730734 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20662040 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7982820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20621808013312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1056383221760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1056383221760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20109508673152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚21678191235072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20621825052672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20621808013312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21756787247616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12688890 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12679220 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22026568541592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1099587167232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 trainer/global_step‚1249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20929760077472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22562479138304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13196699 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20929777116832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4923645 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1096808464120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3589015 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1096894267244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 525.516379239 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22562479138304 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21462979400704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21462996440064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8308329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1099499737600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858062472242877e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8512660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1099587167232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21462979400704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22562583607296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22026568541592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1099587167232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 trainer/global_step‚1249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20929760077472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22562479138304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13196699 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20929777116832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4923645 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1096808464120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3589015 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1096894267244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 525.516379239 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22562479138304 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21462979400704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21462996440064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8308329 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1099499737600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858062472242877e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8512660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1099587167232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚21462979400704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22562583607296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1100466793984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21522192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8519427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1130486181376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22662438536704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21531864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22044289029730 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22580629997056 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1100379357184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1100379357184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21531952355328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22044186187134 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8519679 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20946517289152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130355514368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22580629997056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21522192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858062476305957e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13216892 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8314972 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3591845 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21480163203072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21480146163712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚ 525.922612187 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8314972 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13207222 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1097771740578 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13216892 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8314970 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4927584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 _step‚98 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20946500249792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13207222…ù¿ôÀ&‡ºM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21531864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22044289029730 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22580629997056 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1100379357184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1100379357184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21531952355328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22044186187134 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8519679 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20946517289152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1130355514368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22580629997056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21522192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858062476305957e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13216892 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8314972 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3591845 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21480163203072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21480146163712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚ 525.922612187 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8314972 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13207222 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1097771740578 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13216892 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8314970 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4927584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 _step‚98 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20946500249792 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13207222 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22662282966016 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1100466793984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21522192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8519427 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1130486181376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22662438536704b}FF0ˆ+ _wandbruntime‚525  _runtime‚525ç\þ‰ø /loss/FRt1/assignment_loss‚5.178997948363021 /loss/FBt1/detection_loss‚25.841580937186684 epoch‚0 0loss/SRqqt2/assignment_loss‚2.02951882426676 1loss/FBt2/assignment_loss‚0.09354103488705891 .loss/FRt1/detection_loss‚4.377130036418502 1loss/SRqqt1/detection_loss‚1.1936106825540556 0loss/SRqqt2/detection_loss‚4.075817508132834 /loss/FBt2/detection_loss‚11.224536366853524 _step‚99 0loss/FRt2/assignment_loss‚3.9464628576028224 /loss/FRt2/detection_loss‚3.2767597460987594 trainer/global_step‚1249 % _timestamp‚1.7858062476319938e+09 %loss/total_loss‚63.97829357207822 1loss/SRqqt1/assignment_loss‚2.313534207016457 0loss/FBt1/assignment_loss‚0.4268034226977481 _runtime‚ 525.923534286¹lú’Šü 0loss/SRqqt2/assignment_loss‚2.02951882426676 1loss/FBt2/assignment_loss‚0.09354103488705891 .loss/FRt1/detection_loss‚4.377130036418502 1loss/SRqqt1/detection_loss‚1.1936106825540556 0loss/SRqqt2/detection_loss‚4.075817508132834 /loss/FBt2/detection_loss‚11.224536366853524 _step‚99 0loss/FRt2/assignment_loss‚3.9464628576028224 /loss/FRt2/detection_loss‚3.2767597460987594 trainer/global_step‚1249 % _timestamp‚1.7858062476319938e+09 %loss/total_loss‚63.97829357207822 1loss/SRqqt1/assignment_loss‚2.313534207016457 0loss/FBt1/assignment_loss‚0.4268034226977481 _runtime‚ 525.923534286 /loss/FRt1/assignment_loss‚5.178997948363021 /loss/FBt1/detection_loss‚25.841580937186684 epoch‚0c­=Vœ‹‚(:‘ µûÄÓ…ù½Ò gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.08055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘‘ŠÉ§Œ‚(:ø µûÄÓ¬íªÕ disk.sda2.in‚0 disk.drbd1743.in‚0" disk.dm-0.out‚1522.13623046875 disk.rbd4.in‚0 disk.rbd4.out‚ 0.306640625* proc.melGÜmory.availableMB‚41454.33984375 network.sent‚ 1.156546e+06 disk.rbd5.in‚0 disk.rbd2.out‚ 0.306640625 disk.dm-1.in‚0% memory_percent‚36.745697259902954 cpu‚14.065260741957498' disk./.usagePercent‚83.065092137726 disk.nvme0n1.in‚10.5625 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 27.46484375 disk.dm-1.out‚ 27.46484375 disk.rbd1.out‚ 0.306640625 disk.rbd3.out‚0% disk.nvme0n1.out‚1522.13623046875 disk.nvme0c0n1.in‚10.5625' disk.nvme0c0n1.out‚1522.13623046875 disk.rbd2.in‚0 proc.cpu.threads‚29 disk.sda2.out‚0 disk.rbd5.out‚ 0.306640625% disk./.usageGB‚1237.7299003601074 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.drbd1743.out‚ 0.6640625 disk.rbd0.in‚0 disk.sda3.out‚ 27.46484375" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 network.recv‚114899 disk.dm-0.in‚10.5625 disk.sda1.out‚0í°Œ¹0+ _wandbruntime‚546  _runtime‚546tÇÆŽÀ trainer/global_step‚1299 % _timestamp‚1.7858062686925123e+09 _runtime‚ 546.984201965 _step‚100 & lr-AdamW/pg1‚3.347807024441262e-05 & lr-AdamW/pg2‚3.347807024441262e-05xÍ"CÊÄ _runtime‚ 546.984201965 _step‚100 & lr-AdamW/pg1‚3.347807024441262e-05 & lr-AdamW/pg2‚3.347807024441262e-05 trainer/global_step‚1299 % _timestamp‚1.7858062686925123e+09dânor0+ _wandbruntime‚546  _runtime‚546èíz2³'‘­O IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23464917893120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22321317551104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 546.988951692 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22375101078528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23549902804480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22321334590464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22374836 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23464813424128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22321334590464 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8851760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5120495 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1143583302656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140782730752 % _timestamp‚1.7858062686935685e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23464813424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1174801725952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143583302656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640477 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22907568463584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13724701 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22321317551104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13724701 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1174671051776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21766785732832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8852003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23464917893120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5120497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22907465621100 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392¨]Åð·'’±O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22365178 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21766768693472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23549747226624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640479 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚23464917893120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22321317551104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 546.988951692 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22375101078528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23549902804480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚101 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22321334590464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22374836 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23464813424128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22321334590464 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8851760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5120495 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1143583302656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1140782730752 % _timestamp‚1.7858062686935685e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23464813424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1174801725952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1143583302656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640477 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22907568463584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13724701 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22321317551104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13724701 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1174671051776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21766785732832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8852003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23464917893120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5120497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22907465621100 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1144462929408 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22338501353472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3734383 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13735224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5124432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22382344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13735224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1141574400850 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13744894 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8647122 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22392016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21783508865792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858062690888917e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22382344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22338484314112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23567856858624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22392016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647120 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23568014526464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22925083266642 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1175706331136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21783525905152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23482859806720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1144375492608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23482964282880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22392283291648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1175573566976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 trainer/global_step‚1299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8858563 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22925186109238 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1144462929408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22338484314112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141660204086 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5124434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22338501353472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 547.380981001 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22392308195328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130姗•»M A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13744894 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8647122 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22392016 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚21783508865792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858062690888917e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMÂ’§“*#onitor.on_train_batch_end/allocation.all.freed‚22382344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22338484314112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23567856858624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22392016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8647120 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23568014526464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22925083266642 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1175706331136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚21783525905152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23482859806720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1144375492608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23482964282880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚22392283291648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1175573566976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 trainer/global_step‚1299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8858563 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22925186109238 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1144462929408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22338484314112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1141660204086 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5124434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8647120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22338501353472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 547.380981001 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22392308195328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1144462929408 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22338501353472 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3734383 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13735224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5124432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22382344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13735224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1141574400850f3ãí~0–+ _wandbruntime‚547  _runtime‚5479ÚÞ;ÿ—ù epoch‚0 trainer/global_step‚1299 % _timestamp‚1.7858062690903912e+09 _runtime‚ 547.381960498 %loss/total_loss‚50.29394739849117 /loss/FRt2/assignment_loss‚4.228147664848639 1loss/SRqqt1/assignment_loss‚2.156970313684208 1loss/FBt2/assignment_loss‚0.14069473746437297 /loss/FRt2/detection_loss‚3.3623906748027217 1loss/SRqqt1/detection_loss‚1.5219298290385197 _step‚103 /loss/FBt1/detection_loss‚16.387563374350428 /loss/FRt1/assignment_loss‚5.066165753773281 1loss/SRqqt2/assignment_loss‚1.864034668298868 .loss/FRt1/detection_loss‚4.627379868711744 0loss/FBt1/assignment_loss‚0.4022456709324615 0loss/SRqqt2/detection_loss‚4.804042608462847 /loss/FBt2/detection_loss‚5.7323822341230795 (œ˜ý /loss/FRt2/detection_loss‚3.3623906748027217 1loss/SRqqt1/detection_loss‚1.5219298290385197 _step‚103 /loss/FBt1/detection_loss‚16.387563374350428 /loss/FRt1/assignment_loss‚5.066165753773281 1loss/SRqqt2/assignment_loss‚1.864034668298868 .loss/FRt1/detection_loss‚4.627379868711744 0loss/FBt1/assignment_loss‚0.4022456709324615 0loss/SRqqt2/detection_loss‚4.804042608462847 /loss/FBt2/detection_loss‚5.7323822341230795 epoch‚0 trainer/global_step‚1299 % _timestamp‚1.7858062690903912e+09 _runtime‚ 547.381960498 %loss/total_loss‚50.29394739849117 /loss/FRt2/assignment_loss‚4.228147664848639 1loss/SRqqt1/assignment_loss‚2.156970313684208 1loss/FBt2/assignment_loss‚0.14069473746437297gv»ñ›™‚(: ÄûÄÓåí‹Ó gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚153.334' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.18555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lu|š‚(:ñ ÄûÄÓ¼„µÕ disk.rbd4.in‚0 disk.dm-1.in‚0$ memory_percent‚36.74571514129639( proc.memory.availableMB‚ 41454.328125 network.sent‚ 1.201569e+06 disk.rbd5.out‚ 0.306640625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚1522.38232421875 disk.sda.in‚0 disk.dm-1.out‚ 27.984375 disk.rbd3.out‚0% disk./.usageGB‚1237.7299575805664 disk.drbd1743.out‚ 0.6640625 disk.sda.out‚ 27.984375 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd4.out‚ 0.306640625 disk.rbd2.in‚0* proc.memory.percent‚17.193293571472168 cpu‚14.060890275288529 proc.cpu.threads‚29 disk.rbd3.in‚0) disk./.usagePercent‚83.06509597783901 disk.dm-0.in‚10.5625 disk.rbd0.out‚0 disk.drbd1743.in‚0% disk.nvme0n1.out‚1522.38232421875 disk.sda1.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda3.in‚0 network.recv‚117235 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.sda3.out‚ 27.984375" proc.memory.rssMB‚ 11267.796875 disk.sda2.in‚0 disk.nvme0c0n1.in‚10.5625" disk.dm-0.out‚1522.38232421875 disk.nvme0n1.in‚10.5625¶»0›+ _wandbruntime‚569  _runtime‚569ÂS7Åœ¿ trainer/global_step‚1349 $ _timestamp‚1.785806291444281e+09 _runtime‚ 569.736984737 _step‚104 & lr-AdamW/pg1‚3.476667956867793e-05 & lr-AdamW/pg2‚3.476667956867793e-05h·¨ Éà _step‚104 & lr-AdamW/pg1‚3.476667956867793e-05 & lr-AdamW/pg2‚3.476667956867793e-05 trainer/global_step‚1349 $ _timestamp‚1.785806291444281e+09 _runtime‚ 569.736984737h›ú0ž+ _wandbruntime‚569  _runtime‚569¶­J³'Ÿ­O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5317347 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24455433311744 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14262359 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1187579438080 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8972627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23235456918528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23235432014848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1187579438080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187492008448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1219849913344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24367147709952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14252703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24367147709952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14252703 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3874199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23788362700608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1219976393216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3874452 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5317345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14262359 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367252178944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23225330 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23179672740864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚105 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8972629 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184671194260 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 569.737116545 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8972629 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22603777309472 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9191799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23179672740864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184585391136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23234988 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23179655701504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24455281928192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1184585391136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23234988 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23179655701504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24455281928192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1187579438080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1187492008448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1219849913344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24367147709952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14252703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24367147709952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14252703 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3874199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23788362700608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1219976393216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3874452 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5317345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14262359 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24367252178944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23225330 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23179672740864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚105 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8972629 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1184671194260 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 569.737116545 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocÌòñßated‚8972629 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22603777309472 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9191799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23179672740864iÊÔÔœ¡‚(:‘ ÓûÄÓ”€…Ó gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚123.266' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.48111111111112 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cOZÉw¢‚(:ì ÓûÄÓëðÙÕ disk.sda1.out‚0 disk.rbd2.out‚ 0.306640625$ memory_percent‚36.74578666687012 disk.drbd1743.in‚0 disk.sda.out‚29.09375 disk.dm-1.in‚0' proc.memory.availableMB‚ 41454.28125 disk.dm-0.in‚10.5625 disk.nvme0c0n1.in‚10.5625 disk.sda3.in‚0 network.recv‚119340 disk.rbd1.in‚0" disk.dm-0.out‚1522.71240234375 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 0.306640625 disk.drbd1743.out‚ 0.70703125 disk.rbd3.out‚0 disk.sda1.in‚0 network.sent‚ 1.246369e+06 disk.nvme0n1.in‚10.5625 disk.rbd4.out‚ 0.306640625 disk.sda3.out‚29.09375$ disk./.usageGB‚1237.839126586914 disk.sda2.in‚0 disk.rbd2.in‚0* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.rbd5.out‚ 0.306640625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚1522.71240234375 disk.dm-1.out‚29.09375" proc.memory.rssMB‚ 11267.796875 cpu‚14.05801629844869 disk.rbd5.in‚0 disk.rbd4.in‚0) disk./.usagePercent‚83.07242240145389 disk.sda2.out‚0% disk.nvme0n1.out‚1522.71240234375]8±Z0£+ _wandbruntime‚570  _runtime‚570‹jн&¤·M E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚ 570.126876734 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23242496 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8979272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23252639131648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5321284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1185462864358 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9198248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚106 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23196839503872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188371628032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3877208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8979270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220748979712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23252664035328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14263226 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23252168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24385298568704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22620534521152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23196822464512 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24473549973504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23806083188746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23252168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14263226 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188459064832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8979272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23196822464512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3876966 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1220885938176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24385298568704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23242496 trainer/global_step‚1349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23196839503872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188459064832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8979270 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14272896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23196822464512 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24473549973504 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23806083188746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23252168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14263226 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1188459064832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8979272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23196822464512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3876966 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1220885938176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24385298568704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23242496 trainer/global_step‚1349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23196839503872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1188459064832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8979270 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14272896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9198248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚106 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23196839503872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1188371628032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3877208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8979270 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1220748979712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23252664035328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14263226 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23252168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24385298568704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22620534521152 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200jí]M0¦+ _wandbruntime‚570  _runtime‚570_uw ý§÷ trainer/global_step‚1349 % _timestamp‚1.7858062918361547e+09 /loss/FBt1/detection_loss‚23.587310629903964 _runtime‚ 570.127651667 0loss/FBt1/assignment_loss‚0.5465632898184205 0loss/SRqqt2/detection_loss‚4.649286589387692 _step‚107 /loss/FRt2/assignment_loss‚4.018540759475863 1loss/SRqqt2/assignment_loss‚1.822782797343803 1loss/FBt2/assignment_loss‚0.10920518049932716 -loss/FRt2/detection_loss‚3.28372650639135 epoch‚0 /loss/FRt1/assignment_loss‚4.833100780844688 2loss/SRqqt1/assignment_loss‚1.9700723338301165 .loss/FRt1/detection_loss‚4.946198220131919 0loss/SRqqt1/detection_loss‚1.321010995434468 .loss/FBt2/detection_loss‚6.586075328845783 &loss/total_loss‚57.673873411907394ØÙ‚ö¨û /loss/FRt2/assignment_loss‚4.018540759475863 1loss/SRqqt2/assignment_loss‚1.822782797343803 1loss/FBt2/assignment_loss‚0.10920518049932716 -loss/FRt2/detection_loss‚3.28372650639135 epoch‚0 /loss/FRt1/assignment_loss‚4.833100780844688 2loss/SRqqt1/assignment_loss‚1.9700723338301165 .loss/FRt1/detection_loss‚4.946198220131919 0loss/SRqqt1/detection_loss‚1.321010995434468 .loss/FBt2/detection_loss‚6.586075328845783 &loss/total_loss‚57.673873411907394 trainer/global_step‚1349 % _timestamp‚1.7858062918361547e+09 /loss/FBt1/detection_loss‚23.587310629903964 _runtime‚ 570.127651667 0loss/FBt1/assignment_loss‚0.5465632898184205 0loss/SRqqt2/detection_loss‚4.649286589387692 _step‚107kÀ„jNœ©‚(:‘ âûÄÓôÿäÒ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚155.758' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.53222222222223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÔS„~ª‚(:ó âûÄÓ²¥ìÕ network.recv‚121158 disk.nvme0c0n1.in‚10.5625 disk.sda.in‚0 disk.sda2.in‚0( disk./.usagePercent‚83.1403004950899 disk.dm-1.out‚ 29.6796875 disk.rbd5.in‚0 disk.rbd4.out‚ 0.306640625' disk.nvme0c0n1.out‚2669.75537109375 disk.rbd3.out‚0 disk.rbd5.out‚ 0.306640625' proc.memory.availableMB‚ 41454.03125" proc.memory.rssMB‚ 11267.796875 proc.cpu.threads‚29 disk.rbd2.out‚ 0.306640625 disk.sda1.out‚0 disk.rbd4.in‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚10.5625" disk.dm-0.out‚2669.75537109375 disk.rbd0.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd1.in‚0 disk.drbd1743.out‚ 0.71484375 disk.nvme0n1.in‚10.5625$ memory_percent‚36.74616813659668 disk.dm-1.in‚0 disk.sda.out‚ 29.6796875 disk.rbd2.in‚0 disk.sda3.out‚ 29.6796875 disk.rbd1.out‚ 0.306640625 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.280455e+06% disk./.usageGB‚1238.8505592346191% disk.nvme0n1.out‚2669.75537109375 cpu‚14.057102371289021й€0«+ _wandbruntime‚591  _runtime‚591þ¿ïÑŬ¿ trainer/global_step‚1399 $ _timestamp‚1.785806313072744e+09 _runtime‚ 591.364463591 _step‚108 & lr-AdamW/pg1‚3.605528889294323e-05 & lr-AdamW/pg2‚3.605528889294323e-05m <–É­Ã & lr-AdamW/pg1‚3.605528889294323e-05 & lr-AdamW/pg2‚3.605528889294323e-05 trainer/global_step‚1399 $ _timestamp‚1.785806313072744e+09 _runtime‚ 591.364463591 _step‚108l`™é—0®+ _wandbruntime‚591  _runtime‚591;Paóz¯¬O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25269481995776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14790361 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264996717568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24669362622600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24085482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9304779 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24038010891264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24095812758528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25269586464768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14780705 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4016328 $ _timestamp‚1.785806313073817e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25269481995776 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231575573504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1231488143872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24095140 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1265133683200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9304777 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5514195 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9304779 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24669259780116 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9530796 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25360784572416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚1399 _step‚109 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24037993851904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1231488143872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1231575573504 YKDeviceStatsMonitUŒ]8 or.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23440785925472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9304777 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14790361 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25269586464768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9530523 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24095787854848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25269481995776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14790361 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264996717568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24669362622600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24085482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9304779 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24038010891264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24095812758528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25269586464768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14780705 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4016328 $ _timestamp‚1.785806313073817e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25269481995776 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1231575573504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1231488143872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24095140 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1265133683200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9304777 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5514195 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9304779 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24669259780116 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9530796 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25360784572416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚1399 _step‚109 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24037993851904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1231488143872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1231575573504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23440785925472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9304777 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14790361 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25269586464768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9530523 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24095787854848 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24112320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25378894867968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24102648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25287632854528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1232455200256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24102648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1232367763456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14800898 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25379054632960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1399 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9537270 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25287632854528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24686877425658 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9311422 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24112994971648 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9311420 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1266034757632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14791228 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24055160614912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1232367763456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858063134365005e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 591.728628117 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4019138 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392Zº¿&³»M G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24112320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24113019875328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518134 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14800898 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9537530 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _runtime‚ 591.728628117 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4019138 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1229351327866 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24055177654272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23457543137152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24112320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25378894867968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24102648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25287632854528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1232455200256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24102648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1232367763456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14800898 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25379054632960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1399 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9537270 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25287632854528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24686877425658 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9311422 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24112994971648 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9311420 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1266034757632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14791228 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24055160614912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1232367763456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858063134365005e+09 EB ±'½«O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24896349041664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24277794541472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275571708928 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5711045 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25550259702108 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 613.586153452 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26171816281600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4159147 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚113 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15308707 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 # _timestamp‚1.78580633529424e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26266453268992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9636929 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24956168598528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26171920750592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24896332002304 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9869933 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1310284670464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26171920750592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275571708928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24896349041664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9636927 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24277811580832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275484279296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24956143694848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26171816281600 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9636929 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15318363 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24955292 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15318363 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4158888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25550156859624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392‰;Êöµ'¾¯O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1272362318152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5711045 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25550259702108 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 613.586153452 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26171816281600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4159147 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚113 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15308707 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 # _timestamp‚1.78580633529424e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26266453268992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9636929 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24956168598528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26171920750592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24896332002304 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9869933 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1310284670464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26171920750592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1275571708928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24896349041664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9636927 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24277811580832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275484279296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24956143694848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26171816281600 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9636929 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15318363 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24955292 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15318363 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4158888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚25550156859624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24896349041664 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24277794541472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275571708928 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9643572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1273325594610 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26189967140352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15328900 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5714984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26284560190464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15319230 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24973350811648 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9876899 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 # _timestamp‚1.78580633566582e+09 _step‚114 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5714982 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26189967140352 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9643572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24972472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25567877347762 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9876664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24913515804672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26284406716928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9643570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24973375715328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4161682 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24913498765312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1276451335680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1449 _runtime‚ 613.957843154 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9643570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24913498765312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1276451335680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_b¦„ݶ¶ytes.large_pool.peak‚ 2645753856 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4161915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25567774505166 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24972472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15319230 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15328900 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160’’Ù¿&Á¹M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4161915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25567774505166 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24972472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15319230 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15328900 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1273239791374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24913515804672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9643572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1273325594610 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26189967140352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15328900 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5714984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26284560190464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15319230 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24973350811648 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9876899 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 # _timestamp‚1.78580633566582e+09 _step‚114 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5714982 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26189967140352 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9643572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24972472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25567877347762 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9876664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24913515804672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26284406716928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9643570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24973375715328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4161682 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24913498765312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1276451335680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1449 _runtime‚ 613.957843154 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9643570 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24913498765312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1276451335680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251r¼kd0Â+ _wandbruntime‚613  _runtime‚613r ¬hÃú /loss/FRt1/assignment_loss‚4.774363463265555 /loss/FRt2/detection_loss‚3.5915499908870525 %loss/total_loss‚65.59651672607819 epoch‚0 0loss/FRt2/assignment_loss‚3.9177162781190336 0loss/SRqqt1/detection_loss‚2.208391158960326 0loss/SRqqt2/detection_loss‚4.752550724479887 % _timestamp‚1.7858063356671994e+09 _runtime‚ 613.958732201 1loss/SRqqt2/assignment_loss‚1.976255351825366 .loss/FRt1/detection_loss‚4.509336867289884 2loss/SRqqt1/assignment_loss‚1.8918947461247444 1loss/FBt1/assignment_loss‚0.49030991677427666 1loss/FBt2/assignment_loss‚0.15518031865538998 /loss/FBt1/detection_loss‚31.459901341241295 .loss/FBt2/detection_loss‚5.869066568455395 trainer/global_step‚1449 _step‚115 ÒnÅÄþ _step‚115 /loss/FRt1/assignment_loss‚4.774363463265555 /loss/FRt2/detection_loss‚3.5915499908870525 %loss/total_loss‚65.59651672607819 epoch‚0 0loss/FRt2/assignment_loss‚3.9177162781190336 0loss/SRqqt1/detection_loss‚2.208391158960326 0loss/SRqqt2/detection_loss‚4.752550724479887 % _timestamp‚1.7858063356671994e+09 _runtime‚ 613.958732201 1loss/SRqqt2/assignment_loss‚1.976255351825366 .loss/FRt1/detection_loss‚4.509336867289884 2loss/SRqqt1/assignment_loss‚1.8918947461247444 1loss/FBt1/assignment_loss‚0.49030991677427666 1loss/FBt2/assignment_loss‚0.15518031865538998 /loss/FBt1/detection_loss‚31.459901341241295 .loss/FBt2/detection_loss‚5.869066568455395 trainer/global_step‚1449sw8}œÅ‚(:‘ €üÄÓÐÿƒÓ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.329' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚63.51611111111111 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úò0zÆ‚(:ï €üÄÓ±ðÞÕ% disk./.usageGB‚1238.9112510681152 disk.drbd1743.in‚0 disk.dm-0.in‚10.5625 disk.sda1.in‚0$ memory_percent‚36.74649000167847) disk./.usagePercent‚83.14437357495966 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚10.5625& disk.nvme0c0n1.out‚2733.7431640625 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd5.in‚0 disk.rbd5.out‚ 0.306640625 disk.rbd0.out‚0 network.recv‚125970 disk.rbd2.out‚ 0.306640625 disk.sda3.out‚ 30.953125 disk.sda2.in‚0! disk.dm-0.out‚2733.7431640625 disk.sda1.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 11267.796875 network.sent‚ 1.360181e+06 disk.rbd1.in‚0 disk.sda2.out‚0$ disk.nvme0n1.out‚2733.7431640625* proc.memory.percent‚17.193293571472168 cpu‚14.051377740479184 proc.cpu.threads‚29 disk.rbd4.in‚0 disk.rbd4.out‚ 0.306640625 disk.dm-1.out‚ 30.953125 disk.drbd1743.out‚ 0.7421875 disk.nvme0c0n1.in‚10.5625 disk.sda.out‚ 30.953125 disk.dm-1.in‚0) proc.memory.availableMB‚ 41453.8203125TT£YœÇ‚(:‘ üÄÓÜšÕÒ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚100.957' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚56.08722222222222 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Tû†…È‚(:ú üÄÓº³îÕ network.recv‚128056' disk.nvme0c0n1.out‚2734.92333984375 disk.dm-1.in‚0$ memory_percent‚36.74551844596863 network.sent‚ 1.404738e+06 disk.nvme0n1.in‚10.5625% disk.nvme0n1.out‚2734.92333984375 disk.rbd5.out‚ 0.306640625 disk.nvme0c0n1.in‚10.5625 disk.dm-1.out‚ 31.78515625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.rbd2.out‚ 0.306640625 disk.dm-0.in‚10.5625" disk.dm-0.out‚2734.92333984375 disk.rbd3.out‚0 disk.rbd5.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625* proc.memory.percent‚17.193293571472168) disk./.usagePercent‚83.14437485499732 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 0.306640625 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 11267.796875 disk.sda.in‚0 disk.sda3.in‚0* proc.memory.availableMB‚41454.45703125 disk.sda.out‚ 31.78515625 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.78515625 disk.rbd3.in‚0 disk.rbd0.out‚0 cpu‚14.050486741276893% disk./.usageGB‚1238.9112701416016 disk.rbd2.in‚0 disk.sda3.out‚ 31.78515625_Th0É+ _wandbruntime‚634  _runtime‚634’ÓÄÆÊÀ _runtime‚ 634.954982631 _step‚116 & lr-AdamW/pg1‚3.863250754147384e-05 & lr-AdamW/pg2‚3.863250754147384e-05 trainer/global_step‚1499 % _timestamp‚1.7858063566631873e+09ßRkúÊËÄ _runtime‚ 634.954982631 _step‚116 & lr-AdamW/pg1‚3.863250754147384e-05 & lr-AdamW/pg2‚3.863250754147384e-05 trainer/global_step‚1499 % _timestamp‚1.7858063566631873e+09tït„0Ì+ _wandbruntime‚634  _runtime‚634¹"¤‡µ'ͯO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27074255036416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4301353 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25754670152704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1355424230912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27074255036416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25816524438528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27172070955008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27171923765760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27074150567424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5907895 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15846365 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10209502 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26431053939132 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10209248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4301605 _runtime‚ 634.956313788 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15846365 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25816499534848 _step‚117 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25114820196832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9969079 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9969079 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25754670152704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15836709 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15836709 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25754687192064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858063566643932e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27074150567424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1316250781660 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319567844352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319567844352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27172070955008 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27171923765760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27074150567424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5907895 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15846365 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10209502 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26431053939132 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10209248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4301605 _runtime‚ 634.956313788 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15846365 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25816499534848 _step‚117 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25114820196832 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9969079 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9969079 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25754670152704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15836709 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15836709 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25754687192064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858063566643932e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27074150567424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1316250781660 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319567844352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319567844352 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25832624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5911832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1356459051520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7858063570320134e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9975720 trainer/global_step‚1499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10216030 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27092301426176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25771836915712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9975722 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚ 635.324090468 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9975720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320360034304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4304463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1317214058118 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27092196950016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26448671584674 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1317128254882 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25822952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15847232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15856902 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25131543329792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚118 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25771853955072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10216297 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1356319995904 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1320447471104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320447471104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25131560369152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5911834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4304198 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25832624 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25832624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5911832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1356459051520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7858063570320134e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9975720 trainer/global_step‚1499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10216030 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27092301426176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25771836915712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9975722 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚ 635.324090468 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9975720 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320360034304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4304463 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1317214058118 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚27092196950016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26448671584674 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1317128254882 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25822952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15847232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15856902 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25131543329792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚118 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25771853955072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10216297 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1356319995904 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1320447471104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320447471104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25131560369152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5911834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4304198v É0Ò+ _wandbruntime‚635  _runtime‚635eÏúÓô .loss/FBt1/detection_loss‚15.56875218263938 %loss/total_loss‚46.79710721075761 /loss/FRt2/assignment_loss‚3.862290625669518 2loss/SRqqt2/assignment_loss‚1.9622204239304002 .loss/FBt2/detection_loss‚4.897366724088571 $ _timestamp‚1.785806357033368e+09 /loss/FRt1/assignment_loss‚4.900933527284199 -loss/FRt2/detection_loss‚3.28184823296508 trainer/global_step‚1499 _step‚119 0loss/FBt2/assignment_loss‚0.1498713831904517 -loss/FRt1/detection_loss‚4.34639081151949 1loss/SRqqt1/detection_loss‚1.6352892560955297 0loss/SRqqt2/detection_loss‚4.053074274111438 epoch‚0 _runtime‚ 635.324855933 2loss/SRqqt1/assignment_loss‚1.7989269787614997 0loss/FBt1/assignment_loss‚0.3401427905020468”|–þÔø /loss/FRt1/assignment_loss‚4.900933527284199 -loss/FRt2/detection_loss‚3.28184823296508 trainer/global_step‚1499 _step‚119 0loss/FBt2/assignment_loss‚0.1498713831904517 -loss/FRt1/detection_loss‚4.34639081151949 1loss/SRqqt1/detection_loss‚1.6352892560955297 0loss/SRqqt2/detection_loss‚4.053074274111438 epoch‚0 _runtime‚ 635.324855933 2loss/SRqqt1/assignment_loss‚1.7989269787614997 0loss/FBt1/assignment_loss‚0.3401427905020468 .loss/FBt1/detection_loss‚15.56875218263938 %loss/total_loss‚46.79710721075761 /loss/FRt2/assignment_loss‚3.862290625669518 2loss/SRqqt2/assignment_loss‚1.9622204239304002 .loss/FBt2/detection_loss‚4.897366724088571 $ _timestamp‚1.785806357033368e+09wCì·_›Õ‚(: žüÄÓ´ÕÈÒ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚113.93' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚63.29444444444445 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦=À²mÖ‚(:â žüÄÓï‹ Ö disk.sda2.in‚0" disk.dm-0.out‚2735.14990234375 disk.rbd3.out‚0% disk.nvme0n1.out‚2735.14990234375 disk.rbd0.out‚0 disk.rbd4.out‚ 0.306640625 disk.sda.out‚32.375 proc.cpu.threads‚29 network.sent‚ 1.40743e+06 disk.rbd5.out‚ 0.306640625 disk.nvme0c0n1.in‚10.5625 disk.rbd2.out‚ 0.306640625 disk.dm-1.in‚0 disk.dm-1.out‚32.375 disk.rbd1.in‚0& proc.memory.availableMB‚ 41454.1875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda3.out‚32.375$ memory_percent‚36.74592971801758' disk./.usagePercent‚83.148537025489 disk.sda1.in‚0 disk.rbd5.in‚0' disk.nvme0c0n1.out‚2735.14990234375 disk.sda.in‚0" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168 cpu‚14.04827692157769$ disk./.usageGB‚1238.973289489746 disk.drbd1743.out‚ 0.78515625 disk.sda1.out‚0 disk.rbd1.out‚ 0.306640625 network.recv‚129552 disk.dm-0.in‚10.5625 disk.nvme0n1.in‚10.5625 disk.rbd4.in‚0¿z&–0×+ _wandbruntime‚657  _runtime‚657Þâ½ÈØ % _timestamp‚1.7858063789048007e+09 _runtime‚ 657.196501966 _step‚120 ' lr-AdamW/pg1‚3.9921116865739145e-05 ' lr-AdamW/pg2‚3.9921116865739145e-05 trainer/global_step‚1549G#_áÌÙÆ _step‚120 ' lr-AdamW/pg1‚3.9921116865739145e-05 ' lr-AdamW/pg2‚3.9921116865739145e-05 trainer/global_step‚1549 % _timestamp‚1.7858063789048007e+09 _runtime‚ 657.196501966x¥ à|0Ú+ _wandbruntime‚657  _runtime‚657î´Á¦!Û±O KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1400582685696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25951811773472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1363563979776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25951828812832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16364711 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4444759 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26675596 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26613025342464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10549758 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549504 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28077597832704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1400717554176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10301227 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26613025342464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26613008303104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26676855374848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26613008303104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1360139245168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10301229 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16374367 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10301229 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6104745 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27976484853248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1363563979776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16364711 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1363476550144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28077438060544 _step‚121 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26676880278528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27976589322240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1363476550144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27311951018640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392Ò\Ñf»'ܵO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4445011 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚1549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _runtime‚ 657.19817308 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16374367 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26676880278528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27976589322240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1363476550144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27311951018640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1400582685696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25951811773472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1363563979776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25951828812832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16364711 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4444759 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26675596 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26613025342464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10549758 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10549504 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28077597832704 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1400717554176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10301227 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26613025342464 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚26613008303104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26676855374848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26613008303104 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1360139245168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10301229 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16374367 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10301229 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6104745 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27976484853248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1363563979776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16364711 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1363476550144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28077438060544 _step‚121 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26630175066112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1364356169728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10307872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26630192105472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26694062491648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16384904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4447793 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26630175066112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26692776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10556477 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1364443606528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10307870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10556207 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 $ _timestamp‚1.785806379279152e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _step‚122 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27994635712000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26694087395328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16375234 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27329568664182 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1364356169728 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401618677760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361102521626 trainer/global_step‚1549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6108682 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4447525 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032ÁàšßÀM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25968551945792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1361016718390 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26630175066112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1364356169728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10307872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26630192105472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26694062491648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16384904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4447793 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26630175066112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26692776 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10556477 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1364443606528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10307870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10556207 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 $ _timestamp‚1.785806379279152e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _step‚122 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚27994635712000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26694087395328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16375234 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27329568664182 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1364356169728 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401618677760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allo:|_{, cated‚ 1361102521626 trainer/global_step‚1549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6108682 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4447525 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26683104 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6301595 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407560115200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27471363492864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28192950940632 _step‚125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445907179520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27526090 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10633377 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16902369 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26788837428832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27471346453504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27535748 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10633379 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16892713 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1407560115200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28878923608064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27537211214848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1407472685568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1404113511800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16902369 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4588407 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28192848098148 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28983143298048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚1599 _runtime‚ 678.867945409 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28878819139072 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28878819139072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27471346453504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4588649 ó(ϼ'ì¶O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27471346453504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4588649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6301597 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28878923608064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6301595 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407560115200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27471363492864 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28192950940632 _step‚125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1445907179520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27526090 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10633377 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16902369 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26788837428832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27471346453504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27535748 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10633379 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16892713 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1407560115200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28878923608064 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27537211214848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1407472685568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1404113511800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16902369 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4588407 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28192848098148 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28983143298048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚1599 _runtime‚ 678.867945409 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28878819139072 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28878819139072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680}ï­Q0í+ _wandbruntime‚679  _runtime‚679ïÊG´]î½M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29001253144064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27554418331648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27552928 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10640022 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404990985134 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16903236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1408352305152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1408352305152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28210568586286 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26805560561792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858064009406362e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27488513216512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _step‚126 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytesEÆoOf.large_pool.allocated‚26805577601152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404905181898 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10896792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10640020 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1446679241728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27554443235328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28210465743690 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27543256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27543256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10640020 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10897038 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚1599 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28896969997824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408439741952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6305534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6305532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29001097573376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10640022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27488530255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27552928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16912906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ßÊSÇ&ïÁM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1408352305152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28210568586286 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚26805560561792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858064009406362e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27488513216512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _step‚126 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚26805577601152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1404905181898 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10896792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10640020 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1446679241728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27554443235328 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28210465743690 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27543256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27543256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10640020 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10897038 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚1599 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28896969997824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408439741952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6305534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6305532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29001097573376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10640022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27488530255872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27552928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16912906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29001253144064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27554418331648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27552928 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10640022 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1404990985134 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16903236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1408352305152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320~p1Nš0ð+ _wandbruntime‚679  _runtime‚679¢}jîñú 1loss/FBt1/assignment_loss‚0.35152238945738645 _runtime‚ 679.233554917 0loss/FRt2/assignment_loss‚3.7805189111016015 2loss/SRqqt1/assignment_loss‚1.9079098304112752 /loss/FRt2/detection_loss‚3.6272104294462637 /loss/FBt1/detection_loss‚12.956320889239253 _step‚127 .loss/FRt1/assignment_loss‚4.51037043424753 0loss/FBt2/assignment_loss‚0.0604517753743234 1loss/SRqqt1/detection_loss‚2.4159638713773854 1loss/SRqqt2/detection_loss‚3.7675178752917993 .loss/FBt2/detection_loss‚3.911114406502176 epoch‚0 2loss/SRqqt2/assignment_loss‚2.0751395225524902 -loss/FRt1/detection_loss‚4.63468001140998 %loss/total_loss‚43.99872034641146 trainer/global_step‚1599 % _timestamp‚1.7858064009420621e+09tšŸ"òþ 1loss/FBt1/assignment_loss‚0.35152238945738645 _runtime‚ 679.233554917 0loss/FRt2/assignment_loss‚3.7805189111016015 2loss/SRqqt1/assignment_loss‚1.9079098304112752 /loss/FRt2/detection_loss‚3.6272104294462637 /loss/FBt1/detection_loss‚12.956320889239253 _step‚127 .loss/FRt1/assignment_loss‚4.51037043424753 0loss/FBt2/assignment_loss‚0.0604517753743234 1loss/SRqqt1/detection_loss‚2.4159638713773854 1loss/SRqqt2/detection_loss‚3.7675178752917993 .loss/FBt2/detection_loss‚3.911114406502176 epoch‚0 2loss/SRqqt2/assignment_loss‚2.0751395225524902 -loss/FRt1/detection_loss‚4.63468001140998 %loss/total_loss‚43.99872034641146 trainer/global_step‚1599 % _timestamp‚1.7858064009420621e+09Ç“‡›ó‚(: ËüÄÓ¢ûÒ gpu.0.gpu‚25.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚135.934' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚75.5188888888889 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RS»¿‚ô‚(:÷ ËüÄÓ¨ÝÖ) disk./.usagePercent‚83.15281055925809% disk./.usageGB‚1239.0369682312012 disk.rbd0.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda3.in‚0 disk.drbd1743.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 11267.796875 disk.rbd2.in‚0 disk.drbd1743.out‚0.84375 disk.rbd3.out‚0 network.sent‚ 1.499398e+06 disk.rbd5.out‚ 0.306640625 disk.rbd4.in‚0 disk.rbd4.out‚ 0.306640625 disk.nvme0c0n1.in‚10.5625 disk.sda.in‚0 disk.sda.out‚ 34.44921875 disk.dm-1.out‚ 34.44921875 disk.nvme0n1.in‚10.5625 disk.dm-0.in‚10.5625" disk.dm-0.out‚2798.97900390625% memory_percent‚36.746108531951904 cpu‚14.041717858372929 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚2798.97900390625 disk.sda3.out‚ 34.44921875 disk.rbd1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 41454.0703125 proc.cpu.threads‚29% disk.nvme0n1.out‚2798.97900390625 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda1.in‚0 disk.rbd5.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.193293571472168 network.recv‚134710Dí 0õ+ _wandbruntime‚700  _runtime‚700kšÈàÈö ' lr-AdamW/pg2‚4.2498335514269754e-05 trainer/global_step‚1649 % _timestamp‚1.7858064224886017e+09 _runtime‚ 700.780298076 _step‚128 ' lr-AdamW/pg1‚4.2498335514269754e-05l0k3Í÷Ç ' lr-AdamW/pg1‚4.2498335514269754e-05 ' lr-AdamW/pg2‚4.2498335514269754e-05 trainer/global_step‚1649 % _timestamp‚1.7858064224886017e+09 _runtime‚ 700.780298076 _step‚128€^šæð0ø+ _wandbruntime‚700  _runtime‚700×ãà¹'ù³O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17420715 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1451556250624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28386242 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27625846044832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29781153424896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10965529 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10965527 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29073745177656 _step‚129 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28329701643264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28395900 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17430371 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29781153424896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17420715 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28397567054848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29781257893888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1448001975308 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28329684603904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1490925645824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1451468820992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6498447 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11229996 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29073848020140 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4731804 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11230251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 700.784950602¬2Qy ú¸O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28329684603904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1490925645824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1451468820992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6498447 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11229996 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29073848020140 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4731804 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11230251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 700.784950602 L?DeviceStatsMonitor.on_train_batch_start©¾RµE/active.small_pool.freed‚17420715 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1451556250624 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚1649 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28386242 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27625846044832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29781153424896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10965529 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10965527 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29073745177656 _step‚129 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28329701643264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28395900 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17430371 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29781153424896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17420715 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28397567054848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29781257893888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1448001975308 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121?7ˆë0û+ _wandbruntime‚701  _runtime‚701Á÷‘ÅÃ&ü½M ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28346868406272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28403408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27642569177792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1491961990144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10972170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1452348440576 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29906599203328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10972170 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17431238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6502384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1448793645406 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28346851366912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4734567 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11236714 % _timestamp‚1.7858064228692224e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17440908 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28413080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28403408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28413080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11236951 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28414774171648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1491825031680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29091362823198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1448879448642 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29906761065472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29799304283648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17440908 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452435877376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1452435877376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10972172 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28414799075328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28346868406272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 701.161360157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚1649 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29799304283648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6502382 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0W4kÈ&ýÂM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10972172 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28414799075328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚28346868406272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 701.161360157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚1649 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29799304283648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6502382 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28346868406272 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28403408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27642569177792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1491961990144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10972170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1452348440576 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29906599203328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10972170 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17431238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6502384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1448793645406 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28346851366912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4734567 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11236714 % _timestamp‚1.7858064228692224e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17440908 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28413080 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28403408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28413080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11236951 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28414774171648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1491825031680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29091362823198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1448879448642 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29906761065472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29799304283648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17440908 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452435877376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1452435877376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130‚dü0þ+ _wandbruntime‚701  _runtime‚701;ª_ÿú 2loss/SRqqt1/assignment_loss‚1.9051237047469916 /loss/FRt2/detection_loss‚3.3196992715622518 epoch‚0 _step‚131 /loss/FRt1/assignment_loss‚4.314866795468686 0loss/FRt2/assignment_loss‚3.8941989858099757 1loss/SRqqt1/detection_loss‚1.4931583201136998 .loss/FBt2/detection_loss‚8.748186203971345 %loss/total_loss‚49.41105501985726 _runtime‚ 701.16237499 0loss/FBt1/assignment_loss‚0.8194091043357427 1loss/FBt2/assignment_loss‚0.18880767218700686 1loss/SRqqt2/detection_loss‚3.7448924485739177 2loss/SRqqt2/assignment_loss‚1.7863504271228592 .loss/FRt1/detection_loss‚4.510852660677993 /loss/FBt1/detection_loss‚14.685509425286789 trainer/global_step‚1649 $ _timestamp‚1.785806422870833e+09ViY³€ÿ 1loss/SRqqt2/detection_loss‚3.7448924485739177 2loss/SRqqt2/assignment_loss‚1.7863504271228592 .loss/FRt1/detection_loss‚4.510852660677993 /loss/FBt1/detection_loss‚14.685509425286789 trainer/global_step‚1649 $ _timestamp‚1.785806422870833e+09 2loss/SRqqt1/assignment_loss‚1.9051237047469916 /loss/FRt2/detection_loss‚3.3196992715622518 epoch‚0 _step‚131 /loss/FRt1/assignment_loss‚4.314866795468686 0loss/FRt2/assignment_loss‚3.8941989858099757 1loss/SRqqt1/detection_loss‚1.4931583201136998 .loss/FBt2/detection_loss‚8.748186203971345 %loss/total_loss‚49.41105501985726 _runtime‚ 701.16237499 0loss/FBt1/assignment_loss‚0.8194091043357427 1loss/FBt2/assignment_loss‚0.18880767218700686ƒ¦l¼_›‚(: ÚüÄÓ¥ýáÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚127.199' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚70.6661111111111 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‹Ü}‚‚(:ò ÚüÄÓûÆÖ disk.sda1.in‚0' disk.nvme0c0n1.out‚2861.56494140625 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.dm-0.in‚10.5625) disk./.usagePercent‚83.15278879861768 disk.drbd1743.out‚0.84375 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 0.306640625 disk.sda.out‚ 35.1015625" proc.memory.rssMB‚ 11267.796875 disk.rbd2.out‚ 0.306640625 disk.dm-1.out‚ 35.1015625" disk.dm-0.out‚2861.56494140625% disk.nvme0n1.out‚2861.56494140625 disk.sda3.out‚ 35.1015625* proc.memory.percent‚17.193293571472168 disk.rbd4.in‚0 network.sent‚ 1.544425e+06 disk.nvme0n1.in‚10.5625 disk.rbd5.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 41454.234375 cpu‚14.039061269621772 disk.sda1.out‚0 disk.rbd5.out‚ 0.306640625 disk.sda3.in‚0$ memory_percent‚36.74585819244385% disk./.usageGB‚1239.0366439819336 disk.rbd3.out‚0 disk.nvme0c0n1.in‚10.5625 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚29 network.recv‚136878¥Pd*ƒ‚(:… éüÄÓˆ—¿Ò gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚124.866' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚69.37 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í…‹Á„„‚(:ù éüÄÓ”ïãÔ disk.sda3.out‚ 35.80078125" disk.dm-0.out‚2862.03466796875$ memory_percent‚36.74591779708862" proc.memory.rssMB‚ 11267.796875 cpu‚14.037339550222276 network.sent‚ 1.589016e+06 disk.sda1.in‚0 disk.rbd5.out‚ 0.306640625' disk.nvme0c0n1.out‚2862.03466796875 disk.rbd2.in‚0 disk.dm-1.out‚ 35.80078125 disk.sda2.in‚0% disk.nvme0n1.out‚2862.03466796875 disk.rbd4.in‚0* proc.memory.percent‚17.193293571472168% disk./.usageGB‚1239.0367164611816 disk.drbd1743.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.drbd1743.out‚ 0.87890625 disk.dm-0.in‚10.5625 disk.nvme0n1.in‚10.5625 disk.rbd0.in‚0 disk.rbd4.out‚ 0.306640625 disk.nvme0c0n1.in‚10.5625 network.recv‚138849 disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 35.80078125) proc.memory.availableMB‚ 41454.1953125) disk./.usagePercent‚83.15279366276083 disk.rbd2.out‚ 0.306640625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd5.in‚0 proc.cpu.threads‚29ImÀ0…+ _wandbruntime‚722  _runtime‚722'e™:ƆÀ & lr-AdamW/pg1‚4.378694483853506e-05 & lr-AdamW/pg2‚4.378694483853506e-05 trainer/global_step‚1699 % _timestamp‚1.7858064445552444e+09 _runtime‚ 722.848006585 _step‚1328>iðˇŠ& lr-AdamW/pg1‚4.378694483853506e-05 & lr-AdamW/pg2‚4.378694483853506e-05 trainer/global_step‚1699 % _timestamp‚1.7858064445552444e+09 _runtime‚ 722.848006585 _step‚132„Sõ0ˆ+ _wandbruntime‚722  _runtime‚722tÆw>‰±O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 trainer/global_step‚1699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29954642257164 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDevicª=µy$eStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11297679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1491890438816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30683592179712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11297679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17958373 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28462854660832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29256052 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6695295 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1495464956416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30683487710720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1491804635692 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17948717 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1495552386048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17958373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11569144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30794193125376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29954745099648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28462837621472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29246394 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29188039793664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17948717 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29257947798528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30683592179712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1536114652672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29188039793664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11569380 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1536245326848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858064445559916e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1495552386048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1495464956416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30683487710720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1491804635692 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17948717 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1495552386048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17958373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11569144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30794193125376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29954745099648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28462837621472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29246394 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29188039793664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17948717 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29257947798528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30683592179712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1536114652672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29188039793664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11569380 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1536245326848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858064445559916e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1495552386048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29954642257164 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11297679 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0…2·hî0‹+ _wandbruntime‚723  _runtime‚723„'€Ã&Œ½M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30701638569472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29972362745302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1492682108914 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11304320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29263560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30812318216704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575939 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4876707 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6699234 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858064449247935e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1496432012800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28479594833152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17968910 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1496344576000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30701638569472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30812158451712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1496344576000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29275130011648 _step‚134 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17959240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1537028440064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29205206556672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29205189517312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17968910 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11576185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29273232 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11304322 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1492767912150 trainer/global_step‚1699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1537163301376 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚ 723.216842195 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29205206556672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11304322 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11304320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29273232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4876951 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29263560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28479577793792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30701534093312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1496432012800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29275154915328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17959240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29972259902706 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360j½D µ ÂM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6699234 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858064449247935e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1496432012800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28479594833152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17968910 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1496344576000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚30701638569472 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30812158451712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1496344576000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29275130011648 _step‚134 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17959240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1537028440064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29205206556672 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29205189517312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17968910 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11576185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsM`[߉onitor.on_train_batch_end/active.all.allocated‚29273232 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11304322 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1492767912150 trainer/global_step‚1699 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1537163301376 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚ 723.216842195 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29205206556672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11304322 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11304320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29273232 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4876951 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29263560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28479577793792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30701534093312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1496432012800 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚29275154915328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17959240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29972259902706 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30701638569472 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29972362745302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1492682108914 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11304320 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29263560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30812318216704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575939 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4876707 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596†‚—„ù0Ž+ _wandbruntime‚723  _runtime‚723€çÞÅý÷ 1loss/SRqqt2/assignment_loss‚1.958546165057591 /loss/FRt2/assignment_loss‚3.609311954949492 1loss/FBt2/assignment_loss‚0.06622814706780547 0loss/FBt1/assignment_loss‚0.5404458704556768 -loss/FRt1/detection_loss‚4.71942623598235 1loss/SRqqt1/detection_loss‚1.4069970875398454 0loss/SRqqt2/detection_loss‚4.039438611056124 %loss/total_loss‚41.31583676330547 trainer/global_step‚1699 % _timestamp‚1.7858064449260414e+09 _runtime‚ 723.217532339 /loss/FRt1/assignment_loss‚4.495248609118992 /loss/FRt2/detection_loss‚3.3653052058271182 /loss/FBt1/detection_loss‚12.448503176403532 .loss/FBt2/detection_loss‚3.231903626221314 epoch‚0 _step‚135 1loss/SRqqt1/assignment_loss‚1.434482073625632eàT ü /loss/FRt2/assignment_loss‚3.609311954949492 1loss/FBt2/assignment_loss‚0.06622814706780547 0loss/FBt1/assignment_loss‚0.5404458704556768 -loss/FRt1/detection_loss‚4.71942623598235 1loss/SRqqt1/detection_loss‚1.4069970875398454 0loss/SRqqt2/detection_loss‚4.039438611056124 %loss/total_loss‚41.31583676330547 trainer/global_step‚1699 % _timestamp‚1.7858064449260414e+09 _runtime‚ 723.217532339 /loss/FRt1/assignment_loss‚4.495248609118992 /loss/FRt2/detection_loss‚3.3653052058271182 /loss/FBt1/detection_loss‚12.448503176403532 .loss/FBt2/detection_loss‚3.231903626221314 epoch‚0 _step‚135 1loss/SRqqt1/assignment_loss‚1.434482073625632 1loss/SRqqt2/assignment_loss‚1.958546165057591‡„uÞ š‘‚(: øüÄÓí›òÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚134.2' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚74.55555555555556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'[çû†’‚(:û øüÄÓ˜û¤Ö disk.sda3.in‚0 disk.dm-1.out‚ 36.44140625 disk.rbd3.in‚0* proc.memory.availableMB‚41453.70703125* proc.memory.percent‚17.193293571472168 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚2862.24951171875% memory_percent‚36.746662855148315 disk.nvme0n1.in‚10.5625 disk.sda1.out‚0 disk.drbd1743.out‚ 0.87890625 cpu‚14.036164557092603 disk.sda.in‚0 disk.rbd1.out‚ 0.306640625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚10.5625 disk.dm-0.in‚10.5625" proc.memory.rssMB‚ 11267.796875 proc.cpu.threads‚29) disk./.usagePercent‚83.16122091877295% disk./.usageGB‚1239.1622886657715 disk.rbd5.out‚ 0.306640625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 36.44140625 disk.drbd1743.in‚0 network.sent‚ 1.591428e+06% disk.nvme0n1.out‚2862.24951171875 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 0.306640625 disk.sda.out‚ 36.44140625 disk.rbd2.in‚0 disk.rbd3.out‚0 network.recv‚139738' disk.nvme0c0n1.out‚2862.24951171875 disk.rbd2.out‚ 0.3066406256mD40“+ _wandbruntime‚744  _runtime‚744¼²G Æ”À _step‚136 & lr-AdamW/pg1‚4.507555416280036e-05 & lr-AdamW/pg2‚4.507555416280036e-05 trainer/global_step‚1749 % _timestamp‚1.7858064662043908e+09 _runtime‚ 744.496100858÷ˆ…ÇË•Å & lr-AdamW/pg1‚4.507555416280036e-05 & lr-AdamW/pg2‚4.507555416280036e-05 trainer/global_step‚1749 % _timestamp‚1.7858064662043908e+09 _runtime‚ 744.496100858 _step‚136ˆ†M¨#0–+ _wandbruntime‚744  _runtime‚744ý,@Á¹'—³O [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1539548521472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30835642179156 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1535778902324 _step‚137 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31699760656384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30118303638528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30106546 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6892145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11629827 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30046377944064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11909257 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6892147 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539461091840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1535693099200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1581457017856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31585821996544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5016863 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30118278734848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858064662056296e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18486375 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31585926465536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30835539336672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30046360904704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11629829 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚ 744.497848325 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11629827 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30046377944064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11909257 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6892147 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1539461091840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1535693099200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1581457017856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31585821996544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5016863 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30118278734848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858064662056296e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18486375 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31585926465536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30835539336672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30046360904704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11629829 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚ 744.497848325 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1535778902324 _step‚137 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31699760656384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30118303638528‰V,œ0™+ _wandbruntime‚744  _runtime‚744&úw¤š½M ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1582220363776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30063527667712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18496912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30063544707072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30123712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5019889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@XpßDeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11636470 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540340711424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540428148224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6896084 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31717872271872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1536656375658 % _timestamp‚1.7858064665841236e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚1749 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6896082 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 744.876213963 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31603972855296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11636470 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18487242 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11915733 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30853156982214 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11636472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11636472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30853259824810 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29316603449152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30063544707072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31603972855296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30135485851648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _step‚138 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31603868379136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30133384•ðÈ&›ÂM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18487242 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11915973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11915733 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30853156982214 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11636472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11636472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30853259824810 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29316603449152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30063544707072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31603972855296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30135485851648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _step‚138 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31603868379136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30133384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1582220363776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30063527667712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18496912 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30063544707072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30123712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5019889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11636470 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540340711424 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540428148224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6896084 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31717872271872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1536656375658 % _timestamp‚1.7858064665841236e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚1749 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6896082 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 744.876213963 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31603972855296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11636470 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18487242 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7088995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1583457227264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11961979 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30966698 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19014377 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32488156282368 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30976356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1583544656896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1626514703872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12248422 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5159425 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12248172 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32488260751360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _runtime‚ 766.18198569 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32605300662272 YLDeviceStatsMonitor.on_train_batch_start/inactive_splitAؼ˜M_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032«ºÁÿ¼'¨¶O _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30904716094464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19014377 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32488156282368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32488260751360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1799 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1583457227264 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32605149278720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19004721 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785806487886241e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30976356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1583544656896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1626514703872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12248422 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5159425 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12248172 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32488260751360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _runtime‚ 766.18198569 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32605300662272 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1626641183744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30136854853472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7088995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1583457227264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11961979 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30966698 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19014377 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32488156282368 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 766.546630051 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1584336846848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30983864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12255232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30993536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32506307141120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1627414810624 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 $ _timestamp‚1.785806488254708e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1584424283648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19015244 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12254977 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1584424283648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30995866595328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5162298 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31734156904318 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30983864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7092932 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1580459035930 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19024914 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30153612065152|‰Ç&«ÁM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 766.546630051 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1584336846848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30983864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12255232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30993536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32506307141120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1627414810624 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 $ _timestamp‚1.785806488254708e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1584424283648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19015244 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12254977 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1584424283648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30995866595328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5162298 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31734156904318 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30983864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7092932 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1580459035930 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19024914 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30153612065152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _step‚142 Eà®ü /loss/FRt2/assignment_loss‚3.929613849618933 0loss/FBt1/assignment_loss‚0.5011139743910462 .loss/FRt1/detection_loss‚3.641916954237968 1loss/SRqqt1/detection_loss‚1.5164632908840179 .loss/FBt2/detection_loss‚7.523396067720295 %loss/total_loss‚54.76303407198914 % _timestamp‚1.7858064882559319e+09 2loss/SRqqt1/assignment_loss‚1.6808678983495786 0loss/FBt2/assignment_loss‚0.0693192687036504 .loss/FRt2/detection_loss‚3.335499820800928 epoch‚0 _runtime‚ 766.547387833 _step‚143 2loss/SRqqt2/assignment_loss‚1.9099360298779275 1loss/SRqqt2/detection_loss‚3.8865798548277883 .loss/FBt1/detection_loss‚22.26163412258029 trainer/global_step‚1799 .loss/FRt1/assignment_loss‚4.50669293999672ó\ÍÕœ¯‚(:‘ ¥ýÄÓóÃäÒ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚142.602' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚79.22333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ö«·-°‚(:í ¥ýÄÓ›¥„Ö% disk./.usageGB‚" “K1239.1617164611816 disk.sda1.out‚0 disk.rbd4.out‚ 0.306640625 cpu‚14.031617934961767 proc.cpu.threads‚29 disk.dm-1.in‚0 disk.dm-0.in‚10.5625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚10.5625% memory_percent‚36.745309829711914 network.sent‚ 1.683459e+06 network.recv‚145313( disk./.usagePercent‚83.1611825176428 disk.sda.out‚ 38.671875 disk.rbd2.out‚ 0.306640625 disk.drbd1743.out‚0.9375' proc.memory.availableMB‚ 41454.59375" proc.memory.rssMB‚ 11267.796875 disk.nvme0n1.in‚10.5625 disk.rbd5.in‚0 disk.rbd5.out‚ 0.306640625 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.dm-1.out‚ 38.671875 disk.rbd1.out‚ 0.306640625" disk.dm-0.out‚2925.84326171875% disk.nvme0n1.out‚2925.84326171875 disk.rbd0.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.drbd1743.in‚0' disk.nvme0c0n1.out‚2925.84326171875 disk.sda3.out‚ 38.671875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0"Œ!á0±+ _wandbruntime‚787  _runtime‚787%‡JƲÀ _step‚144 & lr-AdamW/pg1‚4.765277281133098e-05 & lr-AdamW/pg2‚4.765277281133098e-05 trainer/global_step‚1849 % _timestamp‚1.7858065095245917e+09 _runtime‚ 787.8162954025Z¡g˳Š% _timestamp‚1.7858065095245917e+09 _runtime‚ 787.816295402 _step‚144 & lr-AdamW/pg1‚4.765277281133098e-05 & lr-AdamW/pg2‚4.765277281133098e-05 trainer/global_step‚1849’¬Íö0´+ _wandbruntime‚787  _runtime‚787«²—Á¹'µ³O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12294127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31763037205504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1627540792320 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33510690116096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12294129 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12588323 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1623470026216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 % _timestamp‚1.7858065095256405e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1671836666880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33390595037184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31839015318528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30973863469472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31826850 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33390490568192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12294127 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1627453362688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚1849 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33390595037184 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19532723 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7285847 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31763054244864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1627453362688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19542379 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31763054244864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623555829340>ápø¾'¶¸O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12294129 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19532723 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12588586 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30973863469472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31826850 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33390490568192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12294127 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1627453362688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚1849 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33390595037184 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19532723 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7285847 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31763054244864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1627453362688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19542379 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31763054244864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623555829340 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12294127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31763037205504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1627540792320 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33510690116096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12294129 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12588323 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1623470026216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 % _timestamp‚1.7858065095256405e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1671836666880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33390595037184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31839015318528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31856197531648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1628420419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1624347499438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5305374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624433302674 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12300770 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31780203968512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1672735513088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12300772 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31780221007872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 788.190926587 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33408641426944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31780203968512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1628420419072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12300772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12595156 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30990620681152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31844016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12595413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1672606943232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31853688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1849 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5305629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19543246 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31780203968512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1672735513088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12300772 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31780221007872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚ 788.190926587 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33408641426944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31780203968512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1628420419072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12300772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12595156 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30990620681152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31844016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12595413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1672606943232 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31853688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1849 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5305629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19543246 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31856197531648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1628420419072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1624347499438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5305374 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624433302674 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12300770 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34292824854016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32621375355904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34416371013120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671536927744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32699371158528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31810889124832 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32687002 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12928135 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34416211240960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚149 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1667358489724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1716864986112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32621392395264 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33478230575196 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32699346254848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33478333417680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20070381 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32621375355904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20060725 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5445673 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5445440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1667444292848"4C&Ä·O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32696660 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚1899 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20060725 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34292824854016 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32621375355904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34416371013120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671536927744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚32699371158528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31810889124832 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32687002 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12928135 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34416211240960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚149 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1667358489724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1716864986112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32621392395264 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33478230575196 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚32699346254848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33478333417680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20070381 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32621375355904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20060725 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5445673 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5445440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1667444292848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34292824854016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672416554496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7486632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32713840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5448488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20071248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20071248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32713840 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1717767190016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32638542118912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32638542118912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34434476132352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33495951063334 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672416554496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1668321766182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672329117696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12632920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32638559158272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34434320561664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1717897857024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34310975712768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12934870 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 % _timestamp‚1.7858065318907325e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7486634 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12632922 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668235962946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12632922 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12935122 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392Û±[‚È&ÉÂM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32638542118912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32638542118912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34434476132352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33495951063334 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672416554496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1668321766182 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1672329117696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12632920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32638559158272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34434320561664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1717897857024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34310975712768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12934870 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 % _timestamp‚1.7858065318907325e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7486634 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12632922 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1668235962946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12632922 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12935122 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32638559158272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5448238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚ 810.183015195 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20080918 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31827612257792 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672416554496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7486632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32713840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1899 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5448488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20071248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20071248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32713840 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1717767190016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392– ú0Ê+ _wandbruntime‚810  _runtime‚810¼–",Ëú 0loss/SRqqt1/detection_loss‚1.368321866903295 0loss/SRqqt2/detection_loss‚3.571344871662165 % _timestamp‚1.7858065318932252e+09 _runtime‚ 810.184756201 /loss/FBt2/detection_loss‚6.7537046803960745 trainer/global_step‚1899 0loss/FRt2/assignment_loss‚3.8673309249642456 2loss/SRqqt1/assignment_loss‚1.5044249251612232 2loss/SRqqt2/assignment_loss‚1.8892574992619062 0loss/FBt1/assignment_loss‚0.6154500531764435 /loss/FRt1/detection_loss‚3.6310572082346138 _step‚151 ,loss/FRt1/assignment_loss‚4.237824498833 1loss/FBt2/assignment_loss‚0.05980319546447513 /loss/FRt2/detection_loss‚3.7323964801099567 /loss/FBt1/detection_loss‚11.240535233701978 &loss/total_loss‚42.471451437869376 epoch‚0T¤ÚÌÿ ,loss/FRt1/assignment_loss‚4.237824498833 1loss/FBt2/assignment_loss‚0.05980319546447513 /loss/FRt2/detection_loss‚3.7323964801099567 /loss/FBt1/detection_loss‚11.240535233701978 &loss/total_loss‚42.471451437869376 epoch‚0 0loss/SRqqt1/detection_loss‚1.368321866903295 0loss/SRqqt2/detection_loss‚3.571344871662165 % _timestamp‚1.7858065318932252e+09 _runtime‚ 810.184756201 /loss/FBt2/detection_loss‚6.7537046803960745 trainer/global_step‚1899 0loss/FRt2/assignment_loss‚3.8673309249642456 2loss/SRqqt1/assignment_loss‚1.5044249251612232 2loss/SRqqt2/assignment_loss‚1.8892574992619062 0loss/FBt1/assignment_loss‚0.6154500531764435 /loss/FRt1/detection_loss‚3.6310572082346138 _step‚151—*Ö…‘Í‚(:† ÒýÄÓøÃÜÒ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚122.049' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚67.805 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä ¾{΂(:ð ÒýÄÓýúÖ# disk.nvme0n1.out‚2988.955078125 disk.rbd4.in‚0 disk.nvme0c0n1.in‚10.5625 disk.dm-1.out‚ 40.6796875 disk.dm-0.in‚10.5625 disk.sda2.out‚0) disk./.usagePercent‚83.16127391233253 disk.sda1.out‚0 disk.rbd5.out‚ 0.306640625 disk.rbd0.in‚0 disk.rbd2.in‚0$ memory_percent‚36.74599528312683 cpu‚14.026957886034648 network.recv‚150754 disk.rbd1.out‚ 0.306640625 disk.drbd1743.in‚0 disk.dm-0.out‚2988.955078125* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29% disk./.usageGB‚1239.1630783081055 disk.sda3.in‚0 disk.sda3.out‚ 40.6796875 disk.rbd1.in‚0 disk.rbd3.out‚0* proc.memory.availableMB‚41454.14453125" proc.memory.rssMB‚ 11267.796875 disk.rbd4.out‚ 0.306640625% disk.nvme0c0n1.out‚2988.955078125 disk.sda.in‚0 disk.sda.out‚ 40.6796875 disk.drbd1743.out‚ 0.98046875 disk.dm-1.in‚0 disk.sda2.in‚0 network.sent‚ 1.80684e+06 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚10.5625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd2.out‚ 0.306640625zäµ0Ï+ _wandbruntime‚831  _runtime‚831ù­6LÅп & lr-AdamW/pg1‚5.022999145986158e-05 & lr-AdamW/pg2‚5.022999145986158e-05 trainer/global_step‚1949 $ _timestamp‚1.785806552867424e+09 _runtime‚ 831.159134759 _step‚152²K:cÊÑÄ $ _timestamp‚1.785806552867424e+09 _runtime‚ 831.159134759 _step‚152 & lr-AdamW/pg1‚5.022999145986158e-05 & lr-AdamW/pg2‚5.022999145986158e-05 trainer/global_step‚1949˜uC~0Ò+ _wandbruntime‚831  _runtime‚831nËÐÓ³O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20588727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34359127654704 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20598383 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13268212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33479730545664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1715533063168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12958429 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35321897733632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32647880701472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35321735864320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12958427 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33547154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5588922 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1711332756356 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚1949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12958427 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35195159139840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33559726998528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35195263608832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 YKDeviceStatsMonitorÝÐ~<é .on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7679547 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1762170735104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33479713506304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33479730545664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33479713506304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1711246953232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5588667 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33559702094848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858065528684685e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1762033769472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32647897740832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251ªI1¾'Ô¸O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20598383 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13268212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33479730545664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1715533063168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12958429 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35321897733632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32647880701472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35321735864320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12958427 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33547154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5588922 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1711332756356 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚1949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12958427 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35195159139840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33559726998528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35195263608832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7679547 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1762170735104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33479713506304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33479730545664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33479713506304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1711246953232 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5588667 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33559702094848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858065528684685e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1762033769472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32647897740832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20588727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34359127654704™u»0Õ+ _wandbruntime‚831  _runtime‚831+srâÂ&Ö¼M E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33564320 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35213309998592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35213205522432 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1716412689920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13275010 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12965070 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33496897308672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1716412689920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33573992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12965072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33573992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12965072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762935023104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858065532420473e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33496880269312 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33564320 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1716325253120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35339844234752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1712210229690 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5591761 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33496880269312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12965070 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33496897308672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34376745300246 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20608920 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35340006096896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34376848142842¤¨;³D%×ÁM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33573992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12965072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33573992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12965072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1762935023104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858065532420473e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚33496880269312 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33564320 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1716325253120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35339844234752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1712210229690 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5591761 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33496880269312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12965070 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚33496897308672 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34376745300246 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20608920 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35340006096896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34376848142842 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33564320 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35213309998592 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35213205522432 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1716412689920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13275010 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12965070 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33496897308672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1716412689920 ICDev_.g ƒiceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130šßQ0Ø+ _wandbruntime‚831  _runtime‚831ãæ[ Ùü _step‚155 2loss/FBt2/assignment_loss‚0.035715073850416744 epoch‚0 2loss/SRqqt2/assignment_loss‚1.8255135466655095 &loss/total_loss‚59.780237664641305 _runtime‚ 831.534929158 /loss/FRt1/assignment_loss‚4.518753722094107 /loss/FRt2/assignment_loss‚4.032338506572849 2loss/SRqqt1/assignment_loss‚1.8152316526092331 0loss/FBt1/assignment_loss‚0.5577021148637868 .loss/FRt2/detection_loss‚3.333570661125602 1loss/SRqqt2/detection_loss‚3.5173289151145863 .loss/FBt1/detection_loss‚28.38473006337881 trainer/global_step‚1949 .loss/FRt1/detection_loss‚4.155924622995266 1loss/SRqqt1/detection_loss‚1.7330892205206765 /loss/FBt2/detection_loss‚5.8703395648504575 % _timestamp‚1.7858065532433777e+09¬¢(LÚ .loss/FRt1/detection_loss‚4.155924622995266 1loss/SRqqt1/detection_loss‚1.7330892205206765 /loss/FBt2/detection_loss‚5.8703395648504575 % _timestamp‚1.7858065532433777e+09 _step‚155 2loss/FBt2/assignment_loss‚0.035715073850416744 epoch‚0 2loss/SRqqt2/assignment_loss‚1.8255135466655095 &loss/total_loss‚59.780237664641305 _runtime‚ 831.534929158 /loss/FRt1/assignment_loss‚4.518753722094107 /loss/FRt2/assignment_loss‚4.032338506572849 2loss/SRqqt1/assignment_loss‚1.8152316526092331 0loss/FBt1/assignment_loss‚0.5577021148637868 .loss/FRt2/detection_loss‚3.333570661125602 1loss/SRqqt2/detection_loss‚3.5173289151145863 .loss/FBt1/detection_loss‚28.38473006337881 trainer/global_step‚1949›%ÇvšÛ‚(: áýÄÓ©ãÎÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚99.3' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚55.166666666666664 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¤®dÜ‚(:ö áýÄÓÜ£­Ö disk.rbd2.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚3051.72705078125 disk.sda1.in‚0% memory_percent‚36.746084690093994 network.recv‚152889 disk.nvme0c0n1.in‚10.5625 disk.sda.out‚ 41.2421875 disk.rbd1.in‚0 disk.nvme0n1.in‚10.5625 disk.rbd5.in‚0 disk.rbd5.out‚ 0.306640625* proc.memory.percent‚17.193293571472168 network.sent‚ 1.841227e+06 disk.rbd2.out‚ 0.306640625 disk.rbd1.out‚ 0.306640625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.dm-0.in‚10.5625 disk.dm-1.out‚ 41.2421875" disk.dm-0.out‚3051.72705078125 disk.rbd3.in‚0) proc.memory.availableMB‚ 41454.0859375 proc.cpu.threads‚29 disk.sda3.out‚ 41.2421875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd4.out‚ 0.306640625 cpu‚14.02559173046912' disk.nvme0c0n1.out‚3051.72705078125 disk.drbd1743.in‚0 disk.drbd1743.out‚ 0.99609375 disk.rbd3.out‚0" proc.memory.rssMB‚ 11267.796875) disk./.usagePercent‚83.17805187810404% disk./.usageGB‚1239.4130821228027åÏ®0Ý+ _wandbruntime‚852  _runtime‚852v‹·ªÅÞ¿ _step‚156 & lr-AdamW/pg1‚5.151860078412689e-05 & lr-AdamW/pg2‚5.151860078412689e-05 trainer/global_step‚1999 $ _timestamp‚1.785806574657856e+09 _runtime‚ 852.949585204ÛÒ`WÊßÄ & lr-AdamW/pg1‚5.151860078412689e-05 & lr-AdamW/pg2‚5.151860078412689e-05 trainer/global_step‚1999 $ _timestamp‚1.785806574657856e+09 _runtime‚ 852.949585204 _step‚156œÀC½'0à+ _wandbruntime‚852  _runtime‚852Œ®u¸¹'á³O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290579 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1755135416740 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34407306 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21116729 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36097493425664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759441768960 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36227221919232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34338051656704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 852.953972964 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 % _timestamp‚1.7858065746589713e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36097493425664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33484889317472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36227377497088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1759529198592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34416964 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34338068696064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1759529198592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21116729 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5732599 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1807163984384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36097597894656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13608996 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290579 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚34338068696064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33484906356832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34420082838528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1755221219864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35240127576696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13608737 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34420057934848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290577 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36097597894656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1759441768960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1807294658560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34338051656704‹Ç¾'â¸O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21126385 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5732342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7876397 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7876395 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13608996 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290579 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚34338068696064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33484906356832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚34420082838528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1755221219864 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35240127576696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13608737 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34420057934848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13290577 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36097597894656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1759441768960 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1807294658560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34338051656704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290579 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1755135416740 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34407306 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21116729 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36097493425664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1759441768960 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36227221919232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34338051656704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 852.953972964 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 % _timestamp‚1.7858065746589713e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36097493425664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33484889317472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36227377497088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1759529198592 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34416964 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚34338068696064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1759529198592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21116729 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5732599 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1807163984384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36097597894656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13615481 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 % _timestamp‚1.7858065750769444e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7880332 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34355218419712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13615726 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1760408825344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34434144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34434144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21136922 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5735149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297220 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7880334 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1756012889962 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34424472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36115644284416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1760321388544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21127252 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1756098693198 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36245503612416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297222 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚34355235459072 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13615481 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 % _timestamp‚1.7858065750769444e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7880332 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚34355218419712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13615726 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1760408825344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34434144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34434144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21136922 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5735149 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297220 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7880334 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1756012889962 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34424472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚36115644284416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1760321388544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21127252 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1756098693198 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36245503612416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13297222 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392žÎ?+0æ+ _wandbruntime‚853  _runtime‚853â¡ çû epoch‚0 /loss/FRt1/assignment_loss‚4.134256238287145 2loss/SRqqt2/assignment_loss‚1.9643849884825093 0loss/FBt1/assignment_loss‚0.6229248845730371 /loss/FRt1/detection_loss‚3.2695433578707953 1loss/SRqqt2/detection_loss‚3.2992524662542912 .loss/FBt1/detection_loss‚7.785735850150768 trainer/global_step‚1999 1loss/FBt2/assignment_loss‚0.09979134082176541 1loss/SRqqt1/detection_loss‚1.2906357398066972 .loss/FBt2/detection_loss‚4.835280049830012 % _timestamp‚1.7858065750787325e+09 _runtime‚ 853.370283972 &loss/total_loss‚36.071319149098215 0loss/FRt2/assignment_loss‚3.7509229515874107 0loss/SRqqt1/assignment_loss‚1.56379970125854 _step‚159 /loss/FRt2/detection_loss‚3.4547915801752445¬=:üè€ &loss/total_loss‚36.071319149098215 0loss/FRt2/assignment_loss‚3.7509229515874107 0loss/SRqqt1/assignment_loss‚1.56379970125854 _step‚159 /loss/FRt2/detection_loss‚3.4547915801752445 epoch‚0 /loss/FRt1/assignment_loss‚4.134256238287145 2loss/SRqqt2/assignment_loss‚1.9643849884825093 0loss/FBt1/assignment_loss‚0.6229248845730371 /loss/FRt1/detection_loss‚3.2695433578707953 1loss/SRqqt2/detection_loss‚3.2992524662542912 .loss/FBt1/detection_loss‚7.785735850150768 trainer/global_step‚1999 1loss/FBt2/assignment_loss‚0.09979134082176541 1loss/SRqqt1/detection_loss‚1.2906357398066972 .loss/FBt2/detection_loss‚4.835280049830012 % _timestamp‚1.7858065750787325e+09 _runtime‚ 853.370283972ŸXq1%‘é‚(:† ðýÄÓ÷àÒ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚119.619' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚66.455 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V‰÷¯}ê‚(:ò ðýÄÓë¼ÍÖ network.sent‚ 1.886017e+06 disk.sda3.in‚0 disk.drbd1743.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚10.5625 disk.sda.out‚ 41.80078125 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 41.80078125 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚83.17808567109856 disk.nvme0n1.in‚10.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.306640625% memory_percent‚36.746132373809814) proc.memory.availableMB‚ 41454.0546875 proc.cpu.threads‚29% disk./.usageGB‚1239.4135856628418% disk.nvme0c0n1.out‚3051.939453125 disk.drbd1743.out‚1.015625 disk.dm-0.out‚3051.939453125 disk.rbd5.in‚0" proc.memory.rssMB‚ 11267.796875 network.recv‚154719 disk.sda3.out‚ 41.80078125 disk.dm-0.in‚10.5625 disk.sda.in‚0 disk.rbd2.out‚ 0.306640625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd4.out‚ 0.306640625* proc.memory.percent‚17.193293571472168 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0# disk.nvme0n1.out‚3051.939453125 cpu‚14.024132824319375â±7Në‚(:’ ÿýÄÓ¾•íÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚109.184' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚60.657777777777774 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y¤Õ…ì‚(:ú ÿýÄÓüÉíÖ disk.sda3.out‚ 42.66015625 disk.drbd1743.out‚ 1.05859375 disk.dm-0.in‚14.5625 disk.rbd3.out‚0 network.recv‚156708) disk./.usagePercent‚83.17831838194718 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 network.sent‚ 1.930829e+06 disk.rbd4.in‚0 disk.sda.out‚ 42.66015625 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.sda1.out‚0 cpu‚14.023364393012173% disk./.usageGB‚1239.4170532226562 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 0.306640625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚3056.26806640625 disk.nvme0n1.in‚14.5625 disk.rbd5.out‚ 0.306640625 disk.dm-1.out‚ 42.66015625% disk.nvme0n1.out‚3056.26806640625* proc.memory.percent‚17.193293571472168 disk.rbd4.out‚ 0.306640625 disk.rbd1.out‚ 0.306640625% memory_percent‚36.746132373809814) proc.memory.availableMB‚ 41454.0546875" proc.memory.rssMB‚ 11267.796875 disk.nvme0c0n1.in‚14.5625' disk.nvme0c0n1.out‚3056.26806640625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.sda3.in‚0òŽDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 874.701059734 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13948570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1803525334016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852361030144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1803437904384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875323 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34321897933472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1799109683372 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21654387 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35280438678528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21644731 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37132936674304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1803525334016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13948310 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13622729 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35196406846464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36999827711488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13622727 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36999932180480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34321914972832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1799023880248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36999827711488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8073247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5875065 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35280413774848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13622727 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36999932180480 % _timestamp‚1.7858065964091434e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21654387 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35196389807104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35196406846464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1803437904384 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875323 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34321897933472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1799109683372 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21654387 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35280438678528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21644731 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37132936674304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1803525334016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13948310 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13622729 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35196406846464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚36999827711488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13622727 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36999932180480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34321914972832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1799023880248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36999827711488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8073247 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5875065 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35280413774848 GADeviceStat°¨rŠsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13622727 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚36999932180480 % _timestamp‚1.7858065964091434e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21654387 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35196389807104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35196406846464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 874.701059734 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13948570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1803525334016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852361030144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 875.082876279 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853397300736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34338638105792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13629370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13955326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35213573609472 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13629370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8077182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35284624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35294296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37150889622528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35213556570112 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37017874094080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858065967906818e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13955079 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5878142 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36138539459262 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21655254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21664924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35297645795328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21664924 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1804317523968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35297620891648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37017978570240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853268730880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34338655145152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35213573609472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136Pî„}È&õÂM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚ 875.082876279 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853397300736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34338638105792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13629370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13955326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35213573609472 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13629370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8077182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35284624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35294296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37150889622528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35213556570112 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37017874094080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858065967906818e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13955079 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5878142 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36138539459262 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21655254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21664924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35297645795328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21664924 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1804317523968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35297620891648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37017978570240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853268730880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34338655145152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35213573609472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13629372 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37902161997312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8270095 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36054727957504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13954879 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1847521469440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1847434039808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22182389 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36054727957504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14288623 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚2099 $ _timestamp‚1.785806618894326e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13954877 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22172733 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1897532271104 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37001921735712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37902266466304 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13954877 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1847434039808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^LDeviceStAËu¥¥atsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1847521469440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚37001818893228 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22172733 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13954879 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1842998146880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37902266466304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1897667139584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36137268 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36140794518528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1842912343756 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6018764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37902161997312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36054744996864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14288861 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35158923588832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36127610 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37902161997312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36054744996864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14288861 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35158923588832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36127610 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37902161997312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8270095 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36054727957504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13954879 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1847521469440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1847434039808 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22182389 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36054727957504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392¥š.³Æ0+ _wandbruntime‚897  _runtime‚897?Ä;Ã&‚½M OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14295625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8274032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6021335 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1898581984768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37019436538770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14295367 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1848313659392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1848401096192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36071911759872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22192926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22183256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36154448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36157976731648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37920312856064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36071911759872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36071894720512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 897.553303328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1848313659392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1843789816978 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37920312856064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7858066192613244e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚166 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13961520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22183256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37019539381366 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37920208379904 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121ÊQ÷´ˆƒÂM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13961520 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22183256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37019539381366 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37920208379904 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14295625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8274032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6021335 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1898581984768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37019436538770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14295367 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1848313659392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1848401096192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36071911759872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22192926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22183256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36154448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36157976731648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37920312856064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36071911759872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36071894720512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 897.553303328 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1848313659392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1843789816978 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37920312856064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7858066192613244e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚166 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816¦*_Ñ0„+ _wandbruntime‚897  _runtime‚897Ó¶…ú %loss/total_loss‚39.40285942269536 1loss/FBt2/assignment_loss‚0.14814712180640738 /loss/FRt1/detection_loss‚3.9039895729699605 .loss/FRt2/detection_loss‚3.264930202563604 1loss/SRqqt1/detection_loss‚1.2677159593536977 /loss/FBt1/detection_loss‚6.3536022504174605 .loss/FBt2/detection_loss‚8.577405475741035 _step‚167 /loss/FRt2/assignment_loss‚4.021710705757141 trainer/global_step‚2099 _runtime‚ 897.554091757 epoch‚0 2loss/SRqqt1/assignment_loss‚1.7355888376171276 1loss/SRqqt2/detection_loss‚3.3987393164634705 /loss/FRt1/assignment_loss‚4.142488855711171 1loss/SRqqt2/assignment_loss‚2.014241503874461 0loss/FBt1/assignment_loss‚0.5742996204198183 % _timestamp‚1.7858066192626278e+09\‰;†ÿ .loss/FRt2/detection_loss‚3.264930202563604 1loss/SRqqt1/detection_loss‚1.2677159593536977 /loss/FBt1/detection_loss‚6.3536022504174605 .loss/FBt2/detection_loss‚8.577405475741035 _step‚167 /loss/FRt2/assignment_loss‚4.021710705757141 trainer/global_step‚2099 _runtime‚ 897.554091757 epoch‚0 2loss/SRqqt1/assignment_loss‚1.7355888376171276 1loss/SRqqt2/detection_loss‚3.3987393164634705 /loss/FRt1/assignment_loss‚4.142488855711171 1loss/SRqqt2/assignment_loss‚2.014241503874461 0loss/FBt1/assignment_loss‚0.5742996204198183 % _timestamp‚1.7858066192626278e+09 %loss/total_loss‚39.40285942269536 1loss/FBt2/assignment_loss‚0.14814712180640738 /loss/FRt1/detection_loss‚3.9039895729699605§ƹ¶È›‡‚(: þÄÓ«úìÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚134.044' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚74.4688888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×½‘‚(:õ þÄÓ©ÜãÕ disk.drbd1743.out‚ 1.08984375 disk.dm-0.in‚14.5625" disk.dm-0.out‚3119.58544921875 cpu‚14.019063675894207 disk.dm-1.in‚0 disk.dm-1.out‚ 44.0234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.306640625' disk.nvme0c0n1.out‚3119.58544921875 disk.rbd2.out‚ 0.306640625 network.sent‚ 1.978084e+06) disk./.usagePercent‚83.17831300578897 disk.rbd4.in‚0 disk.nvme0c0n1.in‚14.5625 disk.drbd1743.in‚0$ memory_percent‚36.74666881561279 disk.nvme0n1.in‚14.5625" proc.memory.rssMB‚ 11267.796875% disk./.usageGB‚1239.4169731140137 disk.rbd0.out‚0 disk.sda.out‚ 44.0234375% disk.nvme0n1.out‚3119.58544921875 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 44.0234375 disk.sda2.in‚0( proc.memory.availableMB‚ 41453.703125* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29 network.recv‚160038 disk.rbd5.in‚0 disk.rbd4.out‚ 0.306640625 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.rbd3.out‚0—`Öœœ‰‚(:‘ ¬þÄÓãðôÒ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚149.545' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.08055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Lj–xŠ‚(:í ¬þÄÓª¾™Ö disk.rbd3.in‚0 disk.rbd0.in‚0$ memory_percent‚36.74628734588623* proc.memory.percent‚17.193293571472168 network.recv‚162003 disk.dm-1.out‚ 44.57421875 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.nvme0n1.in‚14.5625 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 3119.91796875 disk.rbd5.in‚0 disk.rbd4.out‚ 0.306640625( proc.memory.availableMB‚ 41453.953125 disk.sda1.in‚0 disk.rbd4.in‚0) disk./.usagePercent‚83.17831710190951 disk.sda3.out‚ 44.57421875 disk.rbd3.out‚0 network.sent‚ 2.022736e+06# disk./.usageGB‚1239.41703414917 disk.rbd2.in‚0 disk.dm-1.in‚0 cpu‚14.017659564575942 disk.sda.out‚ 44.57421875 disk.rbd2.out‚ 0.306640625$ disk.nvme0c0n1.out‚ 3119.91796875 proc.cpu.threads‚29 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.dm-0.out‚ 3119.91796875 disk.rbd5.out‚ 0.306640625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚14.5625" proc.memory.rssMB‚ 11267.796875 disk.sda3.in‚0 disk.drbd1743.out‚ 1.08984375 disk.dm-0.in‚14.5625xá´Q0‹+ _wandbruntime‚919  _runtime‚919°ZèÇŒÁ $ _timestamp‚1.785806640987553e+09 _runtime‚ 919.279251881 _step‚168 ' lr-AdamW/pg1‚5.5384428756922805e-05 ' lr-AdamW/pg2‚5.5384428756922805e-05 trainer/global_step‚2149TÐ2™ÌÆ _step‚168 ' lr-AdamW/pg1‚5.5384428756922805e-05 ' lr-AdamW/pg2‚5.5384428756922805e-05 trainer/global_step‚2149 $ _timestamp‚1.785806640987553e+09 _runtime‚ 919.279251881¨ÈÁXF0Ž+ _wandbruntime‚919  _runtime‚919#è¹'³O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1891430175232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14629608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38804496283136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36997420 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37001150358528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37882818815220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1886800807264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120188416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6162661 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38804496283136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36913066107904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38943998860288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36913083147264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858066409885812e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14629356 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8466947 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14287029 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35995932204832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36987762 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35995915165472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 919.283951076 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14287027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚2149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14287029 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36913083147264 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38804600752128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8466945 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22700735 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36913066107904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22710391 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6162411 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37001125454848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1891517604864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38804600752128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1942728313344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0F¤ýû¾'¸O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37001125454848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1891517604864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38804600752128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1942728313344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1891430175232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14629608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38804496283136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36997420 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37001150358528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚37882818815220 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1886800807264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 120188416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6162661 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38804496283136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36913066107904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38943998860288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36913083147264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858066409885812e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14629356 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8466947 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14287029 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35995932204832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36987762 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35995915165472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚ 919.283951076 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14287027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚2149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14287029 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36913083147264 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38804600752128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8466945 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22700735 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36913066107904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22710391 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6162411 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640© I_0‘+ _wandbruntime‚919  _runtime‚919TòlHR’¼M ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37900436460874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36930232870912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36012672377152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1892397231616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚170 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14293670 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8470882 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37018332571648 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14636134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1943761929728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UIDeviceStatsMonitor.on_train_batch_end/i-_O5pnactive_split.large_pool.allocated‚8470884 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38822647141888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14293672 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22720928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37018357475328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 919.64595815 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1892309794816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38962119405056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38961963834368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37900333618278 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14293670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1887764083722 % _timestamp‚1.7858066413537316e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37004928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14636392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6165508 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36930249910272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1943631262720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37014600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37014600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36012655337792 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38822647141888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36930232870912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22711258 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130š âiÇ&“ÁM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8470884 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38822647141888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14293672 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22720928 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37018357475328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _runtime‚ 919.64595815 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1892309794816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚38962119405056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38961963834368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37900333618278 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14293670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1887764083722 % _timestamp‚1.7858066413537316e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37004928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14636392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6165508 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36930249910272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1943631262720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37014600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37014600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36012655337792 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚38822647141888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36930232870912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22711258 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37900436460874 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36930232870912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36012672377152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1892397231616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚170 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14293670 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8470882 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37018332571648 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14636134 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1943761929728ª½)¥H0”+ _wandbruntime‚919  _runtime‚919]éŽÎ•û /loss/FRt1/assignment_loss‚4.507766503177277 1loss/SRqqt1/assignment_loss‚1.604718337016823 /loss/FRt2/detection_loss‚3.0861752374718585 0loss/SRqqt1/detection_loss‚1.379290347370825 epoch‚0 trainer/global_step‚2149 $ _timestamp‚1.785806641355082e+09 0loss/FRt2/assignment_loss‚3.9077889546751976 .loss/FBt1/detection_loss‚9.053445603166308 &loss/total_loss‚36.811986823439874 _step‚171 2loss/SRqqt2/assignment_loss‚1.9437600672245026 0loss/FBt1/assignment_loss‚0.5082177138205485 /loss/FRt1/detection_loss‚3.8262915055025113 1loss/SRqqt2/detection_loss‚3.4720206918319065 _runtime‚ 919.646627346 1loss/FBt2/assignment_loss‚0.04693567415857355 /loss/FBt2/detection_loss‚3.4755761880235396{+–€ /loss/FRt1/assignment_loss‚4.507766503177277 1loss/SRqqt1/assignment_loss‚1.604718337016823 /loss/FRt2/detection_loss‚3.0861752374718585 0loss/SRqqt1/detection_loss‚1.379290347370825 epoch‚0 trainer/global_step‚2149 $ _timestamp‚1.785806641355082e+09 0loss/FRt2/assignment_loss‚3.9077889546751976 .loss/FBt1/detection_loss‚9.053445603166308 &loss/total_loss‚36.811986823439874 _step‚171 2loss/SRqqt2/assignment_loss‚1.9437600672245026 0loss/FBt1/assignment_loss‚0.5082177138205485 /loss/FRt1/detection_loss‚3.8262915055025113 1loss/SRqqt2/detection_loss‚3.4720206918319065 _runtime‚ 919.646627346 1loss/FBt2/assignment_loss‚0.04693567415857355 /loss/FBt2/detection_loss‚3.4755761880235396«þ €œ—‚(:‘ »þÄÓ­«äÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.72611111111111 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„2Mý‚˜‚(:÷ »þÄÓÀãÇÖ disk.sda2.in‚0 disk.rbd5.out‚ 0.306640625 network.sent‚ 2.025193e+06 disk.nvme0c0n1.in‚14.5625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd5.in‚0 proc.cpu.threads‚29) disk./.usagePercent‚83.17831863795472 disk.rbd4.in‚0 disk.sda3.out‚ 45.37109375 disk.drbd1743.out‚ 1.1171875 disk.nvme0n1.in‚14.5625 network.recv‚163147 disk.sda.out‚ 45.37109375( proc.memory.availableMB‚ 41453.671875* proc.memory.percent‚17.193293571472168% disk./.usageGB‚1239.4170570373535' disk.nvme0c0n1.out‚3182.88720703125 disk.rbd2.out‚ 0.392578125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.455078125 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1743.in‚0 disk.rbd3.out‚0$ memory_percent‚36.74671649932861 cpu‚14.016176522402287 disk.dm-1.in‚0 disk.dm-0.in‚14.5625% disk.nvme0n1.out‚3182.88720703125 disk.rbd0.in‚0 disk.rbd2.in‚0" disk.dm-0.out‚3182.88720703125 disk.sda1.out‚0" proc.memory.rssMB‚ 11267.796875 disk.rbd0.out‚0 disk.rbd4.out‚ 0.306640625 disk.sda3.in‚0 disk.dm-1.out‚ 45.37109375„_S0™+ _wandbruntime‚940  _runtime‚940Ì —kÄš¾ _runtime‚ 940.840901143 _step‚172 % lr-AdamW/pg1‚5.66730380811881e-05 % lr-AdamW/pg2‚5.66730380811881e-05 trainer/global_step‚2199 % _timestamp‚1.7858066625492237e+09"ù#¥É›Ã % lr-AdamW/pg1‚5.66730380811881e-05 % lr-AdamW/pg2‚5.66730380811881e-05 trainer/global_step‚2199 % _timestamp‚1.7858066625492237e+09 _runtime‚ 940.840901143 _step‚172¬4íD0œ+ _wandbruntime‚940  _runtime‚940sIŠŠ¹'³O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6306138 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1987915663360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8663795 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚173 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14969935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23238393 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23228737 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1930775073896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14619177 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6305897 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38763715894728 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1988050531840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39849396958208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37847914 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930689270772 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37771421297664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1935426310656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23238393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39706830568960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39849556730368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37771404258304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1935426310656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚2199 % _timestamp‚1.7858066625502274e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39706935037952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38763613052244 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36832940820832 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39706830568960 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14619177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1935513740288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712_Ô^s ž¸O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23238393 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39706830568960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39849556730368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37771404258304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1935426310656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 trainer/global_step‚2199 % _timestamp‚1.7858066625502274e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39706935037952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38763613052244 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36832940820832 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39706830568960 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14619177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1935513740288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6306138 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1987915663360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8663795 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚173 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14969935 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23238393 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23228737 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1930775073896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14619177 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6305897 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38763715894728 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1988050531840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39849396958208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37847914 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1930689270772 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37771421297664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1935426310656­UÒƒ_0Ÿ+ _wandbruntime‚941  _runtime‚941B&³ŠÃ& ½M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39867668739072 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39867500585472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1936393367040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14625822 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37865080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38781230697786 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 941.204018002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37878688411648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23248930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14976450 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8667732 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6308962 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23248930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23239260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1936393367040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23239260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37874752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37878713315328 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39724981427712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14625822 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37874752 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858066629118564e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1936305930240 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39724876951552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1988955423744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1931566743994 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37788571021312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251«Üd…È&¡ÂM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23239260 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37874752 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37878713315328 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39724981427712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14625822 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37874752 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858066629118564e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1936305930240 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39724876951552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1988955423744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1931566743994 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37788571021312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39867668739072 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39867500585472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1936393367040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14625822 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37865080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38781230697786 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 941.204018002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37878688411648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23248930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14976450 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8667732 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6308962 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23248930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23239260 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1936393367040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40609269323776 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6449764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1979422446080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1979509875712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40609164854784 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _runtime‚ 962.714255458 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38629742408704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860645 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979509875712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40755062489600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37669932397472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40609164854784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6450001 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40609269323776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2033200451072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14951327 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23756739 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38721837134848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1974577734280 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40609269323776 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6449764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1979422446080 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1979509875712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40609164854784 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _runtime‚ 962.714255458 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38629742408704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860645 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979509875712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40755062489600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37669932397472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40609164854784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6450001 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40609269323776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2033200451072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14951327 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23756739 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚2249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38721837134848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392±¹Z¯^0¯+ _wandbruntime‚963  _runtime‚963Â&°¼M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38646909171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14957972 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38739069155328 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15317183 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2034105809920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8864582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1980389502464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38734904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40773013103104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2033968851456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1980302065664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8864584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40773174965248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38646926211072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15317459 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37686672569792 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38734904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40627315713536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38646926211072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39662230619890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785806684847755e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40627315713536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38739044251648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23767262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38646909171712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23767262 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40627211237376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 963.140064679 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14957970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1980302065664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37686689609152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251)2(X ±ÁM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37686689609152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38646909171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14957972 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38739069155328 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15317183 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2034105809920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8864582 PBDeviceS€™#*tatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1980389502464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38734904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40773013103104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2033968851456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1980302065664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8864584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40773174965248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38646926211072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15317459 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37686672569792 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38734904 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40627315713536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38646926211072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39662230619890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785806684847755e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40627315713536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38739044251648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23767262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38646909171712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23767262 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40627211237376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 963.140064679 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14957970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1980302065664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0²¶„L•0²+ _wandbruntime‚963  _runtime‚963î8гú 0loss/FRt2/assignment_loss‚3.8894009891597703 -loss/FRt1/detection_loss‚3.11840126543552 0loss/SRqqt2/detection_loss‚4.059248780683865 2loss/SRqqt1/assignment_loss‚1.5147583391084227 2loss/SRqqt2/assignment_loss‚1.8618699797089138 &loss/total_loss‚35.146796915252466 epoch‚0 _step‚179 /loss/FBt1/detection_loss‚7.4395564510122405 /loss/FRt1/assignment_loss‚4.148041845738203 1loss/FBt1/assignment_loss‚0.46495310462099343 1loss/FBt2/assignment_loss‚0.03577632131494939 -loss/FRt2/detection_loss‚3.35658142827023 1loss/SRqqt1/detection_loss‚1.3390650224503773 trainer/global_step‚2249 % _timestamp‚1.7858066848494902e+09 .loss/FBt2/detection_loss‚3.919143387748978 _runtime‚ 963.141095623+˜y´ÿ 1loss/SRqqt1/detection_loss‚1.3390650224503773 trainer/global_step‚2249 % _timestamp‚1.7858066848494902e+09 .loss/FBt2/detection_loss‚3.919143387748978 _runtime‚ 963.141095623 0loss/FRt2/assignment_loss‚3.8894009891597703 -loss/FRt1/detection_loss‚3.11840126543552 0loss/SRqqt2/detection_loss‚4.059248780683865 2loss/SRqqt1/assignment_loss‚1.5147583391084227 2loss/SRqqt2/assignment_loss‚1.8618699797089138 &loss/total_loss‚35.146796915252466 epoch‚0 _step‚179 /loss/FBt1/detection_loss‚7.4395564510122405 /loss/FRt1/assignment_loss‚4.148041845738203 1loss/FBt1/assignment_loss‚0.46495310462099343 1loss/FBt2/assignment_loss‚0.03577632131494939 -loss/FRt2/detection_loss‚3.35658142827023³oÖ䑵‚(:† èþÄÓ¸¤ŽÓ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚135.693' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚75.385 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[20¸~¶‚(:ó èþÄÓµÌÑÖ) disk./.usagePercent‚83.17729204774241 disk.rbd2.out‚ 0.455078125 disk.drbd1743.out‚1.15625" disk.dm-0.out‚3246.07177734375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.392578125% disk./.usageGB‚1239.4017601013184 disk.dm-1.out‚ 47.328125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd5.in‚0 disk.rbd0.out‚0* proc.memory.availableMB‚41453.35546875* proc.memory.percent‚17.193293571472168' disk.nvme0c0n1.out‚3246.07177734375 disk.rbd2.in‚0 disk.sda3.out‚ 47.328125 network.recv‚167845 disk.drbd1743.in‚0 disk.nvme0n1.in‚14.5625 proc.cpu.threads‚29 disk.nvme0c0n1.in‚14.5625 disk.sda.out‚ 47.328125 disk.rbd4.out‚ 0.306640625 network.sent‚ 2.117038e+06 disk.sda.in‚0 disk.sda3.in‚0 cpu‚14.012702161150676 disk.rbd1.out‚ 0.4873046875 disk.sda2.in‚0 disk.dm-0.in‚14.5625 disk.rbd3.out‚0% disk.nvme0n1.out‚3246.07177734375" proc.memory.rssMB‚ 11267.796875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0% memory_percent‚36.747199296951294ÜÑ0·+ _wandbruntime‚984  _runtime‚984Bl‘¨È¸Â ' lr-AdamW/pg1‚5.9250256729718725e-05 ' lr-AdamW/pg2‚5.9250256729718725e-05 trainer/global_step‚2299 % _timestamp‚1.7858067063436346e+09 _runtime‚ 984.635329853 _step‚180…ï]Í¹Ç % _timestamp‚1.7858067063436346e+09 _runtime‚ 984.635329853 _step‚180 ' lr-AdamW/pg1‚5.9250256729718725e-05 ' lr-AdamW/pg2‚5.9250256729718725e-05 trainer/global_step‚2299´«G;0º+ _wandbruntime‚984  _runtime‚9847‘Î9¸'»²O ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2018552000912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39582192974848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39488080559104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39577876 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2078279569920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2078414438400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41511603609600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24284741 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2023506011136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15283479 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41511499140608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41660632316928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2023506011136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9057495 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39582217878528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41660472544768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38506958052832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15651795 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24294397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2023418581504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39488097598464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15283477 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41511499140608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39488097598464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39568218 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40525407211260 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2018466197788 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2023418581504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785806706344695e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24294397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2023418581504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39488097598464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15283477 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41511499140608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39488097598464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚224 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39568218 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40525407211260 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2018466197788 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2023418581504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785806706344695e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15283479 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41511499140608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41660632316928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2023506011136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9057495 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39582217878528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41660472544768 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38506958052832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15651795 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41529545523200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39505264361472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24304934 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39599400091648 E?DeviceStatsMonitor.on¶ß1- _train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40543127699398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39505264361472 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39595056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40543024856802 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39505247322112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚182 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39585384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15658547 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38523681185792 % _timestamp‚1.7858067067289498e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2024298201088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6597361 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24295264 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9061434 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41678754606592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15290120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41529649999360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15290122 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24295264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39595056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2019429474246 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41678603230208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2079329611264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 985.021067645 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2024385637888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2024385637888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39585384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2024298201088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39595056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40543024856802 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39505247322112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚182 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39585384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15658547 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38523681185792 % _timestamp‚1.7858067067289498e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2024298201088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6597361 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24295264 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9061434 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41678754606592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15290120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41529649999360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15290122 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24295264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39595056 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2019429474246 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41678603230208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2079329611264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚ 985.021067645 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2024385637888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2024385637888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2299 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39585384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2024298201088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41529545523200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39505264361472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24304934 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39599400091648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40543127699398 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640¶Ø‚S0À+ _wandbruntime‚985  _runtime‚985n6B]Áü 0loss/FBt1/assignment_loss‚0.3997148547496181 1loss/SRqqt2/detection_loss‚2.9365080976965783 .loss/FBt2/detection_loss‚5.724325804101305 /loss/FRt1/assignment_loss‚4.449815610593015 /loss/FBt1/detection_loss‚11.319795740975273 %loss/total_loss‚39.37040054491901 _runtime‚ 985.021830366 _step‚183 0loss/FRt2/assignment_loss‚3.8237596420531577 /loss/FRt1/detection_loss‚3.0937330749055203 /loss/FRt2/detection_loss‚3.0302378839634834 epoch‚0 1loss/SRqqt2/assignment_loss‚1.984872291142913 2loss/FBt2/assignment_loss‚0.042086906372527925 1loss/SRqqt1/detection_loss‚0.8484590834917489 trainer/global_step‚2299 % _timestamp‚1.7858067067303703e+09 1loss/SRqqt1/assignment_loss‚1.717091554873868g¼LÉ 1loss/SRqqt1/assignment_loss‚1.717091554873868 0loss/FBt1/assignment_loss‚0.3997148547496181 1loss/SRqqt2/detection_loss‚2.9365080976965783 .loss/FBt2/detection_loss‚5.724325804101305 /loss/FRt1/assignment_loss‚4.449815610593015 /loss/FBt1/detection_loss‚11.319795740975273 %loss/total_loss‚39.37040054491901 _runtime‚ 985.021830366 _step‚183 0loss/FRt2/assignment_loss‚3.8237596420531577 /loss/FRt1/detection_loss‚3.0937330749055203 /loss/FRt2/detection_loss‚3.0302378839634834 epoch‚0 1loss/SRqqt2/assignment_loss‚1.984872291142913 2loss/FBt2/assignment_loss‚0.042086906372527925 1loss/SRqqt1/detection_loss‚0.8484590834917489 trainer/global_step‚2299 % _timestamp‚1.7858067067303703e+09·N¸œÃ‚(:‘ ÷þÄÓ¥ÉÚÒ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚148.947' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.74833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åà ‰Ä‚(:þ ÷þÄÓÄ‹®Õ! disk.nvme0c0n1.in‚ 14.56640625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 14.56640625 cpu‚14.011527400396142 proc.cpu.threads‚29 disk.rbd3.in‚0$ memory_percent‚36.74760460853577 disk.sda3.in‚0 disk.rbd4.out‚ 0.306640625 disk.sda1.out‚0 network.recv‚170005 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd5.in‚0* proc.memory.availableMB‚41453.08984375( disk./.usagePercent‚83.1773248167068 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 14.56640625 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 disk.sda3.out‚ 48.921875 disk.rbd1.in‚0 disk.drbd1743.out‚1.1875% disk.nvme0n1.out‚3246.78173828125 network.sent‚ 2.161818e+06' disk.nvme0c0n1.out‚3246.78173828125 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.rbd5.out‚ 0.392578125% disk./.usageGB‚1239.4022483825684 disk.sda.out‚ 48.921875 disk.rbd2.out‚ 0.4873046875 disk.dm-1.out‚ 48.921875 disk.rbd1.out‚ 0.55029296875" disk.dm-0.out‚3246.78173828125^¿œÅ‚(:‘ †ÿÄÓ¡—ÚÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚122.969' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.31611111111111 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pCfn‰Æ‚(:þ †ÿÄÓÒ¸êÖ disk.nvme0c0n1.in‚18.53125 disk.dm-1.out‚ 49.48828125 network.sent‚ 2.206401e+06 disk.drbd1743.in‚0" disk.dm-0.out‚3299.04052734375 disk.rbd5.in‚0" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 cpu‚14.01087855486829' disk.nvme0c0n1.out‚3299.04052734375 disk.rbd2.out‚ 0.4873046875% disk./.usageGB‚1239.6544761657715 disk.dm-0.in‚18.53125 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.recv‚171609 disk.rbd3.in‚0 disk.sda.in‚0 proc.cpu.threads‚29 disk.nvme0n1.in‚18.53125 disk.rbd5.out‚ 0.455078125 disk.sda.out‚ 49.48828125 disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 49.48828125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 0.392578125$ memory_percent‚36.74687147140503) disk./.usagePercent‚83.19425203487076 disk.drbd1743.out‚ 1.19140625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 0.55029296875) proc.memory.availableMB‚ 41453.5703125% disk.nvme0n1.out‚3299.04052734375möÝŸ2Ç- _wandbruntime‚1005  _runtime‚1005íLLSÆÈÀ & lr-AdamW/pg1‚6.053886605398402e-05 & lr-AdamW/pg2‚6.053886605398402e-05 trainer/global_step‚2349 $ _timestamp‚1.785806727569813e+09 _runtime‚1005.861575079 _step‚184§œÂLËÉÅ trainer/global_step‚2349 $ _timestamp‚1.785806727569813e+09 _runtime‚1005.861575079 _step‚184 & lr-AdamW/pg1‚6.053886605398402e-05 & lr-AdamW/pg2‚6.053886605398402e-05¸ï‘á2Ê- _wandbruntime‚1005  _runtime‚1005 Ó41$˳O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40346435748864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123580678144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2067414716928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2062440464420 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24822399 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42413833426432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2067414716928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40438028 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40346418709504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6736393 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6736156 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15615629 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24822399 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚1005.865976144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062354661296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.÷{Z?ˆfreed‚24812743 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40442573718528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15990740 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40428370 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41406304290768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121ÐÅ;e¾'̸O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42413937895424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40346418709504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067502146560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15615627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2067502146560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39343966668832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40346435748864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9254347 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42566129492992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40438028 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40346418709504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6736393 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6736156 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15615629 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24822399 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚1005.865976144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062354661296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24812743 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40442573718528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15990740 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40428370 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41406304290768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40346435748864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123580678144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2067414716928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2062440464420 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24822399 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42413833426432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2067414716928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15622270 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9258282 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _step‚186 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42431879809024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124621143040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063232134518 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39360689801792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42431984285184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24823266 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40455208 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15997287 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15622270 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41424024778906 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 _runtime‚1006.221577128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2068381773312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41423921936310 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40455208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0äˆm²É&ÏÃM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6739005 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40363602511872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2063317937754 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40363585472512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858067279295752e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2068294336512 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40363585472512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9258284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2068381773312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2068294336512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15622272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40459755931648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24832936 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42584244310528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40445536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24832936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40459780835328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124488378880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15622270 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9258282 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _step‚186 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42431879809024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124621143040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063232134518 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39360689801792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42431984285184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24823266 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40455208 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15997287 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15622270 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41424024778906 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 _runtime‚1006.221577128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2068381773312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚41423921936310 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40455208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40180958245472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15947777 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41204773899264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111410852352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2111498281984 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41288522 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43316167712256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9451195 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42287304212760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25350401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25340745 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6880494 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41302929558528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111410852352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25350401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25340745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40180975284832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9451197 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43316272181248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15947779 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚43316167712256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43471688507904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚189 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41204756859904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16331689 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2168914527232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2106328927928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858067515206842e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16331946 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43316272181248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43471844085760 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42287201370276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚43316167712256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43471688507904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚189 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚41204756859904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16331689 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2168914527232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2106328927928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858067515206842e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16331946 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚43316272181248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43471844085760 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42287201370276 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚40180958245472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15947777 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41204773899264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2111410852352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2111498281984 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41288522 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43316167712256 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9451195 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42287304212760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25350401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25340745 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6880494 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41302929558528 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111410852352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25350401 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25340745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40180975284832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121½1¥ý2Û- _wandbruntime‚1030  _runtime‚1030ŽÊpÄ&ܾM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9455134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112290471936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41221940662272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25360938 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43334214094848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25351268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6883619 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40197715457152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43334318571008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15954422 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15954420 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858067520050662e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112290471936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41320111771648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41221923622912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2107206401262 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9455132 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6883370 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2107120598026 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚1030.297280444 _step‚190 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41315360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15954420 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16338502 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40197698417792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43334318571008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41315360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43489794560000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42304819015818 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112377908736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43489952227840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41305688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25351268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42304921858414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15954422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2169682788352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41221940662272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41320136675328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0²T0•ÊÝÃM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41221940662272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41320136675328 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_±?Ôÿ$batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9455134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112290471936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41221940662272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25360938 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚43334214094848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25351268 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6883619 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40197715457152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43334318571008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15954422 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15954420 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858067520050662e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112290471936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41320111771648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚41221923622912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2107206401262 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9455132 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6883370 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2107120598026 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚1030.297280444 _step‚190 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41315360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15954420 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16338502 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚40197698417792 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43334318571008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41315360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43489794560000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42304819015818 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112377908736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43489952227840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41305688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25351268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42304921858414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15954422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2169682788352¾­"2Þ- _wandbruntime‚1030  _runtime‚1030S|…Ãßý 1loss/FBt2/assignment_loss‚0.41443595322493204 /loss/FBt1/detection_loss‚6.7542211810747785 .loss/FBt2/detection_loss‚6.712958464257968 %loss/total_loss‚37.82431549620655 /loss/FRt1/assignment_loss‚4.423798061079449 0loss/SRqqt1/detection_loss‚1.415157060314118 1loss/SRqqt2/detection_loss‚3.6985617446548797 _runtime‚1030.298707852 epoch‚0 2loss/SRqqt2/assignment_loss‚1.8851770925171234 /loss/FRt1/detection_loss‚3.5667536159356437 trainer/global_step‚2399 _step‚191 0loss/FRt2/assignment_loss‚3.8177129689683307 .loss/FRt2/detection_loss‚3.000481178785892 % _timestamp‚1.7858067520067008e+09 2loss/SRqqt1/assignment_loss‚1.6374126035170478 1loss/FBt1/assignment_loss‚0.49764557187639485îsà‚ 1loss/FBt2/assignment_loss‚0.41443595322493204 /loss/FBt1/detection_loss‚6.7542211810747785 .loss/FBt2/detection_loss‚6.712958464257968 %loss/total_loss‚37.82431549620655 /loss/FRt1/assignment_loss‚4.423798061079449 0loss/SRqqt1/detection_loss‚1.415157060314118 1loss/SRqqt2/detection_loss‚3.6985617446548797 _runtime‚1030.298707852 epoch‚0 2loss/SRqqt2/assignment_loss‚1.8851770925171234 /loss/FRt1/detection_loss‚3.5667536159356437 trainer/global_step‚2399 _step‚191 0loss/FRt2/assignment_loss‚3.8177129689683307 .loss/FRt2/detection_loss‚3.000481178785892 % _timestamp‚1.7858067520067008e+09 2loss/SRqqt1/assignment_loss‚1.6374126035170478 1loss/FBt1/assignment_loss‚0.49764557187639485¿NdTœá‚(:‘ ¤ÿÄÓÕ¥†Ó gpu.0.gpu‚26.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚128.121' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.17833333333334 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Çý2â‚(:ö ¤ÿÄÓ¶Œ²× disk.sda.out‚ 51.21484375 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.61328125 disk.rbd3.out‚0 disk.nvme0n1.in‚18.53125 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.4873046875 disk.rbd0.out‚0 disk.sda3.out‚ 51.21484375 disk.rbd1.in‚0 disk.drbd1743.out‚1.21875 disk.rbd3.in‚0 proc.cpu.threads‚29% disk./.usageGB‚1240.1565551757812 disk.rbd5.in‚0 disk.rbd4.in‚0* proc.memory.availableMB‚41454.11328125 cpu‚14.005283650378725) disk./.usagePercent‚83.22794697850487 disk.rbd2.out‚ 0.55029296875 disk.drbd1743.in‚0 disk.dm-0.in‚18.53125$ memory_percent‚36.74604296684265* proc.memory.percent‚17.193293571472168 network.sent‚ 2.253821e+06 network.recv‚174976% disk.nvme0c0n1.out‚3826.736328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚18.53125 disk.dm-0.out‚3826.736328125" proc.memory.rssMB‚ 11267.796875# disk.nvme0n1.out‚3826.736328125 disk.sda1.in‚0 disk.rbd4.out‚ 0.455078125 disk.dm-1.out‚ 51.21484375 disk.sda2.in‚0 disk.sda2.out‚0¸l¸ñœã‚(:‘ ³ÿÄÓòóŒÓ gpu.0.gpu‚25.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚132.148' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚73.41555555555556 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ_×,yä‚(:î ³ÿÄÓûÚ÷Õ disk.sda.out‚51.875 cpu‚14.003075105511169 disk.sda1.out‚0$ memory_percent‚36.74643635749817 network.sent‚ 2.298691e+06 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 0.61328125% disk./.usageGB‚1240.1565780639648 disk.rbd1.out‚ 0.61328125 disk.rbd4.out‚ 0.4873046875' disk.nvme0c0n1.out‚3827.27197265625) disk./.usagePercent‚83.22794851455008 disk.drbd1743.in‚0 disk.rbd5.out‚ 0.4873046875 disk.nvme0c0n1.in‚18.53125 disk.sda3.out‚51.875* proc.memory.availableMB‚41453.85546875" disk.dm-0.out‚3827.27197265625 disk.rbd4.in‚0 proc.cpu.threads‚29 network.recv‚177034 disk.drbd1743.out‚ 1.24609375 disk.nvme0n1.in‚18.53125% disk.nvme0n1.out‚3827.27197265625 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚51.875" proc.memory.rssMB‚ 11267.796875 disk.sda2.in‚0 disk.dm-0.in‚18.53125 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.out‚01[-2å- _wandbruntime‚1053  _runtime‚1053°t^ÇæÁ _runtime‚1053.870529133 _step‚192 & lr-AdamW/pg1‚6.311608470251464e-05 & lr-AdamW/pg2‚6.311608470251464e-05 trainer/global_step‚2449 % _timestamp‚1.7858067755785306e+09ü`-ÌçÆ % _timestamp‚1.7858067755785306e+09 _runtime‚1053.870529133 _step‚192 & lr-AdamW/pg1‚6.311608470251464e-05 & lr-AdamW/pg2‚6.311608470251464e-05 trainer/global_step‚2449À³¯JS2è- _wandbruntime‚1053  _runtime‚1053Ùœ`˺'é´O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43168201292268 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42063112049664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878403 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16671937 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2155406987776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2155406987776 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44218501998080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚1053.872216077 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16672185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44377411700224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2150131588312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41017983900832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _step‚193 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878403 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43168098449784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2155494417408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44218501998080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42163260494848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2214126301696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16279927 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42063112049664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42063095010304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16279929 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7024138 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7023892 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2213989336064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41017966861472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42163285398528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42148674 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2155494417408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44377249830912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42158332 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16279927 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9648045 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0kkr0ê¹O N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42148674 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2155494417408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44377249830912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42158332 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16279927 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9648045 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43168201292268 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42063112049664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878403 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16671937 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2155406987776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2155406987776 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44218501998080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚1053.872216077 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16672185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44377411700224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2150131588312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41017983900832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _step‚193 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25878403 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43168098449784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2155494417408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44218501998080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42163260494848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2214126301696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16279927 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42063112049664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42063095010304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16279929 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7024138 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7023892 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2213989336064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41017966861472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚42163285398528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121Á÷ëóf2ë- _wandbruntime‚1054  _runtime‚1054™ðúÄ&ì¾M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9651982 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42165840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42165840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚2449 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 % _timestamp‚1.7858067760132585e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16286570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2156374044160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42175512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16678727 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2156374044160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16286572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44395518689280 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44236652856832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2215026173952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9651984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16286570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25888940 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42175512 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25888940 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42080261773312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2151009061534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25879270 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25879270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16286572 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2156286607360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43185716095326 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41034724073152 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16678727 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2156374044160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16286572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44395518689280 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44236652856832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2215026173952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9651984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16286570 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25888940 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42175512 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25888940 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42080261773312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2151009061534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25879270 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25879270 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16286572 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2156286607360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚43185716095326 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41034724073152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42165840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚2449 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 % _timestamp‚1.7858067760132585e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16286570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2156374044160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42175512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2194020051820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43008826 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9844897 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26396749 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45282962191872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1081.783004327 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9844895 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41854975477472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚2499 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26406405 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44048995529292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7167039 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44049098371776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17011669 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45120940752896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858068034875464e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2199403123200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45120836283904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42921433160704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43023641238528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45120940752896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45120836283904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42921433160704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43023641238528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45120940752896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2194020051820 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43008826 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9844897 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26396749 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45282962191872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1081.783004327 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9844895 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41854975477472ÅtÿsŽ2û- _wandbruntime‚1082  _runtime‚1082ª @qÃ&ü½M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45138987142656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41871732689152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45138882666496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200282742784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43035664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200370179584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26416942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2194897525042 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42938616963072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43025992 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45301070333952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43025992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194983328278 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16618720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26416942 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚1082.277691711 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2200370179584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7169942 $ _timestamp‚1.785806803985568e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44066716017430 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41871715649792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9848834 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42938599923712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7169683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17018515 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚198 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45300908471808ÁèÉ”ýÂM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194983328278 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16618720 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26416942 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚1082.277691711 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2200370179584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7169942 $ _timestamp‚1.785806803985568e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44066716017430 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41871715649792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9848834 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42938599923712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7169683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17018515 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _step‚198 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45300908471808 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45138987142656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41871732689152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/activX&U¹4 e_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45138882666496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200282742784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43035664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200370179584 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26416942 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2194897525042 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42938616963072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43025992 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45301070333952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43025992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ÆèfØQ2þ- _wandbruntime‚1082  _runtime‚1082ÿ Çêüÿö 0loss/SRqqt1/assignment_loss‚1.66461670210981 2loss/SRqqt2/assignment_loss‚1.9211781436906141 -loss/FRt1/detection_loss‚3.48383447000136 #loss/total_loss‚39.040998217238 % _timestamp‚1.7858068039876544e+09 _runtime‚1082.279224947 0loss/FRt1/assignment_loss‚4.1733421766095695 1loss/SRqqt1/detection_loss‚1.2205508974560242 .loss/FRt2/assignment_loss‚3.91663882845924 epoch‚0 trainer/global_step‚2499 _step‚199 1loss/FBt1/assignment_loss‚0.37676324098073466 1loss/FBt2/assignment_loss‚0.04372718578679269 /loss/FRt2/detection_loss‚3.2268310050879205 /loss/SRqqt2/detection_loss‚3.54611876782249 /loss/FBt1/detection_loss‚10.722900999916924 .loss/FBt2/detection_loss‚4.744495799316516•œ€û 1loss/SRqqt1/detection_loss‚1.2205508974560242 .loss/FRt2/assignment_loss‚3.91663882845924 epoch‚0 trainer/global_step‚2499 _step‚199 1loss/FBt1/assignment_loss‚0.37676324098073466 1loss/FBt2/assignment_loss‚0.04372718578679269 /loss/FRt2/detection_loss‚3.2268310050879205 /loss/SRqqt2/detection_loss‚3.54611876782249 /loss/FBt1/detection_loss‚10.722900999916924 .loss/FBt2/detection_loss‚4.744495799316516 0loss/SRqqt1/assignment_loss‚1.66461670210981 2loss/SRqqt2/assignment_loss‚1.9211781436906141 -loss/FRt1/detection_loss‚3.48383447000136 #loss/total_loss‚39.040998217238 % _timestamp‚1.7858068039876544e+09 _runtime‚1082.279224947 0loss/FRt1/assignment_loss‚4.1733421766095695ÇÙ¥o›‚(: àÿÄÓ–‡‚Ó gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚136.224' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚75.67999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾f]~z‚‚(:ï àÿÄÓ—ÓèÕ disk.sda3.out‚ 53.9609375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚18.53125# disk.nvme0n1.out‚3890.373046875 proc.cpu.threads‚29 disk.sda2.out‚0% disk./.usageGB‚1240.1566467285156 disk.rbd4.out‚ 0.55029296875 disk.nvme0c0n1.in‚18.53125 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 53.9609375 disk.dm-1.out‚ 53.9609375 disk.dm-0.out‚3890.373046875) disk./.usagePercent‚83.22795312268569 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0 disk.drbd1743.out‚1.28125 disk.nvme0n1.in‚18.53125 disk.sda1.out‚0 disk.rbd2.out‚ 0.61328125 disk.drbd1743.in‚0 disk.rbd5.in‚0" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 network.sent‚ 2.348435e+06 network.recv‚181222% disk.nvme0c0n1.out‚3890.373046875 disk.sda1.in‚0 disk.rbd5.out‚ 0.61328125$ memory_percent‚36.74658536911011) proc.memory.availableMB‚ 41453.7578125 cpu‚13.990394682559923çÏãf2ƒ- _wandbruntime‚1106  _runtime‚1106…Uù›Ç„Á % _timestamp‚1.7858068284071534e+09 _runtime‚1106.699066508 _step‚200 & lr-AdamW/pg1‚6.569330335104525e-05 & lr-AdamW/pg2‚6.569330335104525e-05 trainer/global_step‚2549uÊ 8Ì…Æ & lr-AdamW/pg1‚6.569330335104525e-05 & lr-AdamW/pg2‚6.569330335104525e-05 trainer/global_step‚2549 % _timestamp‚1.7858068284071534e+09 _runtime‚1106.699066508 _step‚200È{VH¹2†- _wandbruntime‚1106  _runtime‚1106ýI¢¼'‡¶O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43779771311104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2304375065600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858068284090955e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10041747 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43883997078528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44929892608800 trainer/global_step‚2549 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42692001132832 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42691984093472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46023170569728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2243399258624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46023275038720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46188504915456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17351266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2243486688256 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944229 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26924751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43779771311104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2237994318452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43868978 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44929995451284 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2237908515328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43883972174848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46023170569728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7309521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43779788350464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26924751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43779788350464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7309787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43878636 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2304507836928 _runtime‚1106.704218542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10041745 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43779771311104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2237994318452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43868978 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44929995451284 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2237908515328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43883972174848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46023170569728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7309521 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43779788350464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26924751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43779788350464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7309787 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43878636 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2304507836928 _runtime‚1106.704218542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10041745 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43883997078528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44929892608800 trainer/global_step‚2549 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42692001132832 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42691984093472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46023170569728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2243399258624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÉNñ¥2‰- _wandbruntime‚1107  _runtime‚1107[«}›€ŠÀM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43796938074112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2244278878208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2244278878208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43796955113472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2305408878592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2238785988550 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2244366315008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17358236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46206459600384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26944944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26944944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44947510254342 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚1188áåŸ5F7 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚2549 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16950870 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46206613073920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43895816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16950872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7312309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16950872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045684 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44947613096938 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43886144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17357991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43895816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251¥{WË&‹ÅM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43796955113472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2305408878592 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2238785988550 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2244366315008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17358236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46206459600384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26944944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26944944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44947510254342 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚2549 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16950870 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46206613073920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43895816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16950872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7312309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16950872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10045684 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44947613096938 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43886144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17357991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43895816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43796938074112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2244278878208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2244278878208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46925504855552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44638126500864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7452388 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17276377 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44638109461504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47093897050112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10238595 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2287482823680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2281796978836 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2599 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44638126500864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46925609324544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44744328014848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43528992709472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43529009748832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17276379 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46925504855552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7452629 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚205 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858068556732988e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27452753 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47094050530816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2287482823680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17276377 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _runtime‚1133.969042501 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2281882781960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚47093897050112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10238595 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2287482823680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2281796978836 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2599 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44638126500864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46925609324544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44744328014848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43528992709472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43529009748832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17276379 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46925504855552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7452629 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚wÔtl205 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858068556732988e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27452753 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47094050530816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2287482823680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17276377 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _runtime‚1133.969042501 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2281882781960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46925504855552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2Ícà‰î2™- _wandbruntime‚1134  _runtime‚1134Ÿ²‚Å&š¿M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2288362450432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43545749921152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44761535131648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2288362450432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43545732881792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2282760255294 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17697754 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44755968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17283020 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚1134.352656206 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17283022 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44746296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2350472254976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463276 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7455222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17283020 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44655293263872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46943551238144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47112007386624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2599 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44761560035328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44746296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17697990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45828407333850 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46943655714304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44655276224512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7455456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2288275013632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27472946 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27472946 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚206 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.785806856060342e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44755968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463276 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44655276224512n~KqÊ&›ÄM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27472946 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚206 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.785806856060342e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44755968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27463276 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44655276224512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2288362450432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43545749921152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44761535131648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2288362450432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43545732881792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2282760255294 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17697754 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44755968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17283020 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚1134.352656206 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17283022 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44746296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2350472254976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27463276 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7455222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17283020 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44655293263872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚46943551238144 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47112007386624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2599 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44761560035328 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44746296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17697990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45828407333850 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46943655714304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44655276224512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7455456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2288275013632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27472946 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800Îÿy"12œ- _wandbruntime‚1134  _runtime‚11345N<ÿù -loss/FBt1/detection_loss‚6.24429011841615 -loss/FBt2/detection_loss‚4.62610924373947 _step‚207 /loss/FRt2/detection_loss‚2.9696158626106346 0loss/SRqqt2/detection_loss‚3.185943857255116 0loss/FRt2/assignment_loss‚3.8178037980969033 2loss/SRqqt2/assignment_loss‚1.7353860803053414 /loss/FRt1/detection_loss‚3.4005328895340505 $loss/total_loss‚33.3127058414105 trainer/global_step‚2599 epoch‚0 $ _timestamp‚1.785806856062087e+09 _runtime‚1134.354072062 0loss/FRt1/assignment_loss‚4.3481979823448285 2loss/SRqqt1/assignment_loss‚1.4177176088935906 0loss/FBt1/assignment_loss‚0.4315428845050822 1loss/FBt2/assignment_loss‚0.03206131650464764 1loss/SRqqt1/detection_loss‚1.1035041992046855ªdk|žþ 0loss/FBt1/assignment_loss‚0.4315428845050822 1loss/FBt2/assignment_loss‚0.03206131650464764 1loss/SRqqt1/detection_loss‚1.1035041992046855 -loss/FBt1/detection_loss‚6.24429011841615 -loss/FBt2/detection_loss‚4.62610924373947 _step‚207 /loss/FRt2/detection_loss‚2.9696158626106346 0loss/SRqqt2/detection_loss‚3.185943857255116 0loss/FRt2/assignment_loss‚3.8178037980969033 2loss/SRqqt2/assignment_loss‚1.7353860803053414 /loss/FRt1/detection_loss‚3.4005328895340505 $loss/total_loss‚33.3127058414105 trainer/global_step‚2599 epoch‚0 $ _timestamp‚1.785806856062087e+09 _runtime‚1134.354072062 0loss/FRt1/assignment_loss‚4.3481979823448285 2loss/SRqqt1/assignment_loss‚1.4177176088935906Ïä‚^œŸ‚(:‘ €ÅÓ¡ÅÀÓ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚37.0 gpu.0.powerWatts‚100.785' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚55.99166666666666 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#¢È‰ ‚(:þ €ÅÓÉŽ¦× disk.sda.out‚ 56.0390625 disk.rbd2.out‚ 0.61328125 disk.sda3.out‚ 56.0390625 disk.rbd3.out‚0# memory_percent‚36.7475688457489 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 56.0390625 disk.sda2.out‚0 disk.rbd5.in‚0" proc.memory.rssMB‚ 11267.796875 disk.sda1.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 18.53515625 disk.sda3.in‚0 network.recv‚186068 disk.sda1.in‚0 disk.rbd5.out‚ 0.61328125 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 network.sent‚ 2.440252e+06 disk.nvme0n1.in‚ 18.53515625% disk.nvme0n1.out‚3953.96240234375 disk.rbd1.out‚ 0.61328125 disk.drbd1743.in‚0 disk.drbd1743.out‚ 1.35546875' disk.nvme0c0n1.out‚3953.96240234375 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 18.53515625" disk.dm-0.out‚3953.96240234375 cpu‚13.980011295448945% disk./.usageGB‚1240.1580276489258 disk.rbd4.out‚ 0.61328125* proc.memory.availableMB‚41453.11328125* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29 disk.sda2.in‚0) disk./.usagePercent‚83.22804579741309¶!âúš¡‚(: œ€ÅÓ·ÆþÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.37777777777778 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>üŸ;¢‚(:ö œ€ÅÓâÀ—×! disk.nvme0c0n1.in‚ 18.53515625 disk.sda.out‚56.6875 disk.rbd2.in‚0 disk.drbd1743.out‚ 1.35546875" disk.dm-0.out‚3954.17724609375 disk.nvme0n1.in‚ 18.53515625 network.sent‚ 2.485175e+06 disk.sda1.out‚0 disk.rbd4.in‚0' disk.nvme0c0n1.out‚3954.17724609375 disk.sda.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚3954.17724609375* proc.memory.availableMB‚41453.36328125) disk./.usagePercent‚83.22804579741309 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 0.61328125 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 11267.796875 disk.sda1.in‚0 disk.rbd5.out‚ 0.61328125 disk.sda3.out‚56.6875* proc.memory.percent‚17.193293571472168 cpu‚13.976763194948736 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 proc.cpu.threads‚29 disk.rbd3.in‚0$ memory_percent‚36.74718737602234% disk./.usageGB‚1240.1580276489258 disk.rbd2.out‚ 0.61328125 disk.dm-1.out‚56.6875 disk.rbd1.in‚0 disk.sda2.out‚0 network.recv‚188224 disk.dm-0.in‚ 18.53515625\dù‚2£- _wandbruntime‚1158  _runtime‚1158}ǤÁ & lr-AdamW/pg2‚6.827052199957586e-05 trainer/global_step‚2649 % _timestamp‚1.7858068805848374e+09 _runtime‚1158.876543504 _step‚208 & lr-AdamW/pg1‚6.827052199957586e-05 aMÌ¥Æ % _timestamp‚1.7858068805848374e+09 _runtime‚1158.876543504 _step‚208 & lr-AdamW/pg1‚6.827052199957586e-05 & lr-AdamW/pg2‚6.827052199957586e-05 trainer/global_step‚2649ÐÀýR]2¦- _wandbruntime‚1158  _runtime‚1158jͧ÷§¶O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1158.880038911 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27980755 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18030623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44366018364832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45598940 trainer/global_step‚2649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27990411 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45589282 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47827943610368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47827839141376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2331391529472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17608529 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7858068805859928e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45496464651264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47827839141376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27990411 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10435445 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2331391529472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27980755 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17608527 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7594934 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2394764252160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44366001325472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2325771245468 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45496464651264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45496447611904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚209 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17608529 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45496447611904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46691686767816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331478959104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2394890732032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0´–‰Á'¨»O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45589282 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47827943610368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47827839141376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2331391529472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17608529 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7858068805859928e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45496464651264 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47827839141376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27990411 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10435445 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2331391529472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27980755 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17608527 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7594934 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2394764252160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44366001325472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2325771245468 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45496464651264 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45496447611904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚209 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17608529 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45496447611904 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46691686767816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331478959104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2394890732032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1158.880038911 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27980755 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18030623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44366018364832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45598940 trainer/global_step‚2649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27990411ÑõZ¿ô2©- _wandbruntime‚1159  _runtime‚1159Èþ)Ä&ª¾M JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7597701 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48017718978560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2326648718802 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332358585856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18037083 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45621890971648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17615172 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45606448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7597943 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45621915875328 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46709407255954 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45616120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28000948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2395803103232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44382741497792 trainer/global_step‚2649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27991278 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45513631414272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45513614374912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2395661950464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 $ _timestamp‚1.785806880991747e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10439384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚210 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45513614374912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45616120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10439382 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2332271149056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44382758537152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45606448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17615172 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46709304413358» Ða «ÃM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27991278 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45513631414272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45513614374912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2395661950464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 $ _timestamp‚1.785806880991747e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10439384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚210 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45513614374912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45616120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10439382 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2332271149056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44382758537152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45606448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17615172 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46709304413358 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7597701 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48017718978560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2326648718802 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332358585856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18037083 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45621890971648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17615172 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45606448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7597943 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45621915875328 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46709407255954 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45616120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28000948 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2395803103232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44382741497792 trainer/global_step‚2649ÒiÃ+2¬- _wandbruntime‚1159  _runtime‚1159äo@.­ÿ /loss/FRt1/detection_loss‚3.3245096473644176 % _timestamp‚1.7858068809932775e+09 2loss/SRqqt1/assignment_loss‚1.5552129882077377 /loss/FRt2/detection_loss‚2.9557257468780773 &loss/total_loss‚50.495690422993235 _runtime‚1159.285074444 0loss/FBt1/assignment_loss‚0.4873418119714188 2loss/FBt2/assignment_loss‚0.015776220448253407 1loss/SRqqt2/detection_loss‚3.2289785175687737 /loss/FRt1/assignment_loss‚4.322086268001133 2loss/SRqqt2/assignment_loss‚1.7943922037051783 1loss/SRqqt1/detection_loss‚1.3924712069869176 /loss/FBt1/detection_loss‚20.764440512657167 .loss/FBt2/detection_loss‚6.722600499089151 epoch‚0 trainer/global_step‚2649 _step‚211 /loss/FRt2/assignment_loss‚3.932154800115007áq`ò ®„ 2loss/FBt2/assignment_loss‚0.015776220448253407 1loss/SRqqt2/detection_loss‚3.2289785175687737 /loss/FRt1/assignment_loss‚4.322086268001133 2loss/SRqqt2/assignment_loss‚1.7943922037051783 1loss/SRqqt1/detection_loss‚1.3924712069869176 /loss/FBt1/detection_loss‚20.764440512657167 .loss/FBt2/detection_loss‚6.722600499089151 epoch‚0 trainer/global_step‚2649 _step‚211 /loss/FRt2/assignment_loss‚3.932154800115007 /loss/FRt1/detection_loss‚3.3245096473644176 % _timestamp‚1.7858068809932775e+09 2loss/SRqqt1/assignment_loss‚1.5552129882077377 /loss/FRt2/detection_loss‚2.9557257468780773 &loss/total_loss‚50.495690422993235 _runtime‚1159.285074444 0loss/FBt1/assignment_loss‚0.4873418119714188ÓÝÜ®¯œ¯‚(:‘ «€ÅÓ¥ãÒ gpu.0.gpu‚24.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚146.471' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.37277777777778 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãú §…°‚(:ú «€ÅÓ鄲Ø disk.sda1.in‚0& disk.nvme0c0n1.out‚4016.9970703125 disk.sda.out‚ 57.4296875 disk.dm-1.out‚ 57.4296875 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.sda1.out‚0 disk.rbd5.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.61328125* proc.memory.availableMB‚41453.04296875" proc.memory.rssMB‚ 11267.796875 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 0.61328125 disk.rbd1.in‚0 disk.drbd1743.out‚ 1.3828125% disk./.usageGB‚1239.9746360778809 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 18.53515625 disk.rbd5.out‚ 0.61328125 disk.rbd4.out‚ 0.61328125 disk.sda3.out‚ 57.4296875 disk.dm-0.in‚ 18.53515625! disk.dm-0.out‚4016.9970703125 disk.rbd3.in‚0# memory_percent‚36.7476761341095 proc.cpu.threads‚29 network.recv‚189312 disk.rbd0.out‚0* proc.memory.percent‚17.193293571472168 cpu‚13.975433627799958$ disk.nvme0n1.out‚4016.9970703125! disk.nvme0c0n1.in‚ 18.53515625 disk.dm-1.in‚0 disk.rbd3.out‚0 network.sent‚ 2.487605e+06) disk./.usagePercent‚83.21573823520448xY8u2±- _wandbruntime‚1182  _runtime‚1182WíƒiƲÀ & lr-AdamW/pg1‚6.955913132384115e-05 & lr-AdamW/pg2‚6.955913132384115e-05 trainer/global_step‚2699 $ _timestamp‚1.785806903879421e+09 _runtime‚1182.171191247 _step‚212Ô°£Ê˳Š_step‚212 & lr-AdamW/pg1‚6.955913132384115e-05 & lr-AdamW/pg2‚6.955913132384115e-05 trainer/global_step‚2699 $ _timestamp‚1.785806903879421e+09 _runtime‚1182.171191247ÔäÀ“ª2´- _wandbruntime‚1182  _runtime‚1182IHÎè¼'µ¶O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48730173427200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46459092 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46465064598528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47572583847324 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18370780 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2440072867840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48904983985664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48730277896192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7738483 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46354802801664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46354785762304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7738238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17940677 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46354785762304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858069038806689e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2369659708976 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48730277896192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28508757 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48730173427200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚213 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46465039694848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46449434 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48905137466368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18370533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45203009941472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2375475094528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17940679 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45203026980832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2375387664896 _runtime‚1182.175939935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17940679 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17940677 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚2699 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2375387664896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439944290816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28508757 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46354802801664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10632295 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48730173427200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46459092 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46465064598528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47572583847324 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18370780 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2440072867840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48904983985664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48730277896192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7738483 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46354802801664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46354785762304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7738238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17940677 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46354785762304 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858069038806689e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2369659708976 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48730277896192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28508757 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48730173427200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚213 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46465039694848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46449434 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48905137466368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18370533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45203009941472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2375475094528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17940679 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45203026980832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2375387664896 _runtime‚1182.175939935 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17940679 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17940677 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚2699 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2375387664896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439944290816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28508757 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46354802801664 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10632295 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46476272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17947322 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47590304335462 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28528950 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2376267284480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18377507 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17947322 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48748324285952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46371952525312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2376267284480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46482246811648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17947320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47² ‰¿–590201492866 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46466600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46476272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858069042767522e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10636232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28519280 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7741273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45219750113792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2370451379074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18377220 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48748324285952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2440835804160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440979054080 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48923250769408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17947320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2699•êˆË&¹ÅM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17947320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47590201492866 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46466600 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46476272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858069042767522e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10636232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28519280 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7741273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45219750113792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2370451379074 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18377220 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48748324285952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2440835804160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2440979054080 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48923250769408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17947320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28528950 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28519280 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48748219809792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46476272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17947322 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47590304335462 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28528950 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2376267284480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18377507 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17947322 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48748324285952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46371952525312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2376267284480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46482246811648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ÖCFž2º- _wandbruntime‚1182  _runtime‚1182 ¦û»þ 2loss/SRqqt1/assignment_loss‚1.6462676775110654 1loss/SRqqt1/detection_loss‚1.3583882448782578 &loss/total_loss‚30.957929481610442 trainer/global_step‚2699 .loss/FRt1/detection_loss‚3.039203215535585 /loss/FRt2/detection_loss‚2.7275601631587314 .loss/FBt1/detection_loss‚7.159340898195903 /loss/FRt1/assignment_loss‚4.219283074527592 2loss/SRqqt2/assignment_loss‚1.8231302681325496 2loss/FBt2/assignment_loss‚0.008721205685362339 1loss/SRqqt2/detection_loss‚2.4711443709916083 /loss/FBt2/detection_loss‚2.1497343180004878 % _timestamp‚1.7858069042781844e+09 _runtime‚ 1182.56973101 0loss/FRt2/assignment_loss‚3.7424452022178887 0loss/FBt1/assignment_loss‚0.6127108427754138 _step‚215 epoch‚0ì” ¼ƒ 1loss/SRqqt2/detection_loss‚2.4711443709916083 /loss/FBt2/detection_loss‚2.1497343180004878 % _timestamp‚1.7858069042781844e+09 _runtime‚ 1182.56973101 0loss/FRt2/assignment_loss‚3.7424452022178887 0loss/FBt1/assignment_loss‚0.6127108427754138 _step‚215 epoch‚0 2loss/SRqqt1/assignment_loss‚1.6462676775110654 1loss/SRqqt1/detection_loss‚1.3583882448782578 &loss/total_loss‚30.957929481610442 trainer/global_step‚2699 .loss/FRt1/detection_loss‚3.039203215535585 /loss/FRt2/detection_loss‚2.7275601631587314 .loss/FBt1/detection_loss‚7.159340898195903 /loss/FRt1/assignment_loss‚4.219283074527592 2loss/SRqqt2/assignment_loss‚1.8231302681325496 2loss/FBt2/assignment_loss‚0.008721205685362339׸\SØš½‚(: º€ÅÓ½ÿ¢Ó gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚147.83' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚82.1277777777778 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:øˆ¾‚(:ý º€ÅÓêæíÖ network.sent‚ 2.53247e+06% disk./.usageGB‚1239.9404335021973 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.rbd5.out‚ 0.61328125 disk.rbd4.in‚0& disk.nvme0c0n1.out‚4798.3095703125" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.out‚1.390625! disk.dm-0.out‚4798.3095703125$ disk.nvme0n1.out‚4798.3095703125 disk.rbd0.out‚0) disk./.usagePercent‚83.21344287165249 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda3.out‚ 58.23046875* proc.memory.availableMB‚41452.32421875 disk.rbd2.in‚0 cpu‚13.973807741886661 disk.dm-0.in‚ 18.53515625 disk.rbd5.in‚0 disk.sda3.in‚0 disk.sda2.out‚0% memory_percent‚36.748772859573364 network.recv‚191276 disk.rbd4.out‚ 0.61328125! disk.nvme0c0n1.in‚ 18.53515625 disk.sda.out‚ 58.23046875 disk.rbd1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 18.53515625 disk.rbd2.out‚ 0.61328125 disk.dm-1.out‚ 58.23046875 disk.rbd1.out‚ 0.61328125* proc.memory.percent‚17.193293571472168ñ'®œ¿‚(:‘ É€ÅÓɳïÒ gpu.0.gpu‚25.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚129.479' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.93277777777779 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú¶ ‰À‚(:þ É€ÅÓ†ªÕ network.recv‚193206% disk./.usageGB‚1239.9403762817383 disk.rbd0.in‚0 disk.rbd4.out‚ 0.61328125 disk.sda3.in‚0 disk.sda2.in‚0* proc.memory.availableMB‚41455.91015625* proc.memory.percent‚17.193293571472168 disk.sda1.out‚0 proc.cpu.threads‚29! disk.dm-0.out‚4798.6318359375! disk.nvme0c0n1.in‚ 18.54296875 disk.sda.out‚ 58.76171875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.61328125 disk.dm-0.in‚ 18.54296875 disk.rbd3.in‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚4798.6318359375 disk.rbd1.out‚ 0.61328125 cpu‚13.97241196847812 network.sent‚ 2.577028e+06 disk.rbd3.out‚0$ disk.nvme0n1.out‚4798.6318359375 disk.rbd5.out‚ 0.61328125 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 58.76171875 disk.rbd1.in‚0) disk./.usagePercent‚83.21343903153947 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 18.54296875 disk.rbd5.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.out‚ 1.41796875 disk.sda.in‚0$ memory_percent‚36.74330115318298 disk.rbd4.in‚0 disk.sda3.out‚ 58.76171875ÞãС2Á- _wandbruntime‚1205  _runtime‚1205+qƒŒÆÂÀ & lr-AdamW/pg1‚7.084774064810645e-05 & lr-AdamW/pg2‚7.084774064810645e-05 trainer/global_step‚2749 % _timestamp‚1.7858069270011878e+09 _runtime‚ 1205.29303767 _step‚2161.Â_ËÃÅ & lr-AdamW/pg1‚7.084774064810645e-05 & lr-AdamW/pg2‚7.084774064810645e-05 trainer/global_step‚2749 % _timestamp‚1.7858069270011878e+09 _runtime‚ 1205.29303767 _step‚216ØBz{~2Ä- _wandbruntime‚1205  _runtime‚1205 NXßÂ%ŵO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49632507713024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419471229952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49632507713024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49810634527232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2413548172484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419471229952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46040018557472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18272827 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47213140952064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2419383800320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7880891 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48453583769316 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18272829 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47213123912704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29046415 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47319244 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47213123912704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49810478949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7880629 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47325395534848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47213140952064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47325420438528 $ _timestamp‚1.785806927002512e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48453480926832 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TGDeviceStatsMonitor.on_train_batch_start/allocation.ls<‚iùarge_pool.allocated‚18272829 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29046415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10829147 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2413462369360 trainer/global_step‚2749#¦LÀ'ƺO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29036759 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10829145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚1205.294537414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2419383800320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2485214088704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46040035596832 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚217 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49632612182016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29036759 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47309586 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49632507713024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2419471229952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49632507713024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49810634527232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2413548172484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2419471229952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46040018557472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18272827 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47213140952064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2419383800320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7880891 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48453583769316 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18272829 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47213123912704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29046415 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47319244 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47213123912704 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49810478949376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7880629 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47325395534848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47213140952064 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47325420438528 $ _timestamp‚1.785806927002512e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48453480926832 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18272829 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29046415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10829147 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2413462369360 trainer/global_step‚2749 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10833082 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29047282 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47230290675712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47326752 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10833084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2486116056064 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18279472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48471098572374 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46056758729792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29056952 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49650554095616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47342602651648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2485983291904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47336424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 _runtime‚1205.692998944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47230290675712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716645 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49650658571776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2420350856704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7883563 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47230307715072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18279472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47326752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47230307715072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18279470 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420263419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49828743611392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.785806927401038e+09 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48471201414970 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2414425645818 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47336424 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856v6Ê&ÉÄM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49650658571776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2420350856704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7883563 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47230307715072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18279472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47326752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47230307715072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18279470 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2420263419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49828743611392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.785806927401038e+09 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚48471201414970 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2414425645818 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47336424 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47342627555328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414339842582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7883809 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46056775769152 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10833082 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29047282 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47230290675712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47326752 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10833084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2486116056064 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18279472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚48471098572374 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46056758729792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29056952 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49650554095616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47342602651648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2485983291904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47336424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 _runtime‚1205.692998944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47230290675712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18716645 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032Ú´ù®J2Ê- _wandbruntime‚1205  _runtime‚1205û¹%ÿËù trainer/global_step‚2749 /loss/FRt2/assignment_loss‚3.560768877665202 1loss/FBt2/assignment_loss‚0.03823044402992713 /loss/FRt1/detection_loss‚3.5088338609506833 _runtime‚1205.693899073 _step‚219 1loss/FBt1/assignment_loss‚0.46619142241090433 1loss/SRqqt1/detection_loss‚1.1671374850760463 /loss/SRqqt2/detection_loss‚4.05917586443516 .loss/FBt2/detection_loss‚5.006688612845431 %loss/total_loss‚37.19054190158825 % _timestamp‚1.7858069274024267e+09 /loss/FRt1/assignment_loss‚4.528779983520508 0loss/SRqqt2/assignment_loss‚2.06481104859939 .loss/FBt1/detection_loss‚7.848798639244503 epoch‚0 2loss/SRqqt1/assignment_loss‚1.5974865012237283 /loss/FRt2/detection_loss‚3.3436391615867613m¹íYÌþ _runtime‚1205.693899073 _step‚219 1loss/FBt1/assignment_loss‚0.46619142241090433 1loss/SRqqt1/detection_loss‚1.1671374850760463 /loss/SRqqt2/detection_loss‚4.05917586443516 .loss/FBt2/detection_loss‚5.006688612845431 %loss/total_loss‚37.19054190158825 % _timestamp‚1.7858069274024267e+09 /loss/FRt1/assignment_loss‚4.528779983520508 0loss/SRqqt2/assignment_loss‚2.06481104859939 .loss/FBt1/detection_loss‚7.848798639244503 epoch‚0 2loss/SRqqt1/assignment_loss‚1.5974865012237283 /loss/FRt2/detection_loss‚3.3436391615867613 trainer/global_step‚2749 /loss/FRt2/assignment_loss‚3.560768877665202 1loss/FBt2/assignment_loss‚0.03823044402992713 /loss/FRt1/detection_loss‚3.5088338609506833ÛH›œÍ‚(:‘ Ø€ÅÓ¹Ó gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.13277777777778 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âþ‰Ão΂(:÷ Ø€ÅÓ¦«¶Ø disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd1743.out‚ 1.41796875! disk.dm-¼*œ·0.out‚4799.2470703125 disk.sda1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 18.54296875) disk./.usagePercent‚83.21345183191619 disk.rbd2.in‚0 disk.rbd2.out‚ 0.61328125 disk.dm-0.in‚ 18.54296875 disk.rbd3.in‚0* proc.memory.percent‚17.193293571472168 cpu‚13.96970838550492 proc.cpu.threads‚29& disk.nvme0c0n1.out‚4799.2470703125% disk./.usageGB‚1239.9405670166016 disk.sda3.out‚ 59.546875 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd4.in‚0" proc.memory.rssMB‚ 11267.796875 disk.sda2.out‚0$ memory_percent‚36.74346208572388 network.sent‚ 2.579615e+06 network.recv‚194149 disk.rbd1.out‚ 0.61328125 disk.nvme0n1.in‚ 18.54296875$ disk.nvme0n1.out‚4799.2470703125 disk.rbd5.out‚ 0.61328125 disk.sda.in‚0 disk.sda.out‚ 59.546875 disk.sda3.in‚0 disk.dm-1.out‚ 59.546875 disk.rbd3.out‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.drbd1743.in‚0 disk.rbd4.out‚ 0.61328125) proc.memory.availableMB‚ 41455.8046875Ž Z'œÏ‚(:‘ ç€ÅÓå–©Ô gpu.0.gpu‚26.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚134.865' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚74.92500000000001 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝœÝr‰Ð‚(:þ ç€ÅÓÀŸ¾Ú disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 41456.0234375 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.rbd4.in‚0 disk.dm-1.in‚0% memory_percent‚36.743128299713135 network.recv‚196326 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd4.out‚ 0.61328125 disk.sda.out‚ 60.24609375 disk.rbd2.in‚0 disk.rbd2.out‚ 0.61328125 disk.sda3.in‚0 network.sent‚ 2.624421e+06 disk.nvme0n1.in‚ 18.54296875! disk.nvme0c0n1.in‚ 18.54296875& disk.nvme0c0n1.out‚4800.1982421875" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168% disk./.usageGB‚1240.4426383972168 disk.rbd3.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 0.61328125 disk.rbd0.in‚0) disk./.usagePercent‚83.24714626353523$ disk.nvme0n1.out‚4800.1982421875 disk.sda3.out‚ 60.24609375 disk.rbd1.in‚0! disk.dm-0.out‚4800.1982421875 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 60.24609375 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0 cpu‚13.966806561452186 disk.drbd1743.out‚ 1.4453125 disk.dm-0.in‚ 18.54296875ÃÁÜ 2Ñ- _wandbruntime‚1231  _runtime‚1231[ÂÛEÇÒÁ _step‚220 & lr-AdamW/pg1‚7.213634997237176e-05 & lr-AdamW/pg2‚7.213634997237176e-05 trainer/global_step‚2799 % _timestamp‚1.7858069530367565e+09 _runtime‚1231.328457797 üyÌÓÆ trainer/global_step‚2799 % _timestamp‚1.7858069530367565e+09 _runtime‚1231.328457797 _step‚220 & lr-AdamW/pg1‚7.213634997237176e-05 & lr-AdamW/pg2‚7.213634997237176e-05Ü_Xwÿ2Ô- _wandbruntime‚1231  _runtime‚1231#§óx¼'Õ¶O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50715975805952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2457436635992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18604977 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50534841998848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48071462063104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11025995 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48169738 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48071462063104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48071479102464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49334378006340 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50534946467840 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48071479102464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2457350832868 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚50534841998848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48179396 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024055 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 trainer/global_step‚2799 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _runtime‚1231.332996601 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29574417 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18604979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29564761 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48185751374848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49334480848824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463467365376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 % _timestamp‚1.7858069530378528e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29564761 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46877044212832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18604979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463379935744 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8023817 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50534841998848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48071462063104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11025995 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48169738 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48071462063104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48071479102464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49334378006340 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50534946467840 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48071479102464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2457350832868 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚50534841998848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48179396 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024055 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 trainer/global_step‚2799 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _runtime‚1231.332996601 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29574417 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18604979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29564761 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48185751374848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚49334480848824 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2463467365376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 % _timestamp‚1.7858069530378528e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29564761 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46877044212832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18604979 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992Ý+/îµ2×- _wandbruntime‚1231  _runtime‚1231XÊÁÕ#ؾM E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1231.720037624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18611620 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48196576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50734248569856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18611622 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48088628826112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48088645865472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464259555328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18611620 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48202958491648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48088628826112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18611622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11029932 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026885 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2531265174528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2464346992128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46893767345792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50552992857600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11029934 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46893784385152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2464346992128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531130313216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49351995651882 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48186904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2458228306090 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49352098494478 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19056819 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 trainer/global_step‚2799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29584954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48186904 VíIÍäïCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50552992857600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575284 # _timestamp‚1.78580695342795e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48202983395328,˜-É&ÙÃM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48088645865472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464259555328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18611620 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48202958491648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48088628826112 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18611622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11029932 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026885 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2531265174528 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2464346992128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46893767345792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50552992857600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11029934 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46893784385152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2464346992128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2531130313216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49351995651882 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48186904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2458228306090 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49352098494478 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19056819 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 trainer/global_step‚2799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29584954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48186904 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚50552992857600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575284 # _timestamp‚1.78580695342795e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48202983395328 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1231.720037624 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18611620 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48196576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50734248569856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18611622 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48088628826112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Þ©Û¢Ë2Ú- _wandbruntime‚1231  _runtime‚1231ÎjÝÿÛù .loss/FRt1/detection_loss‚3.173665193013019 /loss/FRt2/detection_loss‚3.3086884673436483 1loss/SRqqt1/detection_loss‚1.3882231363762993 %loss/total_loss‚34.44051582308594 _runtime‚1231.720792389 epoch‚0 /loss/FRt2/assignment_loss‚3.784991677602132 0loss/SRqqt2/detection_loss‚2.478868392074392 trainer/global_step‚2799 % _timestamp‚1.7858069534293272e+09 2loss/SRqqt1/assignment_loss‚1.5772293590401347 _step‚223 .loss/FRt1/assignment_loss‚4.15267047782739 1loss/SRqqt2/assignment_loss‚1.527433458893072 0loss/FBt1/assignment_loss‚0.6367712281644344 1loss/FBt2/assignment_loss‚0.03377504999517602 .loss/FBt1/detection_loss‚8.191516697406769 /loss/FBt2/detection_loss‚4.1866826853494725wëýÜþ .loss/FRt1/assignment_loss‚4.15267047782739 1loss/SRqqt2/assignment_loss‚1.527433458893072 0loss/FBt1/assignment_loss‚0.6367712281644344 1loss/FBt2/assignment_loss‚0.03377504999517602 .loss/FBt1/detection_loss‚8.191516697406769 /loss/FBt2/detection_loss‚4.1866826853494725 .loss/FRt1/detection_loss‚3.173665193013019 /loss/FRt2/detection_loss‚3.3086884673436483 1loss/SRqqt1/detection_loss‚1.3882231363762993 %loss/total_loss‚34.44051582308594 _runtime‚1231.720792389 epoch‚0 /loss/FRt2/assignment_loss‚3.784991677602132 0loss/SRqqt2/detection_loss‚2.478868392074392 trainer/global_step‚2799 % _timestamp‚1.7858069534293272e+09 2loss/SRqqt1/assignment_loss‚1.5772293590401347 _step‚223ßÃ2No›Ý‚(: ö€ÅÓèö„Ó gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚68.7761111111111 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¸ ‚Þ‚(:÷ ö€ÅÓë—»Ö disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 60.984375) disk./.usagePercent‚83.24700597140647 disk.rbd1.out‚ 0.61328125 disk.rbd5.out‚ 0.61328125 disk.sda.in‚0 cpu‚13.96475235053813 disk.rbd1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 18.54296875 disk.sda1.in‚0 disk.sda3.in‚0* proc.memory.availableMB‚41455.97265625 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 0.61328125* proc.memory.percent‚17.193293571472168! disk.dm-0.out‚5099.3115234375% disk./.usageGB‚1240.4405479431152 disk.dm-0.in‚ 18.54296875! disk.nvme0c0n1.in‚ 18.54296875& disk.nvme0c0n1.out‚5099.3115234375 disk.dm-1.in‚0$ memory_percent‚36.74320578575134 network.sent‚ 2.626903e+06 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.rbd2.out‚ 0.61328125" proc.memory.rssMB‚ 11267.796875 proc.cpu.threads‚29 network.recv‚197324 disk.drbd1743.out‚ 1.4453125$ disk.nvme0n1.out‚5099.3115234375 disk.sda1.out‚0 disk.rbd5.in‚0 disk.sda.out‚ 60.984375 disk.dm-1.out‚ 60.984375 disk.rbd3.out‚0 disk.rbd0.out‚05Ïîý2ß- _wandbruntime‚1255  _runtime‚1255bOÛÇàÁ & lr-AdamW/pg2‚7.342495929663707e-05 trainer/global_step‚2849 % _timestamp‚1.7858069767997694e+09 _runtime‚1255.091532318 _step‚224 & lr-AdamW/pg1‚7.342495929663707e-05!ƒm.ÌáÆ % _timestamp‚1.7858069767997694e+09 _runtime‚1255.091532318 _step‚224 & lr-AdamW/pg1‚7.342495929663707e-05 & lr-AdamW/pg2‚7.342495929663707e-05 trainer/global_step‚2849àqYñ2â- _wandbruntime‚1255  _runtime‚12557/@š¼'ã¶O `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49046107214848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18937129 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30092763 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30102419 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2507376071168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50215377928332 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47714035789472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2575411160576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8167682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11222847 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2501239296376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49039548 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51621680244736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18937127 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2507463500800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30092763 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19390796 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19390527 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2575548126208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47714052828832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11222845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51437176284672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51437280753664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48929800213504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8167949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2507463500800 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51437176284672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51437280753664 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48929800213504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8167949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2507463500800 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50215377928332 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47714035789472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2849 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2575411160576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8167682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11222847 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2501239296376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49039548 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51621680244736 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18937127 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2507463500800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30092763 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19390796 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19390527 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2575548126208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47714052828832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11222845ኘyë2å- _wandbruntime‚1255  _runtime‚1255bžú`Å&æ¿M FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48946966976512 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49056728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19397267 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11226784 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51639786349056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49063339235328 _step‚226 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502202572834 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8170746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51639626584064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50232995573986 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1255.497511424 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51455327143424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19397530 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48946966976512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2508343127552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18943770 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49047056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51455222667264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8170485 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11226782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2576312252416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51455327143424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48946984015872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502116769598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18943772 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30103286 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2576447113728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785806977205312e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49056728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50232892731390 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚01îÊ&çÄM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2576447113728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785806977205312e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49056728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50232892731390 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48946966976512 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49056728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19397267 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11226784 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51639786349056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49063339235328 _step‚226 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2502202572834 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8170746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51639626584064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50232995573986 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1255.497511424 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51455327143424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19397530 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚48946966976512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2508343127552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18943770 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49047056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51455222667264 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8170485 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11226782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2576312252416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51455327143424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48946984015872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2502116769598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18943772 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30103286âl5•2è- _wandbruntime‚1255  _runtime‚1255®ï²¢éþ % _timestamp‚1.7858069772070296e+09 _runtime‚1255.498564309 /loss/FRt2/detection_loss‚2.8776580915692147 0loss/FRt2/assignment_loss‚3.4363014403353915 1loss/SRqqt1/detection_loss‚1.0034003920440882 /loss/FBt2/detection_loss‚2.5588437124031507 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚4.222918718580216 .loss/FRt1/detection_loss‚3.686563482833287 1loss/SRqqt2/detection_loss‚3.7177313522056297 .loss/FBt1/detection_loss‚4.532863001103414 &loss/total_loss‚29.679738887015432 _step‚227 2loss/SRqqt1/assignment_loss‚1.5309225386556458 2loss/SRqqt2/assignment_loss‚1.5315534466946568 0loss/FBt1/assignment_loss‚0.4324652502364188 1loss/FBt2/assignment_loss‚0.14851746035431765 epoch‚0éÓ êƒ /loss/FRt2/detection_loss‚2.8776580915692147 0loss/FRt2/assignment_loss‚3.4363014403353915 1loss/SRqqt1/detection_loss‚1.0034003920440882 /loss/FBt2/detection_loss‚2.5588437124031507 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚4.222918718580216 .loss/FRt1/detection_loss‚3.686563482833287 1loss/SRqqt2/detection_loss‚3.7177313522056297 .loss/FBt1/detection_loss‚4.532863001103414 &loss/total_loss‚29.679738887015432 _step‚227 2loss/SRqqt1/assignment_loss‚1.5309225386556458 2loss/SRqqt2/assignment_loss‚1.5315534466946568 0loss/FBt1/assignment_loss‚0.4324652502364188 1loss/FBt2/assignment_loss‚0.14851746035431765 epoch‚0 % _timestamp‚1.7858069772070296e+09 _runtime‚1255.498564309ã Û¦³ë‚(:’ …ÅÓœ‹æÒ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚103.261' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚57.367222222222225 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êj‡ì‚(:ü …ÅÓ¿åÕ! disk.nvme0c0n1.in‚ 18.54296875% disk./.usageGB‚1240.4404754638672 disk.sda3.out‚ 61.71484375 disk.sda2.in‚0 disk.rbd4.out‚ 0.61328125& disk.nvme0c0n1.out‚5099.6767578125 disk.rbd0.in‚0( disk./.usagePercent‚83.2470011072633 disk.sda2.out‚0! disk.dm-0.out‚5099.6767578125 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 41455.953125 network.recv‚199758 disk.drbd1743.out‚ 1.45703125 disk.rbd3.out‚0 disk.rbd5.out‚ 0.61328125 disk.dm-1.out‚ 61.71484375 disk.rbd1.out‚ 0.61328125 disk.drbd1743.in‚0 disk.rbd2.in‚0$ memory_percent‚36.74323558807373 cpu‚13.963327587422913 disk.dm-0.in‚ 18.54296875$ disk.nvme0n1.out‚5099.6767578125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 61.71484375 disk.rbd2.out‚ 0.61328125 disk.sda3.in‚0" proc.memory.rssMB‚ 11267.796875 network.sent‚ 2.671974e+06 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 18.54296875 disk.sda1.in‚0* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29MÍKœí‚(:‘ ”ÅÓÓ¿—Ó gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚125.693' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚69.82944444444445 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ÚÈ~î‚(:ó ”ÅÓ¯‘ÙÕ network.sent‚ 2.716699e+06 disk.rbd0.in‚0 disk.sda.out‚62.28125 disk.sda3.in‚0$ memory_percent‚36.74323558807373 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚62.28125 cpu‚13.962262299568895 disk.rbd4.in‚0! disk.nvme0c0n1.in‚ 18.54296875 disk.rbd2.in‚0( proc.memory.availableMB‚ 41455.953125 proc.cpu.threads‚29 disk.sda1.out‚0 disk.nvme0n1.in‚ 18.54296875 disk.sda1.in‚0 disk.rbd5.out‚ 0.61328125 disk.rbd4.out‚ 0.61328125 disk.rbd1.out‚ 0.61328125 disk.sda2.out‚0" proc.memory.rssMB‚ 11267.796875) disk./.usagePercent‚83.24700648342153 disk.dm-0.in‚ 18.54296875 disk.rbd5.in‚0 disk.rbd0.out‚0 network.recv‚201497% disk./.usageGB‚1240.4405555725098 disk.drbd1743.out‚ 1.4765625$ disk.nvme0n1.out‚5099.9111328125& disk.nvme0c0n1.out‚5099.9111328125 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0! disk.dm-0.out‚5099.9111328125 disk.rbd2.out‚ 0.61328125 disk.sda3.out‚62.28125* proc.memory.percent‚17.193293571472168´M‘2ï- _wandbruntime‚1278  _runtime‚1278Î_‰ÇðÁ _runtime‚1278.450907471 _step‚228 & lr-AdamW/pg1‚7.471356862090237e-05 & lr-AdamW/pg2‚7.471356862090237e-05 trainer/global_step‚2899 % _timestamp‚1.7858070001590157e+09;“Û­ÌñÆ % _timestamp‚1.7858070001590157e+09 _runtime‚1278.450907471 _step‚228 & lr-AdamW/pg1‚7.471356862090237e-05 & lr-AdamW/pg2‚7.471356862090237e-05 trainer/global_step‚2899ä㜾w2ò- _wandbruntime‚1278  _runtime‚1278']Ðx‚ó¶O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19269279 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30620765 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49906487958528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52339510570496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2551372206592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30620765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48551061444832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52527207826944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2620595485696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545213563008 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚229 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49890042 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49788138363904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49899700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11419697 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52339615039488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11419695 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52527058540544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_byt¡xM:es.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19269279 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30630421 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51096275007840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚2899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52339615039488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30630421 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48551044405472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19731602 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52339510570496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2551372206592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30620765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48551061444832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52527207826944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2620595485696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2545213563008 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚229 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49890042 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49788138363904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49899700 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11419697 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52339615039488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11419695 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52527058540544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19269279 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30630421 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51096275007840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚2899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52339615039488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30630421 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48551044405472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19731602 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11423632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52545172801024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2899 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49916880 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49923695075328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30640958 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52357661429248 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52357661429248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51113789810898 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48567784577792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19275922 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2552339262976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2552251826176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2621502629376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48567801617152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30640958 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51113892653494 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2546005233106 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30631288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19738168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11423634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8314785 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49805305126912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858070005695953e+09 _runtime‚1278.861591233 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49916880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19275920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52545330468864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2621635393536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49805322166272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49805305126912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2552339262976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚230 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19738168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11423634 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8314785 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49805305126912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858070005695953e+09 _runtime‚1278.861591233 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49916880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19275920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52545330468864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2621635393536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49805322166272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49805305126912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2552339262976 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚230 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11423632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52545172801024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚2899 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49916880 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49923695075328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30640958 SCMúÙá@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52357661429248 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52357661429248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51113789810898 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48567784577792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19275922 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2552339262976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2552251826176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2621502629376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48567801617152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30640958 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51113892653494 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680æš©˜2ø- _wandbruntime‚1278  _runtime‚1278©˜¢ ùû /loss/FRt2/assignment_loss‚3.628386402130127 1loss/FBt2/assignment_loss‚0.07697694666062606 0loss/SRqqt1/detection_loss‚1.029208290753305 .loss/FBt1/detection_loss‚7.369480419158935 trainer/global_step‚2899 /loss/FRt1/assignment_loss‚4.095816337400013 0loss/FBt1/assignment_loss‚0.7260573484003544 .loss/FRt1/detection_loss‚3.142230878273646 % _timestamp‚1.7858070005711205e+09 _runtime‚1278.862649477 2loss/SRqqt1/assignment_loss‚1.4960609559518423 2loss/SRqqt2/assignment_loss‚1.7136598955030027 /loss/FRt2/detection_loss‚2.8299233900176155 1loss/SRqqt2/detection_loss‚2.8899906459061997 /loss/FBt2/detection_loss‚2.9482151055763133 %loss/total_loss‚31.94600661573198 epoch‚0 _step‚2318@Kjú€ /loss/FRt2/assignment_loss‚3.628386402130127 1loss/FBt2/assignment_loss‚0.07697694666062606 0loss/SRqqt1/detection_loss‚1.029208290753305 .loss/FBt1/detection_loss‚7.369480419158935 trainer/global_step‚2899 /loss/FRt1/assignment_loss‚4.095816337400013 0loss/FBt1/assignment_loss‚0.7260573484003544 .loss/FRt1/detection_loss‚3.142230878273646 % _timestamp‚1.7858070005711205e+09 _runtime‚1278.862649477 2loss/SRqqt1/assignment_loss‚1.4960609559518423 2loss/SRqqt2/assignment_loss‚1.7136598955030027 /loss/FRt2/detection_loss‚2.8299233900176155 1loss/SRqqt2/detection_loss‚2.8899906459061997 /loss/FBt2/detection_loss‚2.9482151055763133 %loss/total_loss‚31.94600661573198 epoch‚0 _step‚231çåv'Oœû‚(:‘ £ÅÓ…ôÓÒ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.16333333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó)KŒü‚(:  £ÅÓ¤à¨Ö% memory_percent‚36.743342876434326 disk.rbd2.out‚ 0.61328125 disk.drbd1743.in‚0" disk.dm-0.out‚5100.54248046875 disk.rbd5.in‚0 disk.rbd4.in‚0$ disk./.usageGB‚1240.440574645996 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚13.963467124018928' disk.nvme0c0n1.out‚5100.54248046875" proc.memory.rssMB‚ 11267.796875 network.sent‚ 2.719246e+06 disk.rbd5.out‚ 0.61328125 disk.sda3.in‚0 disk.dm-1.out‚ 62.98046875) proc.memory.availableMB‚ 41455.8828125 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.nvme0n1.in‚ 18.54296875 disk.sda1.out‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 18.54296875 disk.sda3.out‚ 62.98046875 disk.sda2.in‚0 network.recv‚202794 disk.dm-0.in‚ 18.54296875 disk.rbd3.out‚0 disk.rbd4.out‚ 0.61328125 disk.sda.out‚ 62.98046875 disk.rbd1.out‚ 0.61328125 disk.sda2.out‚0* proc.memory.percent‚17.193293571472168 disk.drbd1743.out‚ 1.50390625 disk.rbd1.in‚0) disk./.usagePercent‚83.24700776345921% disk.nvme0n1.out‚5100.54248046875 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd2.in‚0>{}2ý- _wandbruntime‚1301  _runtime‚1301  uÆþÀ _step‚232 & lr-AdamW/pg1‚7.600217794516768e-05 & lr-AdamW/pg2‚7.600217794516768e-05 trainer/global_step‚2949 % _timestamp‚1.7858070233013256e+09 _runtime‚ 1301.59391037>çò0ËÿÅ & lr-AdamW/pg1‚7.600217794516768e-05 & lr-AdamW/pg2‚7.600217794516768e-05 trainer/global_step‚2949 % _timestamp‚1.7858070233013256e+09 _runtime‚ 1301.59391037 _step‚232è‘z?C2€ - _wandbruntime‚1301  _runtime‚1301œ[ËW¼' ¶O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49388070060832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19601427 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50646476514304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2595368342016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8454954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50766843798528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31148767 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50759852 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚50646493553664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11616547 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2949 _runtime‚1301.594159548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11616545 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50766818894848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50646476514304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20071499 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53432783393280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19601429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2595368342016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51977172087348 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8455206 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53432621523968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2589016223392 _step‚233 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50646493553664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53241844856320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31158423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49388053021472 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53241844856320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2595455771648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2595455771648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2665939594752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2665802629120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7858070233022013e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50750194 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53241949325312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31158423 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19601427 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20071753 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53241949325312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032Û€H¯Á'‚ »O TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53241844856320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2595455771648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2595455771648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2665939594752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2665802629120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7858070233022013e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50750194 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53241949325312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31158423 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19601427 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20071753 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53241949325312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49388070060832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19601427 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50646476514304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2595368342016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8454954 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50766843798528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31148767 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50759852 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚50646493553664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11616547 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2949 _runtime‚1301.594159548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11616545 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50766818894848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50646476514304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20071499 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53432783393280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19601429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2595368342016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51977172087348 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8455206 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚53432621523968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2589016223392 _step‚233 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50646493553664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53241844856320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31158423 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49388053021472éå ¦ 2ƒ - _wandbruntime‚1301  _runtime‚1301†o+n¸„ ¿M _step‚234 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 $ _timestamp‚1.785807023688912e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19608072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31159290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2666697304576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50663643277312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚2949 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2596247961600 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50777032 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2596335398400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20078520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50784026011648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _runtime‚1301.981014079 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53259995715072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2666838457344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8457790 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49404793193792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50663660316672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31168960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53259995715072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50663660316672 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.fr÷¨Ë eed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2596247961600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53259891238912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50767360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53450889372672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31159290 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20078272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8458036 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50784050915328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49404810233152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19608070 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50663643277312 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51994686890406 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2589979499850 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640ˆ‡ÙÊ&… ÄM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2589979499850 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚234 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 $ _timestamp‚1.785807023688912e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19608072 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31159290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2666697304576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50663643277312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚2949 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2596247961600 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50777032 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2596335398400 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20078520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50784026011648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _runtime‚1301.981014079 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53259995715072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2666838457344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8457790 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49404793193792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50663660316672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31168960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53259995715072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50663660316672 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2596247961600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚53259891238912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50767360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53450889372672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31159290 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20078272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8458036 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50784050915328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49404810233152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19608070 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50663643277312 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51994686890406 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392êy” Ö2† - _wandbruntime‚1301  _runtime‚1301¤ò`‡ ý 2loss/SRqqt1/assignment_loss‚1.5447542606774023 2loss/SRqqt2/assignment_loss‚1.8451486414200382 1loss/FBt2/assignment_loss‚0.09890428875610269 trainer/global_step‚2949 % _timestamp‚1.7858070236906147e+09 _runtime‚1301.982163414 /loss/FRt1/assignment_loss‚4.460691236532652 /loss/FRt1/detection_loss‚2.9243235658758726 .loss/FBt2/detection_loss‚3.422043555215822 epoch‚0 _step‚235 /loss/FRt2/assignment_loss‚3.874910731064646 0loss/FBt1/assignment_loss‚0.4977193068276392 1loss/SRqqt1/detection_loss‚1.2060379189920154 .loss/FBt1/detection_loss‚6.680621349811554 /loss/FRt2/detection_loss‚2.7190040306041117 1loss/SRqqt2/detection_loss‚2.6598628692721067 &loss/total_loss‚31.934021755049965~žoˆ ‚ % _timestamp‚1.7858070236906147e+09 _runtime‚1301.982163414 /loss/FRt1/assignment_loss‚4.460691236532652 /loss/FRt1/detection_loss‚2.9243235658758726 .loss/FBt2/detection_loss‚3.422043555215822 epoch‚0 _step‚235 /loss/FRt2/assignment_loss‚3.874910731064646 0loss/FBt1/assignment_loss‚0.4977193068276392 1loss/SRqqt1/detection_loss‚1.2060379189920154 .loss/FBt1/detection_loss‚6.680621349811554 /loss/FRt2/detection_loss‚2.7190040306041117 1loss/SRqqt2/detection_loss‚2.6598628692721067 &loss/total_loss‚31.934021755049965 2loss/SRqqt1/assignment_loss‚1.5447542606774023 2loss/SRqqt2/assignment_loss‚1.8451486414200382 1loss/FBt2/assignment_loss‚0.09890428875610269 trainer/global_step‚2949ëãÔ¬zœ‰ ‚(:‘ ²ÅÓäùþÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚117.825' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.45833333333333 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹;oPŠ ‚(:ö ²ÅÓû°ÃÖ disk.nvme0n1.in‚ 18.54296875 disk.rbd4.in‚0 disk.rbd2.out‚ 0.61328125" proc.memory.rssMB‚ 11267.796875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 18.54296875 disk.rbd5.in‚0 disk.rbd5.out‚ 0.61328125! disk.nvme0c0n1.in‚ 18.54296875 disk.sda.out‚63.6875 disk.rbd2.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚5100.72216796875 disk.rbd0.in‚0 disk.rbd4.out‚ 0.61328125' disk.nvme0c0n1.out‚5100.72216796875 network.sent‚ 2.764034e+06 network.recv‚204703$ memory_percent‚36.74343228340149 disk.sda.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.out‚0 disk.sda3.out‚63.6875 cpu‚13.961983033996392) disk./.usagePercent‚83.24703259619002% disk./.usageGB‚1240.4409446716309 disk.sda1.in‚0 disk.dm-1.out‚63.6875" disk.dm-0.out‚5100.72216796875 disk.rbd0.out‚0 proc.cpu.threads‚29 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.availableMB‚41455.82421875* proc.memory.percent‚17.193293571472168 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 1.50390625 disk.rbd3.in‚0Ý0:œ‹ ‚(:‘ ÁÅÓÌí¥Ó gpu.0.gpu‚26.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚142.147' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚78.97055555555555 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ÿAÛŒ ‚(:ö ÁÅÓåÜùÕ disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 18.54296875 disk.rbd2.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 41455.828125 network.recv‚206901) disk./.usagePercent‚83.24703387622769 disk.nvme0n1.in‚ 18.54296875 disk.rbd4.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd4.out‚ 0.61328125 disk.rbd1.in‚0 disk.rbd3.in‚0$ memory_percent‚36.74342632293701* proc.memory.percent‚17.193293571472168% disk./.usageGB‚1240.4409637451172 disk.sda.in‚0 disk.sda.out‚ 64.2890625 disk.drbd1743.in‚0 disk.drbd1743.out‚ 1.51953125 proc.cpu.threads‚29 network.sent‚ 2.808643e+06# disk.nvme0n1.out‚5101.091796875% disk.nvme0c0n1.out‚5101.091796875 disk.rbd2.out‚ 0.61328125 disk.dm-1.out‚ 64.2890625" proc.memory.rssMB‚ 11267.796875 disk.sda2.out‚0 cpu‚13.96054619972254 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 0.61328125 disk.sda3.in‚0 disk.dm-0.in‚ 18.54296875 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 64.2890625 disk.dm-0.out‚5101.091796875|Éa°2 - _wandbruntime‚1324  _runtime‚1324|¾¤öÃŽ ½ $ lr-AdamW/pg2‚7.7290787269433e-05 trainer/global_step‚2999 % _timestamp‚1.7858070463522527e+09 _runtime‚1324.643951672 _step‚236 $ lr-AdamW/pg1‚7.7290787269433e-05tE×È Â _runtime‚1324.643951672 _step‚236 $ lr-AdamW/pg1‚7.7290787269433e-05 $ lr-AdamW/pg2‚7.7290787269433e-05 trainer/global_step‚2999 % _timestamp‚1.7858070463522527e+09ìƒá’V2 - _wandbruntime‚1324  _runtime‚1324Ñ×i¼'‘ ¶O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31686425 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2639364477440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597709 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51504831704064 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚237 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2632904686900 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31676769 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51504814664704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52857966324372 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51610346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31686425 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2710969535488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20411349 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 % _timestamp‚1.7858070463532882e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54144179142144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8597952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19933579 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54338144270336 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52858069166856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20411104 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54338301945344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54144283611136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54144179142144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11813397 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54144179142144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8597952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19933579 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54338144270336 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52858069166856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20411104 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54338301945344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54144283611136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54144179142144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11813397 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2639364477440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597709 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51504831704064 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚237 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2632904686900 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31676769 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51504814664704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52857966324372 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51610346 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31686425 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2710969535488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2í'k2“ - _wandbruntime‚1325  _runtime‚1325ðÁqÔÄ&” ¾M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54162330000896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19940220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 141152768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50241818849152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31687292 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51521981427712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2640331533824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600814 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51637184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600563 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54162330000896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31687292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51627512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51521981427712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 1325.03125475 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2633782160122 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2633867963358 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 trainer/global_step‚2999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20417895 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31696962 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31696962 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50241801809792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20418148 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19940222 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _step‚238 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51637184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600563 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54162330000896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31687292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51627512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51521981427712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 1325.03125475 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2633782160122 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2633867963358 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 trainer/global_step‚2999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20417895 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31696962 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31696962 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50241801809792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20418148 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19940222 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _step‚238 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600814 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32204771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12010245 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2683360612864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32204771 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55243786065408 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51062070253472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55046617896960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _runtime‚1348.833906002 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52363152815104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20750580 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2676793150408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8740082 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20265729 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55046513427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32214427 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.785807070541636e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2756230586880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32214427 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51062087292832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53738966246364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52487555478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676878953532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52487530574848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55046617896960 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52363152815104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12010247 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2683448042496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20265729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55243622098944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32204771 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12010245 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2683360612864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32204771 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55243786065408 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51062070253472 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55046617896960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _runtime‚1348.833906002 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52363152815104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20750580 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2676793150408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8740082 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20265729 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55046513427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32214427 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.785807070541636e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2756230586880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32214427 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51062087292832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53738966246364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52487555478528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676878953532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52487530574848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55046617896960 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52363152815104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12010247 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2683448042496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20265729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55243622098944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20757072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52504762595328 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52497336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2684240232448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚1349.252436735 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2684327669248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20272372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2684240232448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52504737691648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53756481049422 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52497336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52380319578112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8743150 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677756426866 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51078827465152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚242 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52487664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12014182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 trainer/global_step‚3049 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52487664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12014184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32224964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55064559810560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53756583892018 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32215294 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858070709602187e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52380336617472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20272372 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8742890 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51078810425792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20272370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20272370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55064664286720 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55064664286720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2684327669248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55261733049856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800Q.yFÚ£ ÅM EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52497336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2684240232448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚1349.252436735 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2684327669248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20272372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2684240232448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52504737691648 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚53756481049422 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52497336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52380319578112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8743150 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677756426866 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51078827465152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚242 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52487664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12014182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 trainer/global_step‚3049 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor._z@Õñon_train_batch_end/allocation.all.freed‚52487664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12014184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32224964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55064559810560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53756583892018 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32215294 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858070709602187e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚52380336617472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20272372 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8742890 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51078810425792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20272370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20272370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55064664286720 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55064664286720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2684327669248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55261733049856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55261897009152 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20757072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52504762595328òšíW2¤ - _wandbruntime‚1349  _runtime‚1349~‰ã,¥ ý /loss/FRt2/assignment_loss‚3.877292826301173 2loss/SRqqt1/assignment_loss‚1.5246689056018086 /loss/FRt1/detection_loss‚3.3786056652665137 1loss/SRqqt1/detection_loss‚0.8937359136383449 /loss/FBt2/detection_loss‚3.0746653921224643 /loss/FRt2/detection_loss‚2.6621697526229053 1loss/SRqqt2/detection_loss‚2.4708032327655114 %loss/total_loss‚32.15870996579503 $ _timestamp‚1.785807070962109e+09 _step‚243 /loss/FRt1/assignment_loss‚4.141733115911483 1loss/SRqqt2/assignment_loss‚1.625322143694288 epoch‚0 trainer/global_step‚3049 _runtime‚1349.253667612 1loss/FBt1/assignment_loss‚0.40083805844187737 1loss/FBt2/assignment_loss‚0.19219545576642202 /loss/FBt1/detection_loss‚7.9166795036622455úK¦ ‚ 1loss/SRqqt2/detection_loss‚2.4708032327655114 %loss/total_loss‚32.15870996579503 $ _timestamp‚1.785807070962109e+09 _step‚243 /loss/FRt1/assignment_loss‚4.141733115911483 1loss/SRqqt2/assignment_loss‚1.625322143694288 epoch‚0 trainer/global_step‚3049 _runtime‚1349.253667612 1loss/FBt1/assignment_loss‚0.40083805844187737 1loss/FBt2/assignment_loss‚0.19219545576642202 /loss/FBt1/detection_loss‚7.9166795036622455 /loss/FRt2/assignment_loss‚3.877292826301173 2loss/SRqqt1/assignment_loss‚1.5246689056018086 /loss/FRt1/detection_loss‚3.3786056652665137 1loss/SRqqt1/detection_loss‚0.8937359136383449 /loss/FBt2/detection_loss‚3.0746653921224643 /loss/FRt2/detection_loss‚2.6621697526229053óªÊ…òœ§ ‚(:‘ ßÅÓÔ¼žÓ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.721' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚72.06722222222223 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PEн‡¨ ‚(:ü ßÅÓîµÖ disk.rbd1.out‚ 0.61328125 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚1240.232681274414 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 18.54296875 network.sent‚ 2.855971e+06 network.recv‚210158' disk.nvme0c0n1.out‚5101.79736328125 disk.rbd2.in‚0% memory_percent‚36.743855476379395 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0 cpu‚13.95738427732296 disk.sda3.out‚ 65.7734375 disk.dm-1.in‚0) disk./.usagePercent‚83.23305586485817 disk.sda2.out‚0" disk.dm-0.out‚5101.79736328125 disk.rbd4.in‚0* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29 disk.rbd3.out‚0 disk.nvme0n1.in‚ 18.54296875( proc.memory.availableMB‚ 41455.546875 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 18.54296875 disk.sda.out‚ 65.7734375 disk.drbd1743.in‚0% disk.nvme0n1.out‚5101.79736328125 disk.rbd5.out‚ 0.61328125 disk.rbd4.out‚ 0.61328125 disk.drbd1743.out‚ 1.56640625" proc.memory.rssMB‚ 11267.796875 disk.dm-1.out‚ 65.7734375 disk.rbd2.out‚ 0.61328125) ùgœ© ‚(:‘ îÅÓñŠÓ gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚148.119' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.28833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÌË„‹ª ‚(:€  îÅÓ†˜ÿÖ network.sent‚ 2.900757e+06" disk.dm-0.out‚5102.02001953125 disk.sda1.out‚0* proc.memory.percent‚17.193293571472168 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 18.54296875 disk.sda1.in‚0 disk.rbd5.in‚0$ memory_percent‚36.74384355545044 cpu‚13.955333083566698 disk.sda.in‚0 network.recv‚212241 disk.rbd5.out‚ 0.61328125! disk.nvme0c0n1.in‚ 18.54296875" proc.memory.rssMB‚ 11267.796875 disk.sda2.out‚0 disk.drbd1743.out‚ 1.5703125 disk.rbd0.in‚0 disk.sda.out‚ 66.39453125 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0 proc.cpu.threads‚29% disk./.usageGB‚1240.7346496582031 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0 disk.dm-0.in‚ 18.54296875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5102.02001953125 disk.dm-1.in‚0) proc.memory.availableMB‚ 41455.5546875) disk./.usagePercent‚83.26674338427378 disk.rbd1.in‚0 disk.rbd4.out‚ 0.61328125 disk.rbd2.out‚ 0.61328125 disk.sda3.out‚ 66.39453125 disk.dm-1.out‚ 66.39453125 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚5102.02001953125Ù)@2« - _wandbruntime‚1373  _runtime‚1373"cŒŸÄ¬ ¾ % lr-AdamW/pg1‚7.98680059179636e-05 % lr-AdamW/pg2‚7.98680059179636e-05 trainer/global_step‚3099 $ _timestamp‚1.785807094755842e+09 _runtime‚1373.047654103 _step‚244œ`± É­ à % lr-AdamW/pg1‚7.98680059179636e-05 % lr-AdamW/pg2‚7.98680059179636e-05 trainer/global_step‚3099 $ _timestamp‚1.785807094755842e+09 _runtime‚1373.047654103 _step‚244ôã@‚Ÿ2® - _wandbruntime‚1373  _runtime‚1373‰›Â¼'¯ ¶O IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32732773 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2720767417040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53221508004864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21089616 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2801415437824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51899078869472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20597879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12207095 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55948847713792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56149326756352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53330650 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32732773 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727444177920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51899095908832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727356748288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20597877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55948847713792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32742429 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53340308 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20597879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20597877 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32732773 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2720767417040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚53221508004864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21089616 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2801415437824 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51899078869472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20597879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12207095 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55948847713792 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56149326756352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53330650 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32732773 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2727444177920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51899095908832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2727356748288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20597877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55948847713792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32742429 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53340308 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20597879 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20597877 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7858070951979537e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2802325490688 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56167443926016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.frúíŸNeed‚21096050 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2728236367872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53238674767872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2728236367872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2802192726528 trainer/global_step‚3099 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55966998572544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51915819041792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20604522 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚1373.490219973 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32743296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32752966 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32752966 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54637480971526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2721644890374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54637378128930 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53347816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53357488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721559087138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12211034 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53357488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728323804672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51915836081152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55966894096384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8885018 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53238657728512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12211032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56167286258176 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21096050 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2728236367872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53238674767872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2728236367872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2802192726528 trainer/global_step‚3099 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55966998572544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51915819041792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20604522 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚1373.490219973 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32743296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32752966 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32752966 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54637480971526 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2721644890374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54637378128930 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53347816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53357488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721559087138 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12211034 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53357488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728323804672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51915836081152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55966894096384 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8885018 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53238657728512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12211032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56167286258176 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7858070951979537e+09öç©c2´ - _wandbruntime‚1373  _runtime‚1373 Ãؘµ þ /loss/FRt1/assignment_loss‚3.970360214197183 .loss/FBt1/detection_loss‚5.731626093387604 2loss/SRqqt1/assignment_loss‚1.5689386595850405 2loss/SRqqt2/assignment_loss‚1.3186091293805633 /loss/FRt1/detection_loss‚2.6271160080840317 /loss/FRt2/detection_loss‚2.8462112780128206 epoch‚0 trainer/global_step‚3099 0loss/FRt2/assignment_loss‚3.5985199071112133 0loss/FBt1/assignment_loss‚0.6090305005200207 /loss/FBt2/detection_loss‚3.5057927028364135 &loss/total_loss‚30.056970362176735 % _timestamp‚1.7858070951999393e+09 1loss/FBt2/assignment_loss‚0.06119545004885902 1loss/SRqqt1/detection_loss‚0.9799138642852644 0loss/SRqqt2/detection_loss‚3.239656554727719 _runtime‚1373.492424575 _step‚247&r ¶ ƒ % _timestamp‚1.7858070951999393e+09 1loss/FBt2/assignment_loss‚0.06119545004885902 1loss/SRqqt1/detection_loss‚0.9799138642852644 0loss/SRqqt2/detection_loss‚3.239656554727719 _runtime‚1373.492424575 _step‚247 /loss/FRt1/assignment_loss‚3.970360214197183 .loss/FBt1/detection_loss‚5.731626093387604 2loss/SRqqt1/assignment_loss‚1.5689386595850405 2loss/SRqqt2/assignment_loss‚1.3186091293805633 /loss/FRt1/detection_loss‚2.6271160080840317 /loss/FRt2/detection_loss‚2.8462112780128206 epoch‚0 trainer/global_step‚3099 0loss/FRt2/assignment_loss‚3.5985199071112133 0loss/FBt1/assignment_loss‚0.6090305005200207 /loss/FBt2/detection_loss‚3.5057927028364135 &loss/total_loss‚30.056970362176735÷ =Â0œ· ‚(:‘ ýÅÓ¡çþÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚152.974' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.98555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 zö}¸ ‚(:ò ýÅÓÇŸ¢Ö disk.rbd2.out‚ 0.61328125 disk.sda3.in‚0 disk.drbd1743.out‚ 1.59765625$ memory_percent‚36.74368858337402" proc.memory.rssMB‚ 11267.796875 disk.nvme0n1.in‚ 18.54296875 disk.rbd5.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 18.54296875 disk.dm-1.in‚0 disk.sda2.in‚0 cpu‚13.954194218993807% disk./.usageGB‚1240.8581047058105 disk.sda.out‚ 67.0703125 disk.sda3.out‚ 67.0703125' proc.memory.availableMB‚ 41455.65625" disk.nvme0n1.out‚ 5700.15234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd4.out‚ 0.61328125 disk.sda2.out‚0 disk.rbd3.out‚0 network.sent‚ 2.903292e+06 disk.rbd5.out‚ 0.61328125 disk.rbd2.in‚0 disk.dm-0.out‚ 5700.15234375 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.rbd4.in‚0$ disk.nvme0c0n1.out‚ 5700.15234375 disk.dm-1.out‚ 67.0703125 disk.rbd1.out‚ 0.61328125 disk.dm-0.in‚ 18.54296875( disk./.usagePercent‚83.2750285561044 disk.rbd1.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.193293571472168 network.recv‚213532 disk.rbd0.in‚0 disk.sda.in‚0·âF›¹ ‚(: Œ‚ÅÓº¼æÒ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.48' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.37777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÉË~º ‚(:ó Œ‚ÅÓ©ùÖ disk.nvme0n1.in‚ 18.546875 disk.rbd0.out‚0 disk.rbd1.out‚ 0.61328125 disk.rbd3.out‚0* proc.memory.percent‚17.193293571472168% disk./.usageGB‚1240.8578071594238 disk.sda2.in‚0 disk.dm-0.out‚ 5825.29296875 disk.rbd0.in‚0 disk.dm-0.in‚ 18.546875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 18.546875 disk.rbd2.out‚ 0.61328125 disk.sda3.out‚ 67.85546875 disk.dm-1.in‚0 disk.dm-1.out‚ 67.85546875" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 0.61328125$ disk.nvme0c0n1.out‚ 5825.29296875 disk.sda.in‚0) disk./.usagePercent‚83.27500858751674 disk.drbd1743.out‚ 1.59765625 disk.sda.out‚ 67.85546875 disk.sda3.in‚0* proc.memory.availableMB‚41455.36328125 proc.cpu.threads‚29 network.recv‚215363 disk.sda2.out‚0 disk.rbd2.in‚0$ memory_percent‚36.74413561820984 cpu‚13.952813546935785 network.sent‚ 2.948065e+06 disk.rbd1.in‚0" disk.nvme0n1.out‚ 5825.29296875 disk.rbd5.out‚ 0.61328125¬:Î2» - _wandbruntime‚1396  _runtime‚1396§ö>‚ż ¿ trainer/global_step‚3149 % _timestamp‚1.7858071182221677e+09 _runtime‚1396.513887786 _step‚248 % lr-AdamW/pg1‚8.11566152422289e-05 % lr-AdamW/pg2‚8.11566152422289e-05Wäøàʽ Ä % lr-AdamW/pg2‚8.11566152422289e-05 trainer/global_step‚3149 % _timestamp‚1.7858071182221677e+09 _runtime‚1396.513887786 _step‚248 % lr-AdamW/pg1‚8.11566152422289e-05ø0™‘2¾ - _wandbruntime‚1396  _runtime‚1396ʲÉÍ!'¿ ¶O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33260775 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57054699314176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55500657562896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2846457059328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56851181999616 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21428605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54208242254848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54200460 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56851286468608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20930029 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54208267158528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52736087485472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57054865377792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1396.518129848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54190802 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52736104524832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚>.eð› 2771352883712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0óår*Á'À »O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56851181999616 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21428605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54208242254848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54200460 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56851286468608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20930029 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54208267158528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52736087485472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57054865377792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1396.518129848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54190802 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52736104524832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2771352883712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33260775 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57054699314176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55500657562896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2846457059328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ù$,Á×2Á - _wandbruntime‚1396  _runtime‚1396DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54217640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54207968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55518275208438 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33271298 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21435640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12407882 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57072969647104 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55518378051034 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33280968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2765447550646 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772232503296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2847495371776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2765533353882 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54097012918272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52752827657792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2772232503296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54096995878912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21435382 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9027756 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858071186206453e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54096995878912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360"kqË&à ÅM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54207968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55518275208438 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33271298 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21435640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12407882 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57072969647104 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20936672 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55518378051034 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33280968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2765447550646 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772232503296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2847495371776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2765533353882 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54097012918272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52752827657792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2772232503296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54096995878912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21435382 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9027756 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858071186206453e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54096995878912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12407884 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57072811979264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54225449371648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚3149 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54217640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20936670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57753516285440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21768123 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _step‚253 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55060612 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9167328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12600797 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55068622998528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2808544344056 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2815349019136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21262177 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12600795 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33788777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54938167266304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33798433 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57753516285440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21768123 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _step‚253 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55060612 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9167328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12600797 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55068622998528 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2808544344056 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2815349019136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21262177 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12600795 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33788777 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54938167266304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33798433 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55077792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53589853313152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53589836273792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2809421817390 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2816228638720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33799300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33808970 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57771667144192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54955334029312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55085805211648 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57978562277888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2816316075520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12604732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21268820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57771667144192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9170145 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55068120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2816228638720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21774877 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816316075520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21268822 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33808970 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56399172287946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2892603592704 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56399275130542 % _timestamp‚1.7858071414207656e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33799300 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54955334029312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2809336014154 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 _step‚254 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12604734 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57978408804352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _runtime‚1419.712813544ÉÓ’Ñ ÅM F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2892603592704 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56399275130542 % _timestamp‚1.7858071414207656e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33799300 YFDeviceStatsMonitor.on_train_batch_end/allocated_bÁǨ+9"ytes.large_pool.freed‚54955334029312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2809336014154 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 _step‚254 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12604734 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57978408804352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _runtime‚1419.712813544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9170383 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57771562668032 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55077792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53589853313152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53589836273792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2809421817390 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2816228638720 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33799300 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33808970 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57771667144192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54955334029312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55085805211648 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57978562277888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2816316075520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12604732 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21268820 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57771667144192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9170145 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55068120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2816228638720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21774877 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2816316075520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21268822 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33808970 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56399172287946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2852347004440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9309650 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54410121756832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34316779 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2859345154560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55796522456064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚257 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55928978838528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7858071638722003e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58655955040256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58865829636608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55796522456064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55928953934848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54410104717472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58655850571264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57262451721912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12797647 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9309394 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57262554564396 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12797645 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2859345154560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34326435 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21594327 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22107297 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2937010570240 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55796505416704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58865989408768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58655850571264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55911106 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _runtime‚ 1442.16437034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21594329 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0æ/áDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55796505416704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58865989408768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58655850571264 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55911106 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 _runtime‚ 1442.16437034 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21594329 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2859432584192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34326435 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34316779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2852347004440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9309650 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54410121756832 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34316779 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2859345154560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55796522456064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚257 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚55928978838528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7858071638722003e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58655955040256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58865829636608 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55796522456064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55928953934848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54410104717472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58655850571264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57262451721912 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12797647 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9309394 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57262554564396 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚858031244„ÂO2ß - _wandbruntime‚1442  _runtime‚1442&£þJÅ&à ¿M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55928272 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21600970 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34336972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55813689219072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2860224774144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2937911321088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12801582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _step‚258 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55813672179712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312457 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2853224477662 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3249 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58673896953856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2860224774144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55937944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21600970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54426844889792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21600972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12801584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55946161051648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2853310280898 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55928272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55813689219072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55946185955328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22114041 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 $ _timestamp‚1.785807164260428e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22113795 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57280069367454 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55937944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58884097276416 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58883947997184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54426861929152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55813672179712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚1442.552615274 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34336972 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2860312210944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21600972 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57280172210050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2860312210944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9312213 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130U`ÆÊ&á ÄM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55813672179712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚1442.552615274 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34336972 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2860312210944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21600972 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57280172210050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2860312210944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9312213 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55928272 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21600970 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34336972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55813689219072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2860224774144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2937911321088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12801582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _step‚258 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55813672179712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9312457 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2853224477662 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3249 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58673896953856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2860224774144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55937944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21600970 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54426844889792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21600972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12801584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55946161051648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2853310280898 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55928272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55813689219072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55946185955328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22114041 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 $ _timestamp‚1.785807164260428e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22113795 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57280069367454 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55937944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58884097276416 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58883947997184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54426861929152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0‚p?C2â - _wandbruntime‚1442  _runtime‚1442oA¯ÿã ù 2loss/SRqqt2/assignment_loss‚1.3253138329170338 % _timestamp‚1.7858071642620523e+09 .loss/FRt1/detection_loss‚2.710643914539412 0loss/SRqqt1/detection_loss‚1.117504353575896 /loss/FBt1/detection_loss‚3.7925589084625244 /loss/FRt1/assignment_loss‚4.131932476916945 0loss/FRt2/assignment_loss‚3.6334782683331035 0loss/FBt1/assignment_loss‚0.6702399873174727 0loss/FBt2/assignment_loss‚0.1836921543534926 0loss/SRqqt2/detection_loss‚2.534656254600172 %loss/total_loss‚26.77979185135883 trainer/global_step‚3249 _step‚259 2loss/SRqqt1/assignment_loss‚1.6599578605294227 -loss/FRt2/detection_loss‚2.55337000929791 /loss/FBt2/detection_loss‚2.4664438305154457 epoch‚0 _runtime‚1442.553605108,brä þ /loss/FRt1/assignment_loss‚4.131932476916945 0loss/FRt2/assignment_loss‚3.6334782683331035 0loss/FBt1/assignment_loss‚0.6702399873174727 0loss/FBt2/assignment_loss‚0.1836921543534926 0loss/SRqqt2/detection_loss‚2.534656254600172 %loss/total_loss‚26.77979185135883 trainer/global_step‚3249 _step‚259 2loss/SRqqt1/assignment_loss‚1.6599578605294227 -loss/FRt2/detection_loss‚2.55337000929791 /loss/FBt2/detection_loss‚2.4664438305154457 epoch‚0 _runtime‚1442.553605108 2loss/SRqqt2/assignment_loss‚1.3253138329170338 % _timestamp‚1.7858071642620523e+09 .loss/FRt1/detection_loss‚2.710643914539412 0loss/SRqqt1/detection_loss‚1.117504353575896 /loss/FBt1/detection_loss‚3.7925589084625244ƒ'»œå ‚(:‘ È‚ÅÓªÓûÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚128.941' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.63388888888889 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$%ôIæ ‚(:ö È‚ÅÓÈÑÉ× cpu‚13.947772260692059) disk./.usagePercent‚83.27497069840166% disk./.usageGB‚1240.8572425842285 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 18.55078125 disk.dm-0.out‚ 5826.7578125" proc.memory.rssMB‚ 11267.796875 disk.rbd5.out‚ 0.61328125 disk.rbd4.out‚ 0.61328125 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 disk.nvme0n1.in‚ 18.55078125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 70.69921875 disk.rbd2.in‚0 network.sent‚ 3.042624e+06 disk.rbd3.out‚0 disk.rbd5.in‚0# disk.nvme0c0n1.out‚ 5826.7578125 disk.sda.in‚0 disk.sda3.out‚ 70.69921875 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.drbd1743.out‚ 1.66015625 disk.dm-0.in‚ 18.55078125* proc.memory.availableMB‚41454.90234375 network.recv‚221495 disk.sda2.in‚0$ memory_percent‚36.74483895301819! disk.nvme0n1.out‚ 5826.7578125 disk.rbd2.out‚ 0.61328125 disk.dm-1.out‚ 70.69921875 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168SÇæ2ç - _wandbruntime‚1466  _runtime‚1466f‡–KÆè À trainer/global_step‚3299 $ _timestamp‚1.785807188670891e+09 _runtime‚1466.962584105 _step‚260 & lr-AdamW/pg1‚8.502244321502481e-05 & lr-AdamW/pg2‚8.502244321502481e-05 ‡æËé Å _runtime‚1466.962584105 _step‚260 & lr-AdamW/pg1‚8.502244321502481e-05 & lr-AdamW/pg2‚8.502244321502481e-05 trainer/global_step‚3299 $ _timestamp‚1.785807188670891e+09„Ñ3ª}2ê - _wandbruntime‚1466  _runtime‚1466ÔÒÇKë µO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59558289326080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21926477 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34844781 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9452723 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59771522332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55247113333472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59558184857088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2982056979968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚261 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55247130372832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59771366754816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56654860606464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9452974 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59558184857088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56654860606464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12994497 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21926479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858071886720035e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59558289326080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 1466.96399564 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34854437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58143348801420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34854437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21926479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56789309774848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56654843567104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56789334678528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0øÍ »À'ì ºO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59558184857088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56654860606464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12994497 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21926479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858071886720035e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59558289326080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 1466.96399564 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34854437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58143348801420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34854437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21926479 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56789309774848 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56654843567104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56789334678528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59558289326080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21926477 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34844781 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9452723 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59771522332672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55247113333472 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59558184857088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2982056979968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚261 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55247130372832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59771366754816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56654860606464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9452974 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21933122 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2904308346368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56672027369472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34855304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59789476919296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚3299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55263853505792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12998432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22453944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55263870545152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2904308346368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2904220909568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21933120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2983092791808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58161069289558 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21933120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56672010330112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59576335715840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2897112941170 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56798096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21933122 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34864974 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56806541795328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59576335715840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2904220909568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58160966446962 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56788424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56798096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2897198744406 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858071890608203e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9455760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚1467.352921052 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887_Iî|&ï ÅM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12998432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22453944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55263870545152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2904308346368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2904220909568 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21933120 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2983092791808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58161069289558 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21933120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56672010330112 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚262 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59576335715840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2897112941170 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56798096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21933122 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34864974 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56806541795328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59576335715840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2904220909568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58160966446962 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56788424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56798096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2897198744406 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858071890608203e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9455760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚1467.352921052 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21933122 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2904308346368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚56672027369472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34855304 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59789476919296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚3299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55263853505792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2üFO768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887†ÑS“2ð - _wandbruntime‚1467  _runtime‚1467z 'Rñ ú 0loss/SRqqt2/detection_loss‚2.934162632388584 /loss/FRt1/assignment_loss‚4.191620616118113 1loss/SRqqt2/assignment_loss‚1.646531418976137 /loss/FRt2/detection_loss‚2.4587269477984486 /loss/FBt1/detection_loss‚7.0727285487311224 0loss/FBt2/assignment_loss‚0.0815419881187457 1loss/SRqqt1/detection_loss‚0.9465637611042703 trainer/global_step‚3299 _step‚263 % _timestamp‚1.7858071890624123e+09 _runtime‚ 1467.35392335 .loss/FBt2/detection_loss‚2.460838399431576 &loss/total_loss‚30.686428744175174 epoch‚0 /loss/FRt2/assignment_loss‚3.564734405162288 2loss/SRqqt1/assignment_loss‚1.6401557127861837 0loss/FBt1/assignment_loss‚0.5874130938734327 /loss/FRt1/detection_loss‚3.1014112196862698þE/ ò ÿ % _timestamp‚1.7858071890624123e+09 _runtime‚ 1467.35392335 .loss/FBt2/detection_loss‚2.460838399431576 &loss/total_loss‚30.686428744175174 epoch‚0 /loss/FRt2/assignment_loss‚3.564734405162288 2loss/SRqqt1/assignment_loss‚1.6401557127861837 0loss/FBt1/assignment_loss‚0.5874130938734327 /loss/FRt1/detection_loss‚3.1014112196862698 0loss/SRqqt2/detection_loss‚2.934162632388584 /loss/FRt1/assignment_loss‚4.191620616118113 1loss/SRqqt2/assignment_loss‚1.646531418976137 /loss/FRt2/detection_loss‚2.4587269477984486 /loss/FBt1/detection_loss‚7.0727285487311224 0loss/FBt2/assignment_loss‚0.0815419881187457 1loss/SRqqt1/detection_loss‚0.9465637611042703 trainer/global_step‚3299 _step‚263‡èä9œó ‚(:‘ ׂÅÓ¾ƒäÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.51166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò"Žpô ‚(:å ׂÅÓÉ«ŸÕ network.sent‚ 3.087419e+06 disk.nvme0n1.in‚ 18.55078125" disk.nvme0n1.out‚ 5826.98046875 disk.rbd5.out‚ 0.61328125 disk.rbd0.in‚0 disk.sda3.in‚0 network.recv‚223576 disk.sda.in‚0 disk.sda3.out‚71.375 disk.dm-1.in‚0 disk.drbd1743.in‚0 disk.dm-0.in‚ 18.55078125 disk.dm-0.out‚ 5826.98046875 disk.rbd3.out‚0) disk./.usagePercent‚83.27497197843932 disk.rbd4.in‚0 disk.rbd4.out‚ 0.61328125 disk.sda.out‚71.375 disk.rbd2.out‚ 0.61328125 disk.rbd1.out‚ 0.61328125 disk.rbd3.in‚0$ memory_percent‚36.74188852310181 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.193293571472168 disk.rbd5.in‚0$ disk.nvme0c0n1.out‚ 5826.98046875 disk.sda2.in‚0 disk.drbd1743.out‚1.6875) proc.memory.availableMB‚ 41456.8359375" proc.memory.rssMB‚ 11267.796875 proc.cpu.threads‚29% disk./.usageGB‚1240.8572616577148 disk.rbd2.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 18.55078125 disk.dm-1.out‚71.375 disk.rbd1.in‚0 cpu‚13.946959874141255Bx*}œõ ‚(:‘ æ‚ÅÓÅñ¼Ò gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚123.112' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.39555555555556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êb$r†ö ‚(:û æ‚ÅÓ³˜œÕ network.sent‚ 3.132115e+06 network.recv‚225385) disk./.usagePercent‚83.35011838197995 disk.rbd2.in‚0 disk.nvme0n1.in‚ 18.55078125 disk.rbd5.in‚0 disk.rbd5.out‚ 0.61328125$ memory_percent‚36.73388361930847% disk./.usageGB‚1241.9769973754883 disk.rbd4.in‚0 disk.sda.out‚ 72.0703125 disk.dm-1.out‚ 72.0703125 disk.rbd1.out‚ 0.61328125 disk.sda2.out‚0 disk.drbd1743.out‚1.6875" disk.dm-0.out‚5859.43310546875 disk.sda1.in‚0' disk.nvme0c0n1.out‚5859.43310546875 disk.dm-0.in‚ 18.55078125" proc.memory.rssMB‚ 11267.796875 cpu‚13.945568379472988 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0* proc.memory.availableMB‚41462.08203125% disk.nvme0n1.out‚5859.43310546875! disk.nvme0c0n1.in‚ 18.55078125 disk.sda3.out‚ 72.0703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 0.61328125 disk.rbd2.out‚ 0.61328125 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.sda.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29 W5:2÷ - _wandbruntime‚1489  _runtime‚1489 £áãÇø Á & lr-AdamW/pg1‚8.631105253929012e-05 & lr-AdamW/pg2‚8.631105253929012e-05 trainer/global_step‚3349 % _timestamp‚1.7858072113080852e+09 _runtime‚1489.599884032 _step‚264°³ÉÌù Æ _step‚264 & lr-AdamW/pg1‚8.631105253929012e-05 & lr-AdamW/pg2‚8.631105253929012e-05 trainer/global_step‚3349 % _timestamp‚1.7858072113080852e+09 _runtime‚1489.599884032ˆ"£yD2ú - _wandbruntime‚1489  _runtime‚1489Ì,Ⱥ¼'û ¶O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60460623611904 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13191347 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9595710 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3027371813888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947337425408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59024245880928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35382439 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57649690518528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57649665614848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2940209734580 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56084138988832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13191345 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60460623611904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3027234848256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59024348723412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947424855040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35382439 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57631410 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56084121949472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22258627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57513181717504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947337425408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947424855040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22258629 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57513198756864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9595458 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60460519142912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2940123931456 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57641068 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35372783 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7858072113092556e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22786803ˆ¬ˆïÁ'ü »O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57649690518528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57649665614848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2940209734580 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56084138988832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13191345 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60460623611904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3027234848256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59024348723412 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947424855040 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35382439 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57631410 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56084121949472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22258627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57513181717504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947337425408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947424855040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22258629 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57513198756864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9595458 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60460519142912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2940123931456 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57641068 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35372783 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7858072113092556e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22786803 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60460623611904 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13191347 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9595710 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3027371813888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947337425408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59024245880928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35382439 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136‰Ùb^2ý - _wandbruntime‚1489  _runtime‚1489Îo86þ ÀM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60478565525504 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22265270 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2948304481792 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57530365519872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60695166640640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57648576 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57658248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2948217044992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60478670001664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22793555 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57666872731648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚3349 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22265272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22265272 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59041966369066 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9598528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2948217044992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13195282 % _timestamp‚1.7858072117002788e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60478670001664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35392976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2941087207914 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13195284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60695008972800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56100862121792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57530365519872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57658248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57666897635328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56100879161152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3028269005312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57658248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2948217044992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚60478670001664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22793555 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57666872731648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚3349 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22265272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22265272 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59041966369066 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9598528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2948217044992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13195282 % _timestamp‚1.7858072117002788e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60478670001664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35392976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2941087207914 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13195284 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60695008972800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56100862121792 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57530365519872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57658248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57666897635328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56100879161152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ŠκP2€ - _wandbruntime‚1489  _runtime‚1489VB3 þ /loss/FRt2/assignment_loss‚3.706669187059208 0loss/FBt1/assignment_loss‚0.6628388805048806 /loss/FRt1/detection_loss‚2.7947882200694747 /loss/FRt2/detection_loss‚2.5059661001575235 /loss/FBt1/detection_loss‚3.6737594604492188 /loss/FRt1/assignment_loss‚4.082509373625119 2loss/SRqqt2/assignment_loss‚1.6378888656179627 1loss/SRqqt1/detection_loss‚1.1610562263717394 0loss/SRqqt2/detection_loss‚2.296882860846334 /loss/FBt2/detection_loss‚3.9975714551173667 % _timestamp‚1.7858072117016928e+09 _runtime‚1489.993185937 2loss/SRqqt1/assignment_loss‚1.6776466178042548 1loss/FBt2/assignment_loss‚0.11079572463725473 &loss/total_loss‚28.308372972260337 epoch‚0 trainer/global_step‚3349 _step‚267öJ˜À ‚ ƒ 0loss/FBt1/assignment_loss‚0.6628388805048806 /loss/FRt1/detection_loss‚2.7947882200694747 /loss/FRt2/detection_loss‚2.5059661001575235 /loss/FBt1/detection_loss‚3.6737594604492188 /loss/FRt1/assignment_loss‚4.082509373625119 2loss/SRqqt2/assignment_loss‚1.6378888656179627 1loss/SRqqt1/detection_loss‚1.1610562263717394 0loss/SRqqt2/detection_loss‚2.296882860846334 /loss/FBt2/detection_loss‚3.9975714551173667 % _timestamp‚1.7858072117016928e+09 _runtime‚1489.993185937 2loss/SRqqt1/assignment_loss‚1.6776466178042548 1loss/FBt2/assignment_loss‚0.11079572463725473 &loss/total_loss‚28.308372972260337 epoch‚0 trainer/global_step‚3349 _step‚267 /loss/FRt2/assignment_loss‚3.706669187059208‹•տœƒ ‚(:‘ õ‚ÅÓÈúöÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚111.901' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚62.16722222222222 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]%û¦Š„ ‚(:ÿ õ‚ÅÓÖ network.sent‚ 3.134549e+06 network.recv‚226623 disk.rbd0.in‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚6974.36279296875 disk.sda.out‚ 72.65234375 disk.dm-1.out‚ 72.65234375 cpu‚13.944650820870308 proc.cpu.threads‚29 disk.rbd4.in‚0% disk.nvme0n1.out‚6974.36279296875 disk.rbd2.out‚ 0.61328125* proc.memory.availableMB‚41461.54296875" proc.memory.rssMB‚ 11267.796875 disk.nvme0n1.in‚ 18.55078125 disk.sda2.out‚0" disk.dm-0.out‚6974.36279296875 disk.sda2.in‚0( disk./.usagePercent‚83.3501688154642 disk.rbd5.in‚0 disk.sda.in‚0 disk.rbd1.out‚ 0.61328125 disk.drbd1743.in‚0 disk.rbd3.in‚0% disk./.usageGB‚1241.9777488708496 disk.sda1.out‚0 disk.rbd2.in‚0 disk.sda3.out‚ 72.65234375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0$ memory_percent‚36.73470616340637 disk.rbd4.out‚ 0.61328125 disk.rbd5.out‚ 0.61328125 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 18.55078125 disk.drbd1743.out‚1.703125 disk.dm-0.in‚ 18.55078125* proc.memory.percent‚17.193293571472168 disk.sda1.in‚0$U;2… - _wandbruntime‚1513  _runtime‚1513ìk+Ídž Á trainer/global_step‚3399 % _timestamp‚1.7858072355118003e+09 _runtime‚1513.803845331 _step‚268 & lr-AdamW/pg1‚8.759966186355543e-05 & lr-AdamW/pg2‚8.759966186355543e-05ÕlÎã̇ Æ & lr-AdamW/pg1‚8.759966186355543e-05 & lr-AdamW/pg2‚8.759966186355543e-05 trainer/global_step‚3399 % _timestamp‚1.7858072355118003e+09 _runtime‚1513.803845331 _step‚268Œ¦¡wp2ˆ - _wandbruntime‚1513  _runtime‚1513 J¼'‰ ¶O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58371536907264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2991420990464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58501220 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13388197 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61582406608896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61362853428736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚3399 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58371519867904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56921130565472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22590777 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2991333560832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61362957897728 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3072385154048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59905245802920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58491562 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61362957897728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22590779 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35900785 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23126753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2984012394964 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2991333560832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13388195 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58371536907264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22590777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58371519867904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23126492 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9738556 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚1513.806174146‡‚.2Š »O QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23126753 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2984012394964 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2991333560832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13388195 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58371536907264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22590777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58371519867904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23126492 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9738556 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚1513.806174146 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58371536907264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2991420990464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58501220 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13388197 YFDeviceStatsMo^Θznitor.on_train_batch_start/inactive_split_bytes.all.freed‚61582406608896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61362853428736 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚3399 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58371519867904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56921130565472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22590777 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2991333560832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61362957897728 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3072385154048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59905245802920 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58491562 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61362957897728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22590779 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35900785 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936âåÎE2‹ - _wandbruntime‚1514  _runtime‚1514ÇÊ%Å&Œ ¿M =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35920978 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23133506 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22597420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2992300617216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58388686630912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 % _timestamp‚1.7858072359100246e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2992213180416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2992213180416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58527228571648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3073418393600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 1514.20220826 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9741372 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚3399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22597422 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58508728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58388703670272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23133251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61600671868928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58508728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13392134 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58527253475328 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58518400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35911308 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59922760605978 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61380899811328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22597420 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61600516298240 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58388703670272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58518400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35920978 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58388686630912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35911308éØ$žÊ& ÄM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13392134 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58527253475328 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58518400 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35911308 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59922760605978 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61380899811328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22597420 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚61600516298240 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58388703670272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58518400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35920978 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚58388686630912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35911308 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35920978 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23133506 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22597420 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2992300617216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58388686630912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 % _timestamp‚1.7858072359100246e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2992213180416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2992213180416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58527228571648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3073418393600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚ 1514.20220826 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9741372 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚3399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22597422 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58508728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58388703670272 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23133251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61600671868928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58508728Ž~|eš2Ž - _wandbruntime‚1514  _runtime‚1514ö‘Qÿ ù /loss/SRqqt2/detection_loss‚2.56018164562005 -loss/FBt1/detection_loss‚2.32474821805954 trainer/global_step‚3399 _runtime‚1514.203255724 _step‚271 2loss/SRqqt1/assignment_loss‚1.6583509745182303 0loss/FBt1/assignment_loss‚0.6057058193449242 1loss/SRqqt1/detection_loss‚1.1523026867256294 $ _timestamp‚1.785807235911681e+09 /loss/FBt2/detection_loss‚1.6097745889053874 .loss/FRt1/assignment_loss‚4.13644351485448 0loss/FRt2/assignment_loss‚3.5614343113369413 2loss/SRqqt2/assignment_loss‚1.3581260181963444 1loss/FBt2/assignment_loss‚0.01585376595052036 &loss/total_loss‚24.236479300539173 epoch‚0 .loss/FRt1/detection_loss‚2.709616428766495 /loss/FRt2/detection_loss‚2.5439413282606336‘+Hj þ 0loss/FRt2/assignment_loss‚3.5614343113369413 2loss/SRqqt2/assignment_loss‚1.3581260181963444 1loss/FBt2/assignment_loss‚0.01585376595052036 &loss/total_loss‚24.236479300539173 epoch‚0 .loss/FRt1/detection_loss‚2.709616428766495 /loss/FRt2/detection_loss‚2.5439413282606336 /loss/SRqqt2/detection_loss‚2.56018164562005 -loss/FBt1/detection_loss‚2.32474821805954 trainer/global_step‚3399 _runtime‚1514.203255724 _step‚271 2loss/SRqqt1/assignment_loss‚1.6583509745182303 0loss/FBt1/assignment_loss‚0.6057058193449242 1loss/SRqqt1/detection_loss‚1.1523026867256294 $ _timestamp‚1.785807235911681e+09 /loss/FBt2/detection_loss‚1.6097745889053874 .loss/FRt1/assignment_loss‚4.13644351485448s<&‘ ‚(:„ „ƒÅÓ¸“ŽÔ gpu.0.gpu‚8.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚116.892' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚64.94 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u©È÷Œ’ ‚(:  „ƒÅÓõ…Ö" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.out‚ 1.72265625* proc.memory.percent‚17.193293571472168 network.sent‚ 3.179413e+06 disk.drbd1743.in‚0% disk.nvme0n1.out‚6975.68701171875 disk.rbd4.in‚0 disk.sda.out‚ 73.33984375 network.recv‚228618) disk./.usagePercent‚83.29286767310533 disk.sda1.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 18.55078125 disk.rbd1.out‚ 0.61328125 disk.sda2.in‚0$ memory_percent‚36.73394322395325 disk.rbd3.out‚0 disk.rbd5.in‚0' disk.nvme0c0n1.out‚6975.68701171875 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.availableMB‚41462.04296875 disk.dm-0.in‚ 18.55078125 disk.sda.in‚0 cpu‚13.94276967418579 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd5.out‚ 0.61328125 disk.sda3.in‚0 disk.dm-1.out‚ 73.33984375 proc.cpu.threads‚29 disk.sda3.out‚ 73.33984375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1241.1239204406738" disk.dm-0.out‚6975.68701171875 disk.nvme0n1.in‚ 18.55078125 disk.rbd4.out‚ 0.61328125 disk.rbd2.out‚ 0.61328125èŒÇ ›“ ‚(: “ƒÅÓŠ·ÍÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚38.0 gpu.0.powerWatts‚98.762' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚54.86777777777778 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹yQl{” ‚(:𠓃ÅÓÒÀˆ× network.sent‚ 3.224013e+06) disk./.usagePercent‚83.27173809126681 disk.drbd1743.out‚1.75 disk.rbd3.out‚0 disk.nvme0n1.in‚ 18.55078125% disk.nvme0n1.out‚6975.97607421875$ proc.memory.availableMB‚41461.75" proc.memory.rssMB‚ 11267.796875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 0.61328125! disk.nvme0c0n1.in‚ 18.55078125 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.dm-0.in‚ 18.55078125 disk.rbd3.in‚0 disk.sda2.in‚0$ memory_percent‚36.73439025878906 network.recv‚230747 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6975.97607421875 disk.sda.out‚ 74.078125 disk.rbd2.out‚ 0.69921875 disk.sda3.out‚ 74.078125 disk.sda3.in‚0 disk.dm-1.out‚ 74.078125 disk.rbd1.in‚0 disk.rbd1.out‚ 0.69921875 disk.sda2.out‚0* proc.memory.percent‚17.193293571472168 cpu‚13.941589923324532% disk./.usageGB‚1240.8090744018555 disk.drbd1743.in‚0" disk.dm-0.out‚6975.97607421875 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd5.out‚ 0.61328125 disk.rbd2.in‚0 proc.cpu.threads‚296Ζ2• - _wandbruntime‚1536  _runtime‚1536,ö£œÇ– Á _runtime‚1536.715976026 _step‚272 & lr-AdamW/pg1‚8.888827118782073e-05 & lr-AdamW/pg2‚8.888827118782073e-05 trainer/global_step‚3449 % _timestamp‚1.7858072584242425e+09÷–aÌ— Æ % _timestamp‚1.7858072584242425e+09 _runtime‚1536.715976026 _step‚272 & lr-AdamW/pg1‚8.888827118782073e-05 & lr-AdamW/pg2‚8.888827118782073e-05 trainer/global_step‚3449´:Úe2˜ - _wandbruntime‚1536  _runtime‚1536èL›"ã ™ ¶O WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62487942621696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62488100296704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62265292183552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3035329696256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22922929 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858072584253614e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36428787 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23466592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3027900858472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚3449 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62265292183552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3035417125888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59361372 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62265187714560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57758156220832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23466342 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9881297 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60786040039944 _runtime‚1536.717429741 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59351714 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3035417125888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59370377294848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3117698098176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36438443 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9881545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59370402198528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÌàÕÁ'š »O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59229858018304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22922927 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9881297 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60786040039944 _runtime‚1536.717429741 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59351714 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3035417125888 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59370377294848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3117698098176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36438443 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9881545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59370402198528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62487942621696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62488100296704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62265292183552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3035329696256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22922929 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 % _timestamp‚1.7858072584253614e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36428787 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23466592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3027900858472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚3449 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62265292183552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3035417125888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59361372 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62265187714560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57758156220832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23466342 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036296752640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22929570 _runtime‚1537.099632541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59378552 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62283234097152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858072588075037e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36439310 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60803760528082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59247041820672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59368880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59368880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 _step‚274 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36448980 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36448980 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59247041820672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23473362 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36439310 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59378552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62506049282048 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59247024781312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3036209315840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23473108 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13588984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13588982 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22929572 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3036209315840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3028778331694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59368880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59368880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 _step‚274 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36448980 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36448980 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59247041820672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23473362 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36439310 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59378552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62506049282048 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59247024781312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3036209315840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23473108 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13588984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13588982 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22929572 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3036209315840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3028778331694 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 27129610Ù}Å€24 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036296752640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22929570 _runtime‚1537.099632541’Œßˆâ2ž - _wandbruntime‚1537  _runtime‚1537`K—Ÿ ý 0loss/FRt2/assignment_loss‚3.6216088957256742 /loss/FRt2/detection_loss‚2.5897085934877397 1loss/SRqqt1/detection_loss‚0.9447280629264118 epoch‚0 trainer/global_step‚3449 _step‚275 /loss/FBt1/detection_loss‚3.8698073774576187 .loss/FBt2/detection_loss‚3.600952289495844 %loss/total_loss‚28.10097105309463 _runtime‚1537.100641502 /loss/FRt1/assignment_loss‚4.235541121869147 2loss/SRqqt1/assignment_loss‚1.6267277904617505 2loss/SRqqt2/assignment_loss‚1.6506777705775726 1loss/SRqqt2/detection_loss‚2.3933394984216303 % _timestamp‚1.7858072588091505e+09 0loss/FBt1/assignment_loss‚0.7244507249706658 1loss/FBt2/assignment_loss‚0.14510122149337035 .loss/FRt1/detection_loss‚2.698327706207203AD ‘  ‚ /loss/FBt1/detection_loss‚3.8698073774576187 .loss/FBt2/detection_loss‚3.600952289495844 %loss/total_loss‚28.10097105309463 _runtime‚1537.100641502 /loss/FRt1/assignment_loss‚4.235541121869147 2loss/SRqqt1/assignment_loss‚1.6267277904617505 2loss/SRqqt2/assignment_loss‚1.6506777705775726 1loss/SRqqt2/detection_loss‚2.3933394984216303 % _timestamp‚1.7858072588091505e+09 0loss/FBt1/assignment_loss‚0.7244507249706658 1loss/FBt2/assignment_loss‚0.14510122149337035 .loss/FRt1/detection_loss‚2.698327706207203 0loss/FRt2/assignment_loss‚3.6216088957256742 /loss/FRt2/detection_loss‚2.5897085934877397 1loss/SRqqt1/detection_loss‚0.9447280629264118 epoch‚0 trainer/global_step‚3449 _step‚275“tÛ3aœ¡ ‚(:‘ ¢ƒÅÓ×ÂìÒ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚149.893' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.27388888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P–ëx¢ ‚(:í ¢ƒÅÓŒ‚ÍÖ disk.dm-0.in‚ 18.7734375 disk.nvme0n1.in‚ 18.7734375 disk.rbd4.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd3.in‚0! disk.nvme0n1.out‚ 6977.3359375 disk.sda1.in‚0 disk.sda.out‚ 75.06640625 disk.rbd5.in‚0 disk.rbd4.out‚ 0.61328125 disk.rbd2.out‚ 0.69921875% disk./.usageGB‚1240.5963516235352 disk.drbd1743.out‚1.75 disk.dm-0.out‚ 6977.3359375 disk.rbd0.out‚0 disk.sda3.out‚ 75.06640625 cpu‚13.941066373050772) disk./.usagePercent‚83.25746208712744 disk.rbd3.out‚0 disk.dm-1.out‚ 75.06640625 network.sent‚ 3.226578e+06 network.recv‚231796 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 18.7734375$ memory_percent‚36.73442006111145 disk.rbd1.in‚0 disk.rbd5.out‚ 0.61328125# disk.nvme0c0n1.out‚ 6977.3359375 disk.sda.in‚0* proc.memory.availableMB‚41461.73046875 proc.cpu.threads‚29 disk.rbd1.out‚ 0.76171875 disk.drbd1743.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 11267.796875 disk.sda2.in‚08†2¢2£ - _wandbruntime‚1558  _runtime‚1558¤A$nǤ Á & lr-AdamW/pg1‚9.017688051208604e-05 & lr-AdamW/pg2‚9.017688051208604e-05 trainer/global_step‚3499 % _timestamp‚1.7858072801125836e+09 _runtime‚1558.404227391 _step‚276½{¨ûÌ¥ Æ & lr-AdamW/pg1‚9.017688051208604e-05 & lr-AdamW/pg2‚9.017688051208604e-05 trainer/global_step‚3499 % _timestamp‚1.7858072801125836e+09 _runtime‚1558.404227391 _step‚276”¤™}2¦ - _wandbruntime‚1558  _runtime‚1558hƒåL½'§ ·O UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23255079 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3079413261312 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36956789 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60088213208064 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63393572974592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23255079 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23255077 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63393417396736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61666937119452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 1558.40892152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3079413261312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3071875125104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3162814936064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60211866 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60088213208064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 trainer/global_step‚3499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13781897 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63167626469376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23806494 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63167522000384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10024597 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3162684261888 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61667039961936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63167626469376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36966445 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63167522000384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23806247 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3071789321980 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60088196168704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60230758038528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36956789 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58595147797472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3079325831680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10024352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3079325831680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60221524 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13781895 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _step‚277ÒÂ'¨ ¼O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _step‚277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23255079 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3079413261312 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36956789 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60088213208064 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63393572974592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23255079 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23255077 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63393417396736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61666937119452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 1558.40892152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3079413261312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3071875125104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3162814936064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60211866 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60088213208064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 trainer/global_step‚3499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13781897 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63167626469376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23806494 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63167522000384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10024597 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3162684261888 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61667039961936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63167626469376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36966445 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63167522000384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23806247 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3071789321980 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60088196168704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60230758038528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36956789 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58595147797472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3079325831680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10024352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3079325831680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60221524 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13781895 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130•A³Æ2© - _wandbruntime‚1558  _runtime‚1558‹ú%ª ÂM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23261720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60247965155328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3080292888064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60229032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36967312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36976982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3163583279104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10027364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23261722 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3072666795202 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13785832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58611905009152 % _timestamp‚1.7858072804829512e+09 _runtime‚1558.775024075 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3080292888064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63185672859136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60238704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36976982 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36967312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 3:ÝÊ£OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23261720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61684657607590 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23813198 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63411687490048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60105379971072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23812948 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 trainer/global_step‚3499 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63185568382976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13785832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58611905009152 % _timestamp‚1.7858072804829512e+09 _runtime‚1558.775024075 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3080292888064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63185672859136 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60238704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36976982 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36967312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23261720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61684657607590 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23813198 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63411687490048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60105379971072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23812948 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 trainer/global_step‚3499 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚63185568382976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62547834198960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚3549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123321967104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24146371 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61072018 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23587227 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64069960755200 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37494447 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64069856286208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3123409396736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10167881 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _step‚281 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60946534319104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123321967104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10167626 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3207950965248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64299064843776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61091113878528 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3115763588612 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60946551358464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60946551358464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59432173452832 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59432156413472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3115677785488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13978747 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62547937041444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7858073014464781e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123409396736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23587227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37484791 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23587229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61091088974848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62547834198960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚3549 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123321967104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24146371 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61072018 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23587227 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64069960755200 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37494447 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64069856286208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3123409396736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10167881 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _step‚281 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60946534319104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123321967104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10167626 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3207950965248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64299064843776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61091113878528 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3115763588612 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60946551358464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60946551358464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59432173452832 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59432156413472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3115677785488 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13978747 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62547937041444 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7858073014464781e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123409396736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23587227 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37484791 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23587229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121™¢ÁÔÐ2¹ - _wandbruntime‚1580  _runtime‚15808\»©È&º ÂM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚3549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23593872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60963718121472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37504984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60963718121472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61098856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24153405 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61089184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13982684 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64317010165248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61108296091648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37504984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3124201586688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10170721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982682 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚62565451844502 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23593870 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64087902668800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64317176221696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚1580.104357142 _step‚282 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3116555258710 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 % _timestamp‚1.7858073018123162e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37495314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60963701082112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59448913625152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3116641061946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64088007144960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23593872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24153140 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61098856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3208714073600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60963701082112 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62565554687098 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124289023488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3208855226368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61108320995328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10170458 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124201586688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251ÄYœ^Í&» ÇM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3124201586688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚3549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23593872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60963718121472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37504984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60963718121472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61098856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24153405 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61089184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13982684 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64317010165248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61108296091648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37504984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3124201586688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10170721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982682 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚62565451844502 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23593870 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64087902668800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64317176221696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚1580.104357142 _step‚282 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3116555258710 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 % _timestamp‚1.7858073018123162e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37495314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60963701082112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59448913625152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3116641061946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64088007144960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23593872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24153140 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61098856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3208714073600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60963701082112 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62565554687098 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124289023488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3208855226368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61108320995328 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10170458 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0š>X2¼ - _wandbruntime‚1580  _runtime‚1580ê +•½ ú 2loss/SRqqt2/assignment_loss‚1.6007105881167996 .loss/FBt2/detection_loss‚2.262984393249956 epoch‚0 trainer/global_step‚3549 0loss/FRt1/assignment_loss‚3.9809943401452266 #loss/total_loss‚72.008835817668 % _timestamp‚1.7858073018140824e+09 _step‚283 1loss/SRqqt1/assignment_loss‚1.395249750593613 1loss/FBt2/assignment_loss‚0.02206846330873618 .loss/FRt1/detection_loss‚2.894646752061266 .loss/FRt2/detection_loss‚2.620546459474347 0loss/FBt1/assignment_loss‚0.4896547584103246 1loss/SRqqt1/detection_loss‚1.1055849599368042 1loss/SRqqt2/detection_loss‚2.4237322437186397 /loss/FBt1/detection_loss‚49.637250900268555 _runtime‚1580.105594642 0loss/FRt2/assignment_loss‚3.5754122083837334ìE‚á¾ ÿ .loss/FRt2/detection_loss‚2.620546459474347 0loss/FBt1/assignment_loss‚0.4896547584103246 1loss/SRqqt1/detection_loss‚1.1055849599368042 1loss/SRqqt2/detection_loss‚2.4237322437186397 /loss/FBt1/detection_loss‚49.637250900268555 _runtime‚1580.105594642 0loss/FRt2/assignment_loss‚3.5754122083837334 2loss/SRqqt2/assignment_loss‚1.6007105881167996 .loss/FBt2/detection_loss‚2.262984393249956 epoch‚0 trainer/global_step‚3549 0loss/FRt1/assignment_loss‚3.9809943401452266 #loss/total_loss‚72.008835817668 % _timestamp‚1.7858073018140824e+09 _step‚283 1loss/SRqqt1/assignment_loss‚1.395249750593613 1loss/FBt2/assignment_loss‚0.02206846330873618 .loss/FRt1/detection_loss‚2.894646752061266›}³¬}œ¿ ‚(:‘ σÅÓóÄÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚128.269' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.26055555555556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„¡°_ˆÀ ‚(:ý σÅÓøƒ†×% disk./.usageGB‚1240.4455070495605 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.rbd4.out‚ 0.61328125 proc.cpu.threads‚29 disk.sda2.in‚0 disk.drbd1743.out‚ 1.80859375! disk.dm-0.out‚7041.9345703125 disk.rbd2.in‚0 disk.sda3.out‚ 77.18359375 disk.dm-1.in‚0 network.sent‚ 3.318333e+06 network.recv‚236719 disk.dm-0.in‚ 19.046875 disk.nvme0n1.in‚ 19.046875 disk.rbd5.out‚ 0.76171875 disk.sda.in‚0 disk.sda.out‚ 77.18359375$ memory_percent‚33.31480622291565 disk.rbd3.in‚0 disk.rbd0.in‚0* proc.memory.availableMB‚43702.80859375) disk./.usagePercent‚83.24733878120098 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 0.7939453125 disk.rbd3.out‚0$ disk.nvme0n1.out‚7041.9345703125 disk.sda1.out‚0* proc.memory.percent‚17.193293571472168 disk.rbd4.in‚0& disk.nvme0c0n1.out‚7041.9345703125 disk.rbd2.out‚ 0.7939453125 disk.dm-1.out‚ 77.18359375" proc.memory.rssMB‚ 11267.796875 cpu‚13.941518856964056 disk.sda2.out‚0 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 19.046875*%ûÚ2Á - _wandbruntime‚1600  _runtime‚1600upßçÆ À trainer/global_step‚3599 % _timestamp‚1.7858073223707476e+09 _runtime‚ 1600.66395773 _step‚284 & lr-AdamW/pg1‚9.275409916061665e-05 & lr-AdamW/pg2‚9.275409916061665e-05ý†îŽËà Å trainer/global_step‚3599 % _timestamp‚1.7858073223707476e+09 _runtime‚ 1600.66395773 _step‚284 & lr-AdamW/pg1‚9.275409916061665e-05 & lr-AdamW/pg2‚9.275409916061665e-05œ¶¼P2Ä - _wandbruntime‚1600  _runtime‚16008p$¨ÔÅ ·O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64972190572032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61804889508864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38022449 trainer/global_step‚3599 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23919377 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61804872469504 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63428834120952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38022449 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61804872469504 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14175597 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14175595 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64972295041024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3159652052120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10311299 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65204591089152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61804889508864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61932170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3167318102528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38012793 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64972190572032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24486894 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61951444814848 $ _timestamp‚1.785807322371555e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3159566248996 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64972295041024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3167405532160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65204433414144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3167405532160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252988599296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888Iõí˜Â'Æ ¼O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61804889508864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38022449 trainer/global_step‚3599 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23919377 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61804872469504 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63428834120952 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38022449 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61804872469504 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14175597 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14175595 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64972295041024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3159652052120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10311299 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65204591089152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61804889508864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61932170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3167318102528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38012793 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64972190572032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24486894 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61951444814848 $ _timestamp‚1.785807322371555e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3159566248996 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64972295041024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3167405532160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65204433414144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3167405532160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252988599296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64972190572032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0À©]2Ç - _wandbruntime‚1601  _runtime‚16018P®Ç&È ÁM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3168285158912 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23926022 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63446451766606 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24493963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10314186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61822056271872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61968676835328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61822039232512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23926020 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24493718 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14179532 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63446348924010 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 $ _timestamp‚1.785807322736226e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3168285158912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38032986 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61959008 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65222715811840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3254038976512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14179534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65222556046848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64990341430784 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38023316 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61822039232512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61959008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3160443722218 trainer/global_step‚3599 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61949336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60285905201792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64990341430784 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38023316 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61822039232512 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61959008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3160443722218 trainer/global_step‚3599 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61949336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60285905201792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61822056271872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61968676835328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61822039232512 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23926020 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24493718 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14179532 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63446348924010 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 $ _timestamp‚1.785807322736226e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3168285158912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38032986 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61959008 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65222715811840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3254038976512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14179534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65222556046848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10454452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62663227659264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14372445 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38540795 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65874629326848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62811800654848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64309731200460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24251527 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38550451 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62801980 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61106173645472 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298152651264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64309628357976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10454195 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3211314237952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66110115175424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38550451 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38540795 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65874524857856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61106190684832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62663227659264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14372447 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚1621.592952675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66109953306112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3203454712504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3298152651264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64309628357976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10454195 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3211314237952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66110115175424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38550451 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38540795 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65874524857856 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61106190684832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62663227659264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14372447 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚1621.592952675 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66109953306112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3203454712504 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10454452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62663227659264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14372445 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38540795 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65874629326848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62811800654848 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64309731200460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24251527 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38550451 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62801980¡ä2× - _wandbruntime‚1621  _runtime‚1621¤ ðñÎØ ÁM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62809488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38560988 REDeviceStatsMonitor.on_train_batch_end/iüDäòù%nactive_split.large_pool.freed‚14376382 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24258172 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62680377382912 _step‚290 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24258170 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24833394 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3212281294336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38551318 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62819160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66128058997760 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62819160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62680394422272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62680377382912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299188184576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62809488 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65892571240448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61122930857152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‚“ïÁÌ&Ù ÆM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62809488 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65892571240448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61122930857152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62809488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38560988 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14376382 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24258172 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62680377382912 _step‚290 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24258170 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24833394 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3212281294336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38551318 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62819160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66128058997760 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62819160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62680394422272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62680377382912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299188184576¢‹¨m2Ú - _wandbruntime‚1621  _runtime‚1621÷!ÈgÿÛ ù /loss/FRt1/assignment_loss‚4.457931364837446 2loss/SRqqt2/assignment_loss‚1.9077768681141047 0loss/SRqqt2/detection_loss‚2.287381648673461 -loss/FBt1/detection_loss‚3.37192430655804 _step‚291 0loss/FBt2/assignment_loss‚0.1180769757180921 .loss/FRt1/detection_loss‚2.650720043598037 2loss/SRqqt1/assignment_loss‚1.4939976612891048 /loss/FRt2/detection_loss‚2.7795133721083403 &loss/total_loss‚27.212847649791126 epoch‚0 $ _timestamp‚1.785807343653989e+09 0loss/FRt2/assignment_loss‚3.7978446066379545 1loss/FBt1/assignment_loss‚0.38441668710517907 0loss/SRqqt1/detection_loss‚1.205108583285863 .loss/FBt2/detection_loss‚2.758155531865503 trainer/global_step‚3649 _runtime‚1621.945436824õ(óáÜ þ 0loss/FRt2/assignment_loss‚3.7978446066379545 1loss/FBt1/assignment_loss‚0.38441668710517907 0loss/SRqqt1/detection_loss‚1.205108583285863 .loss/FBt2/detection_loss‚2.758155531865503 trainer/global_step‚3649 _runtime‚1621.945436824 /loss/FRt1/assignment_loss‚4.457931364837446 2loss/SRqqt2/assignment_loss‚1.9077768681141047 0loss/SRqqt2/detection_loss‚2.287381648673461 -loss/FBt1/detection_loss‚3.37192430655804 _step‚291 0loss/FBt2/assignment_loss‚0.1180769757180921 .loss/FRt1/detection_loss‚2.650720043598037 2loss/SRqqt1/assignment_loss‚1.4939976612891048 /loss/FRt2/detection_loss‚2.7795133721083403 &loss/total_loss‚27.212847649791126 epoch‚0 $ _timestamp‚1.785807343653989e+09£Uy!vÝ ‚(:… üƒÅÓ¸«ƒÓ gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚124.182' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚68.99 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f©52ŒÞ ‚(:  üƒÅÓ‹Æ× disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 21.3203125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.85693359375 disk.dm-1.out‚ 79.56640625 disk.rbd1.out‚ 0.919921875 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 21.3203125 disk.nvme0n1.in‚ 21.3203125% disk.nvme0n1.out‚7167.88525390625 network.recv‚241856% disk./.usageGB‚1240.5268478393555 disk.rbd5.out‚ 0.7939453125 disk.sda.in‚0 disk.drbd1743.out‚1.84375% memory_percent‚32.815003395080566 cpu‚13.942596030425962 network.sent‚ 3.410521e+06) disk./.usagePercent‚83.25279762985357 disk.rbd4.out‚ 0.76171875' disk.nvme0c0n1.out‚7167.88525390625 disk.sda2.in‚0" disk.dm-0.out‚7167.88525390625 disk.rbd3.in‚0 disk.rbd5.in‚0 disk.sda.out‚ 79.56640625 disk.sda3.out‚ 79.56640625 disk.drbd1743.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 44030.359375" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚29 disk.sda1.out‚0n§¤62ß - _wandbruntime‚1643  _runtime‚1643«ÌbKÆà À _step‚292 & lr-AdamW/pg1‚9.533131780914726e-05 & lr-AdamW/pg2‚9.533131780914726e-05 trainer/global_step‚3699 $ _timestamp‚1.785807365692426e+09 _runtime‚1643.984172768º”$7Ëá Å _step‚292 & lr-AdamW/pg1‚9.533131780914726e-05 & lr-AdamW/pg2‚9.533131780914726e-05 trainer/global_step‚3699 $ _timestamp‚1.785807365692426e+09 _runtime‚1643.984172768¤*1Y:2â - _wandbruntime‚1643  _runtime‚1643°À=L¡$ã ·O IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66776859143680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63672181398528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63672156494848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63521548770304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247428979136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66776963612672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39068797 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66776963612672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3247343176012 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚293 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61943199300832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63521565809664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14569295 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67015697945600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3343381678592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24583677 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3255310373376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25167304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 $ _timestamp‚1.785807365693545e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3255397803008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3255397803008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61943182261472 trainer/global_step‚3699 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25167558 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66776859143680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65190628279968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24583679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39078453 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3343516547072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39068797 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24583679 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1643.988965873 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3255310373376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10598009 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39078453 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63521548770304 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14569297 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¨žÂ'ä ¼O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39078453 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3343516547072 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39068797 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24583679 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1643.988965873 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3255310373376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10598009 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39078453 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63521548770304 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14569297 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66776859143680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63672181398528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63672156494848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63521548770304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247428979136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66776963612672 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39068797 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚66776963612672 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3247343176012 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚293 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61943199300832 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63521565809664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14569295 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67015697945600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3343381678592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24583677 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3255310373376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25167304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 $ _timestamp‚1.785807365693545e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3255397803008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3255397803008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61943182261472 trainer/global_step‚3699 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25167558 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66776859143680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65190628279968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24583679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144¥áÃ_2å - _wandbruntime‚1644  _runtime‚1644ÒÓ¾óÈ&æ ÂM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24590322 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63679312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63669640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66795010002432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67033650975744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3248220649234 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66794905526272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3256277429760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63538732572672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10601068 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 % _timestamp‚1.7858073661051247e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63538715533312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25174302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14573234 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39088990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63689363611648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1644.397284179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39088990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66795010002432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14573232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3256189992960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3344287364096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61959922433792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24590320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63689388515328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248306452470 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25174072 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24590320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24590322 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63669640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10600840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67033806546432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63679312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3256189992960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61959939473152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3699 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3344418031104 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65208245925622 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65208143083026 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‹™¦Í&ç ÇM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63538715533312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25174302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14573234 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39088990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63689363611648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1644.397284179 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39088990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66795010002432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14573232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3256189992960 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3344287364096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61959922433792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24590320 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63689388515328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248306452470 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25174072 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24590320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚294 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24590322 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63669640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10600840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67033806546432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63679312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3256189992960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚61959939473152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3699 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3344418031104 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65208245925622 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65208143083026 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24590322 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63679312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63669640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚66795010002432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67033650975744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3248220649234 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66794905526272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3256277429760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63538732572672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10601068 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 % _timestamp‚1.7858073661051247e+09¦c7_!2è - _wandbruntime‚1644  _runtime‚1644Ì·uªé € 0loss/FBt1/assignment_loss‚0.5617654315079562 1loss/SRqqt1/detection_loss‚1.0216434023039798 &loss/total_loss‚35.630815039807516 trainer/global_step‚3699 _runtime‚1644.398316624 0loss/FRt1/assignment_loss‚3.7736807326747948 2loss/SRqqt1/assignment_loss‚1.4408189149668618 2loss/SRqqt2/assignment_loss‚1.7436541632720919 _step‚295 /loss/FRt1/detection_loss‚2.6054011816031313 /loss/FBt1/detection_loss‚12.523031763359905 1loss/FBt2/assignment_loss‚0.33298523515622946 .loss/FRt2/detection_loss‚2.685565387786821 1loss/SRqqt2/detection_loss‚2.5140445749416496 /loss/FBt2/detection_loss‚2.8814402200112537 epoch‚0 % _timestamp‚1.7858073661067307e+09 0loss/FRt2/assignment_loss‚3.5467840322228366vö@ ê … 1loss/FBt2/assignment_loss‚0.33298523515622946 .loss/FRt2/detection_loss‚2.685565387786821 1loss/SRqqt2/detection_loss‚2.5140445749416496 /loss/FBt2/detection_loss‚2.8814402200112537 epoch‚0 % _timestamp‚1.7858073661067307e+09 0loss/FRt2/assignment_loss‚3.5467840322228366 0loss/FBt1/assignment_loss‚0.5617654315079562 1loss/SRqqt1/detection_loss‚1.0216434023039798 &loss/total_loss‚35.630815039807516 trainer/global_step‚3699 _runtime‚1644.398316624 0loss/FRt1/assignment_loss‚3.7736807326747948 2loss/SRqqt1/assignment_loss‚1.4408189149668618 2loss/SRqqt2/assignment_loss‚1.7436541632720919 _step‚295 /loss/FRt1/detection_loss‚2.6054011816031313 /loss/FBt1/detection_loss‚12.523031763359905§(À~Ùë ‚(:‘ ‹„ÅÓ¶êôÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚128.801' gpu.0.enûƒi»ÃforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.55611111111111 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q?u”ì ‚(:‰  ‹„ÅÓÚÒóÖ disk.rbd1.out‚ 0.919921875 disk.sda3.in‚0 disk.sda3.out‚ 80.14453125 disk.sda2.in‚0% memory_percent‚32.815009355545044 network.recv‚243982 disk.rbd3.out‚0 disk.rbd4.out‚ 0.7939453125 disk.dm-1.out‚ 80.14453125 disk.drbd1743.in‚0 cpu‚13.94259888490828 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚83.25288313637002 disk.dm-1.in‚0 disk.dm-0.in‚ 21.33203125* proc.memory.percent‚17.193293571472168 disk.sda1.in‚0 disk.rbd5.out‚ 0.85693359375 disk.rbd0.out‚0 disk.rbd4.in‚0! disk.nvme0c0n1.in‚ 21.33203125 disk.sda.in‚0 network.sent‚ 3.455323e+06 disk.rbd0.in‚0 disk.sda.out‚ 80.14453125 disk.drbd1743.out‚ 1.87109375" disk.dm-0.out‚7168.28759765625* proc.memory.availableMB‚44030.35546875 proc.cpu.threads‚29% disk./.usageGB‚1240.5281219482422% disk.nvme0n1.out‚7168.28759765625 disk.rbd2.out‚ 0.919921875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 21.33203125' disk.nvme0c0n1.out‚7168.28759765625" proc.memory.rssMB‚ 11267.796875RIeÄ›í ‚(: š„ÅÓãÉâÒ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚129.63' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚72.01666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0* jŠî ‚(:ÿ š„ÅÓÖÐíÕ network.recv‚245807$ disk./.usageGB‚1240.528060913086 disk.rbd5.out‚ 0.85693359375 disk.rbd0.out‚0 disk.rbd1.out‚ 0.919921875 disk.sda2.in‚0$ memory_percent‚32.70424008369446 network.sent‚ 3.500074e+06 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚21.34375 disk.sda1.in‚0 disk.rbd4.out‚ 0.7939453125 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd2.out‚ 0.919921875" proc.memory.rssMB‚ 11267.796875 proc.cpu.threads‚29 disk.drbd1743.out‚ 1.87109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚21.34375 disk.sda.out‚ 81.05078125 disk.dm-1.in‚0 cpu‚13.943245895124836) disk./.usagePercent‚83.25287904024947 disk.dm-0.in‚21.34375 disk.sda3.out‚ 81.05078125* proc.memory.availableMB‚44102.94921875* proc.memory.percent‚17.193293571472168 disk.rbd2.in‚0% disk.nvme0n1.out‚7168.93896484375 disk.rbd4.in‚0 disk.dm-1.out‚ 81.05078125 disk.sda2.out‚0" disk.dm-0.out‚7168.93896484375 disk.rbd3.out‚0' disk.nvme0c0n1.out‚7168.93896484375 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0Ç‚É2ï - _wandbruntime‚1666  _runtime‚1666X1@üÇð Á trainer/global_step‚3749 % _timestamp‚1.7858073881032598e+09 _runtime‚1666.396021314 _step‚296 & lr-AdamW/pg1‚9.661992713341257e-05 & lr-AdamW/pg2‚9.661992713341257e-05=øN¬Ìñ Æ & lr-AdamW/pg2‚9.661992713341257e-05 trainer/global_step‚3749 % _timestamp‚1.7858073881032598e+09 _runtime‚1666.396021314 _step‚296 & lr-AdamW/pg1‚9.661992713341257e-05¨8ªô/2ò - _wandbruntime‚1666  _runtime‚1666`€w¾'ó ¸O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚3749 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67921260547072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64379886920704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3291317442644 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64379903960064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39596799 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64532537238528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3299393938432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25507569 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25507313 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66071525359476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24915827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62780207916832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3299393938432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67679297898496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14766145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388723308544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67679193429504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62780190877472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3291231639520 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67679193429504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39606455 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66071422516992 _runtime‚1666.396151227 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67679297898496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64379903960064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10741168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64532512334848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10741422 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39606455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24915829 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130ÇÄùbÃ'ô ½O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388723308544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67679193429504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62780190877472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3291231639520 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67679193429504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39606455 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66071422516992 _runtime‚1666.396151227 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67679297898496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64379903960064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10741168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64532512334848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10741422 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39606455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24915829 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚3749 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67921260547072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64379886920704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3291317442644 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64379903960064 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39596799 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64532537238528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3299393938432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25507569 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25507313 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66071525359476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24915827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62780207916832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3299393938432 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67679297898496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14766145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0©Ãkž52õ - _wandbruntime‚1666  _runtime‚1666?馆"ö ÂM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3300186128384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64539464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64397070723072 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24922472 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64539464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14770082 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3292194915978 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24922470 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62796948089152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3292109112742 trainer/global_step‚3749 % _timestamp‚1.7858073885203214e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64397070723072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3300273565184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66089143005130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39607322 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67697344288256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64529792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25513973 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64549719451648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10743891 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64397053683712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚1666.812506133 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64529792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24922472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3300273565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3389491891200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool+»KB.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67697239812096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39616992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67697344288256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62796931049792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0zÝ¢ñÍ&÷ ÇM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3300186128384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24922472 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64539464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14770082 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3292194915978 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24922470 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62796948089152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3292109112742 trainer/global_step‚3749 % _timestamp‚1.7858073885203214e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64397070723072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3300273565184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66089143005130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39607322 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67697344288256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64529792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25513973 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64549719451648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10743891 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64397053683712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚1666.812506133 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64529792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24922472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3300273565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3389491891200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67697239812096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39616992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67697344288256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62796931049792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3300186128384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64539464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64397070723072 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _step‚301 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25247977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10883186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68581527715328 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68826834272768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40134457 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3343390073856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335120103028 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25846431 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63617199493472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65372778 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40124801 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25247979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3343390073856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25846181 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65392893078528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65382436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66952319596500 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40124801 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14962997 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3433802131456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65238242110464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858074109873223e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68581632184320 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 _step‚301 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25247977 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10883186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68581527715328 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68826834272768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40134457 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3343390073856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3335120103028 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25846431 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63617199493472 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65372778 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40124801 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25247979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3343390073856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25846181 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65392893078528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65382436 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66952319596500 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40124801 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14962997 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3433802131456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65238242110464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858074109873223e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68581632184320 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63633939665792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68599574097920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 $ _timestamp‚1.785807411358495e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68599678574080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25852915 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25254622 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336083379486 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25254622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65410100195328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1689.650643006 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65255408873472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3344269700608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66969937242042 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3344182263808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68599678574080 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10885983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚3799 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3344182263808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65255408873472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3344269700608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14966932 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65399616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25853149×Á[FÌ&… ÆM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25254620 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65255391834112 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63633939665792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68599574097920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 $ _timestamp‚1.785807411358495e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68599678574080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25852915 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25254622 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336083379486 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25254622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65410100195328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1689.650643006 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65255408873472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3344269700608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66969937242042 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3344182263808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68599678574080 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10885983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚3799 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3344182263808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65255408873472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3344269700608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14966932 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65399616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25853149 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032®¥ž2† - _wandbruntime‚1689  _runtime‚1689¢)_Q‡ ú /loss/FRt2/assignment_loss‚3.507551840373448 1loss/FBt2/assignment_loss‚0.07830230945222476 0loss/SRqqt1/detection_loss‚0.984177680055044 /loss/FBt2/detection_loss‚2.0084550257506115 %loss/total_loss‚24.44915409429913 epoch‚0 2loss/SRqqt1/assignment_loss‚1.5537527000362223 .loss/FRt2/detection_loss‚2.719051378500926 _runtime‚1689.651637901 0loss/FBt1/assignment_loss‚0.6267081375448139 /loss/FBt1/detection_loss‚2.3591440590946164 % _timestamp‚1.7858074113599331e+09 /loss/FRt1/assignment_loss‚3.793212357163429 1loss/SRqqt2/assignment_loss‚1.895829578532892 .loss/FRt1/detection_loss‚2.280429474450648 1loss/SRqqt2/detection_loss‚2.6425395533442497 trainer/global_step‚3799 _step‚303–N¯ˆ ÿ %loss/total_loss‚24.44915409429913 epoch‚0 2loss/SRqqt1/assignment_loss‚1.5537527000362223 .loss/FRt2/detection_loss‚2.719051378500926 _runtime‚1689.651637901 0loss/FBt1/assignment_loss‚0.6267081375448139 /loss/FBt1/detection_loss‚2.3591440590946164 % _timestamp‚1.7858074113599331e+09 /loss/FRt1/assignment_loss‚3.793212357163429 1loss/SRqqt2/assignment_loss‚1.895829578532892 .loss/FRt1/detection_loss‚2.280429474450648 1loss/SRqqt2/detection_loss‚2.6425395533442497 trainer/global_step‚3799 _step‚303 /loss/FRt2/assignment_loss‚3.507551840373448 1loss/FBt2/assignment_loss‚0.07830230945222476 0loss/SRqqt1/detection_loss‚0.984177680055044 /loss/FBt2/detection_loss‚2.0084550257506115¯ Âœ‰ ‚(:‘ ¸„ÅÓ€æØÒ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚132.881' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚73.82277777777779 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…𲎊 ‚(:ƒ  ¸„ÅÓñ®öÖ$ disk.nvme0n1.out‚7171.2666015625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 82.14453125) disk./.usagePercent‚83.25288185633234 disk.sda1.out‚0 disk.rbd5.out‚ 0.919921875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚7171.2666015625 disk.sda.in‚0 network.recv‚248928 disk.rbd3.in‚0* proc.memory.percent‚17.193293571472168 cpu‚13.943899421408968 proc.cpu.threads‚29 disk.rbd4.out‚ 0.85693359375 disk.rbd2.out‚ 0.919921875 disk.sda3.in‚0" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 21.62890625$ memory_percent‚30.97078800201416 network.sent‚ 3.547392e+06 disk.dm-0.in‚ 21.62890625 disk.rbd5.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 21.62890625 disk.sda.out‚ 82.14453125 disk.dm-1.out‚ 82.14453125 disk.rbd1.out‚ 0.919921875% disk./.usageGB‚1240.5281028747559 disk.drbd1743.out‚ 1.90234375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 45238.984375! disk.dm-0.out‚7171.2666015625ºô;Rœ‹ ‚(:‘ Ç„ÅÓùƒÔÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚148.657' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.58722222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’n‰ŒŒ ‚(:  Ç„ÅÓ÷Í¥Õ disk.dm-0.in‚ 21.62890625 disk.rbd5.in‚0 disk.rbd5.out‚ 0.919921875! disk.nvme0c0n1.in‚ 21.62890625 disk.dm-1.out‚ 82.91796875 disk.rbd1.out‚ 0.919921875 disk.nvme0n1.in‚ 21.62890625$ disk.nvme0n1.out‚7171.5205078125 disk.sda1.out‚0 disk.sda.out‚ 82.91796875 disk.rbd1.in‚0 disk.rbd4.out‚ 0.919921875) proc.memory.availableMB‚ 45250.3359375 disk.drbd1743.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 0.919921875 disk.sda3.out‚ 82.91796875 disk.sda3.in‚0 network.recv‚250901 disk.sda1.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 11267.796875 cpu‚13.943768566270077$ disk./.usageGB‚1240.527271270752 disk.drbd1743.out‚ 1.93359375 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚29 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚30.95346689224243 network.sent‚ 3.592019e+06) disk./.usagePercent‚83.25282604668988! disk.dm-0.out‚7171.5205078125& disk.nvme0c0n1.out‚7171.5205078125 disk.dm-1.in‚0* proc.memory.percent‚17.193293571472168xÕ‰2 - _wandbruntime‚1710  _runtime‚1710+J¡yÆŽ À & lr-AdamW/pg1‚9.919714578194316e-05 & lr-AdamW/pg2‚9.919714578194316e-05 trainer/global_step‚3849 % _timestamp‚1.7858074321763208e+09 _runtime‚ 1710.46885993 _step‚304B`îGË Å _runtime‚ 1710.46885993 _step‚304 & lr-AdamW/pg1‚9.919714578194316e-05 & lr-AdamW/pg2‚9.919714578194316e-05 trainer/global_step‚3849 % _timestamp‚1.7858074321763208e+09°‡ýêo2 - _wandbruntime‚1710  _runtime‚1710ƦÀ‘ ¸O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66096580260864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚64454208109472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40652803 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66242588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69732396232192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69483862001152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25580127 WEDeviceStatsMonitor.on_train_batch_staøz^x®rt/active_bytes.small_pool.freed‚ 3387298779648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3379094369660 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26186277 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚3849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66096563221504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858074321771894e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69483966470144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66253224014848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3479016639488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66096563221504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40662459 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25580129 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25580127 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25580129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3379008566536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3479147313664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40662459 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66096580260864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69732240654336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67833319518492 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64454225148832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3387386209280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15159845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1îtKšÃ'’ ½O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25580129 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25580127 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25580129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3379008566536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3479147313664 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40662459 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66096580260864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69732240654336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67833319518492 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64454225148832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3387386209280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15159845 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66096580260864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚64454208109472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40652803 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66242588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69732396232192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69483862001152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25580127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3387298779648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3379094369660 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26186277 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚3849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66096563221504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858074321771894e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69483966470144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66253224014848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3479016639488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66096563221504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40662459 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121±óŠs%2“ - _wandbruntime‚1710  _runtime‚1710½¿pŒÈ&” ÂM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3480055662592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3388265836032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3479912412672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25586772 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64470965321152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66250096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66259768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40663326 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64470948281792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69750511697920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66259768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26193106 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25586770 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚67850834321550 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3388178399232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26192844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69501908383744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3379886039758 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858074325497367e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66113747023872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69502012859904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3379971842994 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 trainer/global_step‚3849 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66113729984512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25586770 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66270456035328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚1710.841870057 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15163782 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67850937164146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11029062 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25586772 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69502012859904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538v7ƒqr• ÇM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25586770 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚67850834321550 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3388178399232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26192844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚69501908383744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3379886039758 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858074325497367e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66113747023872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69502012859904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3379971842994 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 trainer/global_step‚3849 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66113729984512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25586770 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66270456035328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚1710.841870057 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15163782 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚67850937164146 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11029062 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25586772 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69502012859904 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3480055662592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3388265836032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3479912412672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25586772 XJDeviceStatsMonitor.ðm‚[ on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64470965321152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66250096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66259768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40663326 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64470948281792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69750511697920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66259768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26193106 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130²oØú2– - _wandbruntime‚1710  _runtime‚1710R.Lœ— ú _step‚307 /loss/FRt1/assignment_loss‚4.064729066801743 1loss/SRqqt1/assignment_loss‚1.540236546799089 0loss/FBt2/assignment_loss‚0.0394310020544883 0loss/SRqqt2/detection_loss‚2.380105984342449 &loss/total_loss‚24.414009728701757 trainer/global_step‚3849 0loss/FRt2/assignment_loss‚3.4130298417547476 2loss/SRqqt2/assignment_loss‚1.8583084739306395 0loss/FBt1/assignment_loss‚0.7061192870140076 .loss/FRt1/detection_loss‚2.531650158720957 .loss/FRt2/detection_loss‚2.543307283650274 /loss/FBt1/detection_loss‚1.0520241498947143 /loss/FBt2/detection_loss‚3.2699770913485944 epoch‚0 % _timestamp‚1.7858074325511432e+09 0loss/SRqqt1/detection_loss‚1.015090842390054 _runtime‚1710.842669786@¹˜ ÿ &loss/total_loss‚24.414009728701757 trainer/global_step‚3849 0loss/FRt2/assignment_loss‚3.4130298417547476 2loss/SRqqt2/assignment_loss‚1.8583084739306395 0loss/FBt1/assignment_loss‚0.7061192870140076 .loss/FRt1/detection_loss‚2.531650158720957 .loss/FRt2/detection_loss‚2.543307283650274 /loss/FBt1/detection_loss‚1.0520241498947143 /loss/FBt2/detection_loss‚3.2699770913485944 epoch‚0 % _timestamp‚1.7858074325511432e+09 0loss/SRqqt1/detection_loss‚1.015090842390054 _runtime‚1710.842669786 _step‚307 /loss/FRt1/assignment_loss‚4.064729066801743 1loss/SRqqt1/assignment_loss‚1.540236546799089 0loss/FBt2/assignment_loss‚0.0394310020544883 0loss/SRqqt2/detection_loss‚2.380105984342449³Zº ¿œ™ ‚(:‘ Ö„ÅÓè‹ÛÒ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚132.591' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚73.66166666666668 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÃ<š ‚(:ö Ö„ÅÓ„ô¥Ö network.sent‚ 3.594602e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0# memory_percent‚30.9542715549469) disk./.usagePercent‚83.25289030458097 disk.rbd2.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 21.62890625* proc.memory.percent‚17.193293571472168% disk./.usageGB‚1240.5282287597656 disk.rbd4.in‚0! disk.nvme0c0n1.in‚ 21.62890625 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 21.62890625 disk.rbd0.in‚0 disk.sda.out‚ 83.578125 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd5.in‚0 network.recv‚252039$ disk.nvme0c0n1.out‚ 7171.91015625 proc.cpu.threads‚29 disk.sda.in‚0 disk.dm-0.out‚ 7171.91015625* proc.memory.availableMB‚45249.80859375 disk.drbd1743.out‚ 1.93359375 disk.rbd4.out‚ 0.919921875 disk.rbd2.out‚ 0.919921875 disk.sda3.out‚ 83.578125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.out‚ 7171.91015625 disk.rbd0.out‚0 disk.dm-1.out‚ 83.578125 disk.rbd5.out‚ 0.919921875" proc.memory.rssMB‚ 11267.796875 cpu‚13.944078227793808Â@2› - _wandbruntime‚1731  _runtime‚1731ƒw)Éœ à ' lr-AdamW/pg1‚0.00010048575510620847 ' lr-AdamW/pg2‚0.00010048575510620847 trainer/global_step‚3899 % _timestamp‚1.7858074532406065e+09 _runtime‚1731.532302302 _step‚308u»çñÎ È ' lr-AdamW/pg1‚0.00010048575510620847 ' lr-AdamW/pg2‚0.00010048575510620847 trainer/global_step‚3899 % _timestamp‚1.7858074532406065e+09 _runtime‚1731.532302302 _step‚308´^­Ÿ2ž - _wandbruntime‚1731  _runtime‚1731šáRn¾'Ÿ ¸O ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚3899 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3422897030044 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41190461 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66954901371904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41190461 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68714113755516 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65291233764832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41180805 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65291216725472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70386300755968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26525930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15356695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67113604758528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11169233 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70637910957568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3431382344704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858074532416673e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66954918411264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26525660 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67093082 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25912277 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67102740 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15356697 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67113579854848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70386196286976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25912279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70386196286976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3431294915072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11168965 _runtime‚1731.536974661 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3422982833168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25912279 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66954918411264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70637751185408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66954901371904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3524306199040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121Ú¥/ŽÃ'  ½O G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3422897030044 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41190461 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66954901371904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41190461 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68714113755516 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65291233764832 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41180805 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65291216725472 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70386300755968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26525930 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15356695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67113604758528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11169233 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70637910957568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3431382344704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 % _timestamp‚1.7858074532416673e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66954918411264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26525660 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67093082 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25912277 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67102740 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15356697 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67113579854848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70386196286976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25912279 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70386196286976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3431294915072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11168965 _runtime‚1731.536974661 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3422982833168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25912279 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66954918411264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70637751185408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66954901371904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3524306199040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚3899µ}Ø‘V2¡ - _wandbruntime‚1731  _runtime‚1731+Á‘ÑA¢ ÁM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.785807453595469e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67119920 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11171762 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68731731401058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70404347145728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25918920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41200998 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70404242669568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67110248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67130811875328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525076721152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3423860306502 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67130786971648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65307956897792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 _step‚310 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66972085174272 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70655863692800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3432261971456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15360632 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚3899 XFDeviceStatsMonitor.on_train_batch_end/allocDs]†ated_bytes.small_pool.freed‚ 3432174534656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26532394 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67110248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525207388160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67119920 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3432174534656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26532641 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70404347145728 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68731834243654 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11172007 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65307973937152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70656019263488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚1731.887455332 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66972068134912&dÂÌ&£ ÆM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3432174534656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26532641 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70404347145728 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68731834243654 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11172007 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65307973937152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70656019263488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _runtime‚1731.887455332 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66972068134912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.785807453595469e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67119920 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11171762 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68731731401058 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70404347145728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25918920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41200998 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70404242669568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67110248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67130811875328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525076721152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3423860306502 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67130786971648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65307956897792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 _step‚310 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚66972085174272 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70655863692800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3432261971456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15360632 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚3899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3432174534656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26532394 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67110248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525207388160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67119920¶áA:‰2¤ - _wandbruntime‚1731  _runtime‚1731(SÒ¥ ý 2loss/SRqqt1/assignment_loss‚1.5399474274339293 0loss/FBt1/assignment_loss‚0.3899952423071227 2loss/FBt2/assignment_loss‚0.004474221828153848 trainer/global_step‚3899 2loss/SRqqt2/assignment_loss‚1.5285078248340789 .loss/FRt1/detection_loss‚2.525881867997254 .loss/FBt1/detection_loss‚4.651765942573547 .loss/FBt2/detection_loss‚2.124854183067446 &loss/total_loss‚26.058265615170974 epoch‚0 _runtime‚1731.888122092 _step‚311 /loss/FRt1/assignment_loss‚4.052971268001991 /loss/FRt2/assignment_loss‚3.490932798967129 /loss/FRt2/detection_loss‚2.6633341228816567 1loss/SRqqt1/detection_loss‚0.8692120847382607 1loss/SRqqt2/detection_loss‚2.2163886305404037 % _timestamp‚1.7858074535966554e+09Ï.ܦ ‚ 2loss/FBt2/assignment_loss‚0.004474221828153848 trainer/global_step‚3899 2loss/SRqqt2/assignment_loss‚1.5285078248340789 .loss/FRt1/detection_loss‚2.525881867997254 .loss/FBt1/detection_loss‚4.651765942573547 .loss/FBt2/detection_loss‚2.124854183067446 &loss/total_loss‚26.058265615170974 epoch‚0 _runtime‚1731.888122092 _step‚311 /loss/FRt1/assignment_loss‚4.052971268001991 /loss/FRt2/assignment_loss‚3.490932798967129 /loss/FRt2/detection_loss‚2.6633341228816567 1loss/SRqqt1/detection_loss‚0.8692120847382607 1loss/SRqqt2/detection_loss‚2.2163886305404037 % _timestamp‚1.7858074535966554e+09 2loss/SRqqt1/assignment_loss‚1.5399474274339293 0loss/FBt1/assignment_loss‚0.3899952423071227·§ÿ“vœ§ ‚(:‘ å„ÅÓ»ÇÜÒ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚151.374' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.09666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07Ħwƒ¨ ‚(:ø å„ÅÓÆÅÖ" disk.nvme0c0n1.out‚ 7172.140625 disk.sda3.out‚ 84.22265625 disk.rbd1.in‚0 disk.rbd5.out‚ 0.919921875 disk.rbd4.in‚0 cpu‚13.944263245127958 network.recv‚254076 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 21.62890625! disk.nvme0c0n1.in‚ 21.62890625% memory_percent‚30.902820825576782 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 84.22265625 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚29 disk.rbd2.out‚ 0.919921875 disk.rbd1.out‚ 0.919921875 disk.dm-0.out‚ 7172.140625* proc.memory.percent‚17.193293571472168 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 1.94921875 disk.nvme0n1.in‚ 21.62890625 disk.rbd5.in‚0) disk./.usagePercent‚83.25288646446796 disk.nvme0n1.out‚ 7172.140625 disk.rbd0.in‚0 disk.rbd4.out‚ 0.919921875* proc.memory.availableMB‚45283.52734375" proc.memory.rssMB‚ 11267.796875 network.sent‚ 3.639408e+06% disk./.usageGB‚1240.5281715393066 disk.sda.out‚ 84.22265625 disk.sda3.in‚0 disk.rbd3.out‚0S ±a2© - _wandbruntime‚1752  _runtime‚1752€ôGšÉª à ' lr-AdamW/pg1‚0.00010177436443047379 ' lr-AdamW/pg2‚0.00010177436443047379 trainer/global_step‚3949 % _timestamp‚1.7858074741369371e+09 _runtime‚1752.428609436 _step‚312ß±OMΫ È ' lr-AdamW/pg1‚0.00010177436443047379 ' lr-AdamW/pg2‚0.00010177436443047379 trainer/global_step‚3949 % _timestamp‚1.7858074741369371e+09 _runtime‚1752.428609436 _step‚312¸Ï’¾2¬ - _wandbruntime‚1752  _runtime‚1752‘¥Q–¼'­ ¶O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26244427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 1752.43295301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26865986 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67973935694848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3475378480128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚71288635041792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11312190 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67973960598528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚3949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67813256561664 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66128225341472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67953234 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚71288635041792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26244427 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15553545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67813239522304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3475291050496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3569322497024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66128242380832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71543258191872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41708807 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785807474137926e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69595113677508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67813256561664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3466871296676 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41718463 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26244429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71288530572800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3475291050496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3466785493552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚313 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71288530572800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291¿¡Ã@® »O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67813256561664 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66128225341472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67953234 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚71288635041792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26244427 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15553545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67813239522304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3475291050496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3569322497024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66128242380832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71543258191872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41708807 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 $ _timestamp‚1.785807474137926e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69595113677508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67813256561664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3466871296676 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41718463 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26244429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚71288530572800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3475291050496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3466785493552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚313 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚71288530572800 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26244427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 1752.43295301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26865986 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67973935694848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3475378480128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚71288635041792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11312190 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67973960598528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚3949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136¹»åƒô2¯ - _wandbruntime‚1752  _runtime‚1752l!îÈ&° ÂM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71306576955392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11314989 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚314 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71561524000768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1752.793875862 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26251070 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67830406285312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26872471 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3570231909888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67970400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3476258106880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚71306681431552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26251072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67980072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67970400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26872725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71561374721536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69612731323162 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66144965513792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26251070 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15557484 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136µ‹—¯Í&± ÇM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3467662966774 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67980072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3467748770010 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26872471 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3570231909888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67970400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3476258106880 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚71306681431552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26251072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67980072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67970400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26872725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71561374721536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69612731323162 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66144965513792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26251070 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15557484 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚71306576955392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11314989 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚314 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71561524000768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1752.793875862 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26251070 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67830406285312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68671594712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27206200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3614480903168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3510673957060 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26576577 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27205942 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15750395 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3519374615552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72190969327616 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72190864858624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68813386 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68671577672704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3519374615552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _runtime‚1773.562995325 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3614617868800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3510759760184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68671577672704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26576577 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68823044 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42246465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72448934307328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3999 % _timestamp‚1.7858074952680385e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72190969327616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3519287185920 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72448772438016»î%TÃ'¾ ½O VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72448772438016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42246465 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68834316438528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70475907914532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68834291534848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26576579 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26576579 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68671594712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27206200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3614480903168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3510673957060 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26576577 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27205942 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15750395 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3519374615552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72190969327616 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72190864858624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68813386 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68671577672704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3519374615552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _runtime‚1773.562995325 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3614617868800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3510759760184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68671577672704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26576577 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68823044 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42246465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72448934307328 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3999 % _timestamp‚1.7858074952680385e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72190969327616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3519287185920½éQ®¨2¿ - _wandbruntime‚1773  _runtime‚1773Ë=x”È&À ÂM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27212946 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3511551430282 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42257002 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26583220 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68830552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3615384000000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3511637233518 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68688761475072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66981991169152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3520166805504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3520166805504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3520254242304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70493628402670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15754332 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26583220 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68840224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27212692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72209015717376 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚3999 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1773.917907745 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26583222 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0L´Âš+Á ÇM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚3999 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚1773.917907745 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26583222 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27212946 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3511551430282 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42257002 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26583220 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68830552 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3615384000000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3511637233518 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68688761475072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66981991169152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3520166805504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3520166805504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3520254242304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70493628402670 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15754332 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26583220 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68840224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27212692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72209015717376 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E Ä … 0loss/FRt1/assignment_loss‚3.9676970936531246 &loss/total_loss‚24.676905618737003 _runtime‚1773.918630971 _step‚319 /loss/FRt2/assignment_loss‚3.748894840806395 2loss/SRqqt2/assignment_loss‚1.4101518949976675 0loss/FBt1/assignment_loss‚0.6881735410009112 .loss/FRt1/detection_loss‚2.504291209370591 2loss/SRqqt1/assignment_loss‚1.6719111188773126 2loss/FBt2/assignment_loss‚0.017588552614085984 1loss/SRqqt2/detection_loss‚2.5213994052675037 trainer/global_step‚3999 % _timestamp‚1.7858074956271696e+09 epoch‚0 /loss/FRt2/detection_loss‚2.4778282865063175 1loss/SRqqt1/detection_loss‚1.3263629475911265 /loss/FBt1/detection_loss‚2.2637509905866215 /loss/FBt2/detection_loss‚2.0788557374653465¿ Ù¾šÅ ‚(: ’…ÅÓ„èôÒ gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚99.3' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚55.166666666666664 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DT+‹Æ ‚(:€  ’…ÅÓÒ›ÌÖ network.sent‚ 3.731128e+06$ disk./.usageGB‚1240.538372039795 disk.sda.out‚ 86.4765625 disk.rbd3.in‚0 disk.rbd5.out‚ 0.919921875* proc.memory.availableMB‚45676.97265625* proc.memory.percent‚17.193293571472168 disk.drbd1743.out‚ 1.99609375" disk.dm-0.out‚7184.06005859375 disk.rbd0.out‚0 cpu‚13.943997076220999 disk.sda3.out‚ 86.4765625 disk.sda3.in‚0 disk.rbd4.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 86.4765625 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 11267.796875 proc.cpu.threads‚29 disk.nvme0n1.in‚ 21.8671875 network.recv‚258912 disk.nvme0c0n1.in‚ 21.8671875' disk.nvme0c0n1.out‚7184.06005859375 disk.rbd2.out‚ 0.919921875 disk.sda2.out‚0 disk.dm-0.in‚ 21.8671875 disk.rbd5.in‚0) disk./.usagePercent‚83.25357102861452 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda2.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚7184.06005859375 disk.sda1.in‚0 disk.rbd4.out‚ 0.919921875 disk.rbd2.in‚0% memory_percent‚30.302470922470093x+Xp2Ç - _wandbruntime‚1797  _runtime‚1797Â#.ƒÉÈ Ã % _timestamp‚1.7858075193771944e+09 _runtime‚1797.669121857 _step‚320 ' lr-AdamW/pg1‚0.00010435158307900438 ' lr-AdamW/pg2‚0.00010435158307900438 trainer/global_step‚4049O{yÎÉ È _runtime‚1797.669121857 _step‚320 ' lr-AdamW/pg1‚0.00010435158307900438 ' lr-AdamW/pg2‚0.00010435158307900438 trainer/global_step‚4049 % _timestamp‚1.7858075193771944e+09Àúß2Ê - _wandbruntime‚1797  _runtime‚1797\W"¾'Ë ¸O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26908729 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11597104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26908729 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11597389 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67802242573472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69529932862464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73093199144448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3554648223692 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73093303613440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27544349 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69683196 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42774467 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26908727 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71356907836524 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554562420568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42774467 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚1797.672984649 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69529932862464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69694647374848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73354302458880 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3659789952512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚4049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27544636 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3659655084032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _step‚321 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121ÝÄpcÃ'Ì ½O JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69529915823104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _step‚321 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26908729 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11597104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26908729 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11597389 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67802242573472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69529932862464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73093199144448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3554648223692 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73093303613440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27544349 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69683196 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42774467 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26908727 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71356907836524 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554562420568 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42774467 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚1797.672984649 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69529932862464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69694647374848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73354302458880 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3659789952512 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚4049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27544636 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3659655084032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42775334 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69547082586112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3564250377728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚322 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42785004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3564162940928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26915372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858075197329972e+09 _runtime‚1798.025068766 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73111245527040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73372412664320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27551100 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26915370 [MDeviceStatsMonitor.on_train_batch_end/inac%,“0âtive_split_bytes.small_pool.current‚ 134861312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3660558172672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71374525482178 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69690704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11599918 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69547099625472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3564250377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69700376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69711854491648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73111350003200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69547099625472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42785004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3564162940928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69700376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15951182 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67818999785152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15951184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032ÀY *Í&Ï ÇM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3564162940928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26915372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858075197329972e+09 _runtime‚1798.025068766 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚73111245527040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73372412664320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27551100 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26915370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3660558172672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71374525482178 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69690704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11599918 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69547099625472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3564250377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69700376 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69711854491648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73111350003200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69547099625472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42785004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3564162940928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69700376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15951182 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67818999785152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15951184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73111350003200 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42775334 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69547082586112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3564250377728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚322 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42785004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024ÂN[­2Ð - _wandbruntime‚1798  _runtime‚1798°”ûÿÑ ù &loss/total_loss‚24.919782669064908 1loss/SRqqt1/assignment_loss‚1.581504330131387 2loss/SRqqt2/assignment_loss‚1.4570320811645503 1loss/FBt2/assignment_loss‚0.08805121328638658 .loss/FRt2/detection_loss‚2.624036667770461 1loss/SRqqt1/detection_loss‚0.9823985652894494 0loss/SRqqt2/detection_loss‚2.593477093075451 .loss/FBt2/detection_loss‚2.109896583528573 0loss/FBt1/assignment_loss‚0.6413423617680868 /loss/FRt1/detection_loss‚2.3197128867420056 $ _timestamp‚1.785807519734279e+09 0loss/FRt2/assignment_loss‚3.5140931543550993 trainer/global_step‚4049 _step‚323 /loss/FRt1/assignment_loss‚4.190095953643322 -loss/FBt1/detection_loss‚2.81814177831014 epoch‚0 _runtime‚1798.026187514Î$OLÒ þ 0loss/FRt2/assignment_loss‚3.5140931543550993 trainer/global_step‚4049 _step‚323 /loss/FRt1/assignment_loss‚4.190095953643322 -loss/FBt1/detection_loss‚2.81814177831014 epoch‚0 _runtime‚1798.026187514 &loss/total_loss‚24.919782669064908 1loss/SRqqt1/assignment_loss‚1.581504330131387 2loss/SRqqt2/assignment_loss‚1.4570320811645503 1loss/FBt2/assignment_loss‚0.08805121328638658 .loss/FRt2/detection_loss‚2.624036667770461 1loss/SRqqt1/detection_loss‚0.9823985652894494 0loss/SRqqt2/detection_loss‚2.593477093075451 .loss/FBt2/detection_loss‚2.109896583528573 0loss/FBt1/assignment_loss‚0.6413423617680868 /loss/FRt1/detection_loss‚2.3197128867420056 $ _timestamp‚1.785807519734279e+09Ã!”{K›Ó ‚(: ¡…ÅÓÌõÒ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚151.26' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.03333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/2ÃÑ„Ô ‚(:ù ¡…ÅÓ¾äêÕ disk.rbd3.in‚0 network.sent‚ 3.776159e+06) disk./.usagePercent‚83.25357922085561 disk.rbd2.out‚ 0.919921875 disk.rbd5.out‚ 0.919921875 proc.cpu.threads‚29 disk.sda.out‚ 86.953125 disk.dm-1.out‚ 86.953125 disk.dm-0.in‚ 21.8671875* proc.memory.percent‚17.193293571472168 cpu‚13.94293641718836 disk.sda3.out‚ 86.953125 disk.drbd1743.out‚ 2.02734375 disk.rbd4.in‚0& proc.memory.availableMB‚ 45677.1875 disk.nvme0c0n1.in‚ 21.8671875 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚7184.51611328125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1743.in‚0 disk.rbd0.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd4.out‚ 0.919921875% memory_percent‚30.302143096923828 network.recv‚261214% disk./.usageGB‚1240.5384941101074" disk.dm-0.out‚7184.51611328125 disk.rbd3.out‚0% disk.nvme0n1.out‚7184.51611328125 disk.rbd5.in‚0 disk.nvme0n1.in‚ 21.8671875 disk.sda1.out‚0" proc.memory.rssMB‚ 11267.796875îH­œÕ ‚(:‘ °…ÅÓ²üæÒ gpu.0.gpu‚24.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚147.588' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.99333333333333 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t“ ’{Ö ‚(:ð °…ÅÓº¸¸Õ% disk./.usageGB‚1240.5387001037598 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.dm-0.out‚ 7185.578125 network.recv‚263053 disk.nvme0n1.in‚ 21.8671875 disk.sda1.in‚0 disk.rbd4.out‚ 0.919921875 disk.rbd2.in‚0 disk.sda3.out‚ 87.69140625* proc.memory.percent‚17.193293571472168 disk.rbd3.in‚0 disk.nvme0n1.out‚ 7185.578125 disk.sda1.out‚0 disk.rbd5.out‚ 0.919921875" disk.nvme0c0n1.out‚ 7185.578125 disk.dm-1.out‚ 87.69140625 disk.sda2.out‚0 proc.cpu.threads‚29 disk.dm-0.in‚ 21.8671875 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0 cpu‚13.944826860551128 disk.dm-1.in‚0 network.sent‚ 3.820838e+06 disk.rbd0.in‚0 disk.sda.out‚ 87.69140625 disk.rbd5.in‚0 disk.nvme0c0n1.in‚ 21.8671875 disk.rbd1.in‚0 disk.drbd1743.out‚ 2.02734375$ memory_percent‚30.30252456665039" proc.memory.rssMB‚ 11267.796875) disk./.usagePercent‚83.25359304526246 disk.rbd2.out‚ 0.919921875 disk.rbd1.out‚ 0.919921875& proc.memory.availableMB‚ 45676.9375fMJ,2× - _wandbruntime‚1819  _runtime‚1819Cû½LÈØ Â trainer/global_step‚4099 $ _timestamp‚1.785807541428395e+09 _runtime‚1819.720071895 _step‚324 ' lr-AdamW/pg1‚0.00010564019240326969 ' lr-AdamW/pg2‚0.00010564019240326969Šž½~ÍÙ Ç ' lr-AdamW/pg1‚0.00010564019240326969 ' lr-AdamW/pg2‚0.00010564019240326969 trainer/global_step‚4099 $ _timestamp‚1.785807541428395e+09 _runtime‚1819.720071895 _step‚324Ää¹R2Ú - _wandbruntime‚1819  _runtime‚1819X‰æ/¾'Û ¸O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43292813 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70543348 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27240879 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27240877 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43302469 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72237804916032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3607279456768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1819.723068683 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27240879 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3607279456768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43292813 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚325 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43302469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3598536687200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73995637899264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11739471 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70555028118528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3598450884076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70533690 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74259857887744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16144097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3607366886400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73995637899264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27240877 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27883566 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3704854672896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74260013465600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚73995533430272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7858075414293766e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73995533430272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70388253973504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16144095 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11739721 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68639268228832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72237702073548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68639251189472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70388271012864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70555003214848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚4099 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70388253973504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3607366886400 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73995533430272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70388253973504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16144095 XKDeviceStatsMonitor.on_W<”áý#train_batch_start/inactive_split.small_pool.allocated‚11739721 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68639268228832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72237702073548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68639251189472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70388271012864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70555003214848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚4099 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70388253973504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3607366886400 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27240879 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27240877 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43302469 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72237804916032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3607279456768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1819.723068683 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27240879 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3607279456768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43292813 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚325 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43302469 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3598536687200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73995637899264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11739471 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70555028118528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3598450884076 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70533690 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74259857887744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16144097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3607366886400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73995637899264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27240877 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27883566 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3704854672896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032Åoü?™2Ý - _wandbruntime‚1820  _runtime‚1820— $@Ç&Þ ÁM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70405420736512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43313006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70572210331648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3599414160534 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3608246513152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27247520 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74277965365248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3705891992064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27247520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3705755033600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74278127227392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43313006 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3608246513152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚326 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3608159076352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27890587 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68656008401152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16148032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70550856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16148034 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43303336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72255319719090 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74013579812864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74013684289024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70560528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27890337 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11742553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43303336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27247522 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70405437775872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70550856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚4099 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3608159076352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70405437775872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70405420736512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74013579812864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74013684289024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70560528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27890337 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11742553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43303336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27247522 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚70405437775872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70550856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚4099 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3608159076352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70405437775872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70405420736512 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27247520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3705755033600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74278127227392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43313006 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3608246513152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚326 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3608159076352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27890587 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68656008401152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16148032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70550856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16148034 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43303336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72255319719090 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27573027 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71415383958528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74897867716096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74897867716096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73118599153056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚1842.718962571 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 % _timestamp‚1.7858075644241068e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75165408782336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚329 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3651275592192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71246609163264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69476259805472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11882368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚71246592123904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27573029 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74897972185088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 % _timestamp‚1.7858075644241068e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75165408782336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚329 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3651275592192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71246609163264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69476259805472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11882368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚71246592123904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27573029 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74897972185088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27573027 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71415383958528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74897867716096 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74897867716096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73118599153056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚1842.718962571 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71263775926272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚4149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71411008 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71420680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43831338 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858075648793414e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69493017017152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3652155211776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75183515538944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74916018574848 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27579672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3652242648576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71263758886912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71420680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74915914098688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3751084228608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3643302624042 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27579672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16344884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11884880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11885126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16344882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28229762 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3652242648576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71432566171648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69492999977792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3643216820806¾U¯Íçí ÇM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71432566171648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69492999977792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3643216820806 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71411008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27579670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71263775926272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚4149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71411008 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71420680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43831338 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858075648793414e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69493017017152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3652155211776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75183515538944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74916018574848 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27579672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3652242648576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71263758886912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71420680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74915914098688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3751084228608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3643302624042 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27579672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16344884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11884880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11885126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16344882 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28229762 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3652242648576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291ÊíWÐ72î - _wandbruntime‚1843  _runtime‚1843r±Žùï ú /loss/FRt1/assignment_loss‚4.143926264327249 2loss/FBt2/assignment_loss‚0.028829677257920506 1loss/SRqqt1/detection_loss‚1.0506673126554669 /loss/FRt2/assignment_loss‚3.629360088280269 2loss/SRqqt1/assignment_loss‚1.6029949482151513 /loss/FRt1/detection_loss‚2.4233832627902796 epoch‚0 _step‚331 0loss/FBt1/assignment_loss‚0.4686133413342759 1loss/SRqqt2/detection_loss‚2.1098175175926266 -loss/FBt1/detection_loss‚2.51627779006958 %loss/total_loss‚24.12978905168575 % _timestamp‚1.7858075648807209e+09 _runtime‚1843.172792974 1loss/SRqqt2/assignment_loss‚1.632698214667685 .loss/FRt2/detection_loss‚2.286257821078203 .loss/FBt2/detection_loss‚2.236962813417042 trainer/global_step‚4149ÞÐqÔð ÿ 0loss/FBt1/assignment_loss‚0.4686133413342759 1loss/SRqqt2/detection_loss‚2.1098175175926266 -loss/FBt1/detection_loss‚2.51627779006958 %loss/total_loss‚24.12978905168575 % _timestamp‚1.7858075648807209e+09 _runtime‚1843.172792974 1loss/SRqqt2/assignment_loss‚1.632698214667685 .loss/FRt2/detection_loss‚2.286257821078203 .loss/FBt2/detection_loss‚2.236962813417042 trainer/global_step‚4149 /loss/FRt1/assignment_loss‚4.143926264327249 2loss/FBt2/assignment_loss‚0.028829677257920506 1loss/SRqqt1/detection_loss‚1.0506673126554669 /loss/FRt2/assignment_loss‚3.629360088280269 2loss/SRqqt1/assignment_loss‚1.6029949482151513 /loss/FRt1/detection_loss‚2.4233832627902796 epoch‚0 _step‚331Ë+ÜØœñ ‚(:‘ Î…ÅÓ“à…Ó gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.26' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚62.922222222222224 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N¢ Ždò ‚(:Ù Î…ÅÓŒÀâÕ disk.rbd1.in‚0 disk.sda2.out‚0% memory_percent‚30.119895935058594 network.recv‚266412 disk.rbd4.out‚ 0.919921875% disk.nvme0c0n1.out‚7191.271484375 disk.sda.out‚89 disk.sda3.in‚0 disk.dm-1.out‚89 disk.rbd1.out‚ 0.919921875 disk.rbd3.in‚0) disk./.usagePercent‚83.25383062025423 disk.nvme0n1.in‚ 22.0703125 disk.sda1.out‚0 disk.rbd4.in‚0 disk.sda2.in‚0 disk.dm-0.out‚7191.271484375 disk.rbd3.out‚0 cpu‚13.94400141115855 disk.sda1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.919921875 proc.cpu.threads‚29 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.drbd1743.out‚2.0625# disk.nvme0n1.out‚7191.271484375 disk.rbd5.out‚ 0.919921875 disk.sda3.out‚89 disk.drbd1743.in‚0 disk.dm-0.in‚ 22.0703125" proc.memory.rssMB‚ 11267.796875* proc.memory.percent‚17.193293571472168 network.sent‚ 3.868057e+06 disk.sda.in‚0 disk.dm-1.in‚0% proc.memory.availableMB‚ 45796.625% disk./.usageGB‚1240.5422401428223 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 22.0703125Õ¨°ó ‚(:… Ý…ÅÓÓÕóÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚124.866' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚69.37 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê⸰…ô ‚(:ú Ý…ÅÓ›ä‘× disk.sda.in‚0 disk.sda3.out‚ 89.7734375 disk.sda.out‚ 89.7734375 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 22.0703125 disk.rbd5.in‚0 disk.dm-1.out‚ 89.7734375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-0.out‚7191.599609375 disk.rbd4.in‚0) disk./.usagePercent‚83.25382831618643 disk.drbd1743.out‚ 2.08984375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 22.0703125 disk.rbd2.out‚ 0.919921875 disk.sda2.in‚0$ memory_percent‚30.06654381752014 disk.rbd5.out‚ 0.919921875 disk.rbd0.in‚0 disk.rbd2.in‚0 network.sent‚ 3.912581e+06% disk./.usageGB‚1240.5422058105469 disk.dm-0.in‚ 22.0703125 disk.rbd3.out‚0* proc.memory.percent‚17.193293571472168 network.recv‚268506# disk.nvme0n1.out‚7191.599609375 disk.sda1.out‚0 disk.rbd4.out‚ 0.919921875% disk.nvme0c0n1.out‚7191.599609375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875* proc.memory.availableMB‚45831.58984375" proc.memory.rssMB‚ 11267.796875 cpu‚13.943388919026285 proc.cpu.threads‚29åÊ£ÿ2õ - _wandbruntime‚1865  _runtime‚1865f˜ÎbÆö À & lr-AdamW/pg1‚0.0001082174110518003 & lr-AdamW/pg2‚0.0001082174110518003 trainer/global_step‚4199 $ _timestamp‚1.785807587469858e+09 _runtime‚1865.761570998 _step‚332œ‡ÔË÷ Å & lr-AdamW/pg2‚0.0001082174110518003 trainer/global_step‚4199 $ _timestamp‚1.785807587469858e+09 _runtime‚1865.761570998 _step‚332 & lr-AdamW/pg1‚0.0001082174110518003Ìg>ï2ø - _wandbruntime‚1865  _runtime‚1865Ž[E½'ù ·O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3695271727616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12024835 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7858075874709408e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3695271727616 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75800202001920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76070958839808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16537795 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16537797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚333 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72104930274304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72104947313664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44358473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44348817 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3695359157248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27905179 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27905179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12024581 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72263652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73999599075048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72104947313664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44358473 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75800306470912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72253994 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28562632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3686313614216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75800202001920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27905177 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27905177 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3695359157248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75800306470912 trainer/global_step‚4199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28562376 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73999496232564 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76071110223360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 1865.76602028 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75800306470912 trainer/global_step‚4199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28562376 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73999496232564 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76071110223360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 1865.76602028 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7858075874709408e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3695271727616 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75800202001920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76070958839808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16537795 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16537797 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _step‚333 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72104930274304 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72104947313664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44358473 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44348817 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3695359157248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27905179 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27905179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12024581 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_Š.‰êstart/allocation.all.allocated‚72263652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73999599075048 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72104947313664ÍcUÖý2û - _wandbruntime‚1866  _runtime‚1866K!éMÈ&ü ÂM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74017113878106 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72280832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3696238784000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72122114076672 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44359340 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3696151347200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44359340 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72271160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16541734 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72280832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72271160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12027712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72122114076672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72292922011648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27911820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3687191087550 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28569187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75818352860672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72122097037312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3687105284314 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72292946915328 trainer/global_step‚4199 % _timestamp‚1.7858075878379276e+09 _runtime‚1866.130145163 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44369010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44369010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75818248384512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16541732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3696238784000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27911822 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76089067365888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72122097037312XhÂ÷Í&ý ÇM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72292922011648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27911820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3687191087550 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28569187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75818352860672 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72122097037312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3687105284314 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72292946915328 trainer/global_step‚4199 % _timestamp‚1.7858075878379276e+09 _runtime‚1866.130145163 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44369010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44369010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75818248384512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16541732 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3696238784000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27911822 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76089067365888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72122097037312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74017113878106 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72280832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3696238784000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72122114076672 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44359340 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3696151347200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44359340 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72271160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16541734 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72280832 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72271160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12027712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72122114076672ÎÿÌ}"2þ - _wandbruntime‚1866  _runtime‚1866¡™¬Ùÿ þ epoch‚0 0loss/FRt1/assignment_loss‚3.5793898798698605 0loss/FBt1/assignment_loss‚0.3687876673066057 % _timestamp‚1.7858075878393555e+09 /loss/FRt2/assignment_loss‚3.449615742849267 2loss/FBt2/assignment_loss‚0.061346506845242205 .loss/FBt1/detection_loss‚9.126504600048065 &loss/total_loss‚29.408820939245587 2loss/SRqqt1/assignment_loss‚1.5585165309862499 /loss/FRt1/detection_loss‚2.3406733087328977 .loss/FRt2/detection_loss‚2.372575150883716 0loss/SRqqt1/detection_loss‚0.885802506137549 1loss/SRqqt2/detection_loss‚2.2388191561200728 /loss/FBt2/detection_loss‚1.7769056005856978 _step‚335 2loss/SRqqt2/assignment_loss‚1.6498842888803624 trainer/global_step‚4199 _runtime‚1866.130888787„vÓî € ƒ 0loss/FBt1/assignment_loss‚0.3687876673066057 % _timestamp‚1.7858075878393555e+09 /loss/FRt2/assignment_loss‚3.449615742849267 2loss/FBt2/assignment_loss‚0.061346506845242205 .loss/FBt1/detection_loss‚9.126504600048065 &loss/total_loss‚29.408820939245587 2loss/SRqqt1/assignment_loss‚1.5585165309862499 /loss/FRt1/detection_loss‚2.3406733087328977 .loss/FRt2/detection_loss‚2.372575150883716 0loss/SRqqt1/detection_loss‚0.885802506137549 1loss/SRqqt2/detection_loss‚2.2388191561200728 /loss/FBt2/detection_loss‚1.7769056005856978 _step‚335 2loss/SRqqt2/assignment_loss‚1.6498842888803624 trainer/global_step‚4199 _runtime‚1866.130888787 epoch‚0 0loss/FRt1/assignment_loss‚3.5793898798698605Ïj+mLœ ‚(:‘ ì…ÅÓÊú÷Ò gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚139.189' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚77.32722222222222 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†³º‰‚ ‚(:þ ì…ÅÓÛ²èÕ disk.rbd1.out‚ 0.919921875 disk.drbd1743.out‚ 2.08984375 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda2.out‚0 network.sent‚ 3.9151e+06 disk.rbd3.in‚0$ disk.nvme0n1.out‚7192.0576171875 disk.nvme0c0n1.in‚ 22.0703125 disk.sda.out‚ 90.35546875 proc.cpu.threads‚29 disk.sda3.out‚ 90.35546875! disk.dm-0.out‚7192.0576171875 disk.rbd5.in‚0 disk.rbd4.out‚ 0.919921875 disk.rbd2.in‚0 disk.dm-1.in‚0% memory_percent‚30.066215991973877" proc.memory.rssMB‚ 11267.796875% disk./.usageGB‚1240.5422096252441 disk.drbd1743.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.919921875& disk.nvme0c0n1.out‚7192.0576171875 cpu‚13.943395119239185 disk.dm-0.in‚ 22.0703125 disk.rbd4.in‚0 disk.sda2.in‚0) disk./.usagePercent‚83.25382857219395 disk.dm-1.out‚ 90.35546875* proc.memory.percent‚17.193293571472168 network.recv‚269569 disk.nvme0n1.in‚ 22.0703125 disk.rbd0.in‚0 disk.rbd2.out‚ 0.919921875) proc.memory.availableMB‚ 45831.8046875 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0Æàõ"2ƒ - _wandbruntime‚1888  _runtime‚1888…¿ÇeÈ„  ' lr-AdamW/pg2‚0.00010950602037606561 trainer/global_step‚4249 % _timestamp‚1.7858076104411242e+09 _runtime‚ 1888.73299156 _step‚336 ' lr-AdamW/pg1‚0.00010950602037606561§°[!Í… Ç ' lr-AdamW/pg2‚0.00010950602037606561 trainer/global_step‚4249 % _timestamp‚1.7858076104411242e+09 _runtime‚ 1888.73299156 _step‚336 ' lr-AdamW/pg1‚0.00010950602037606561ÐZy^ý2† - _wandbruntime‚1888  _runtime‚1888fÎ]ßr!‡ ¸O TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44886475 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76702536287744 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28237327 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1888.734803397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72963268424704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16734645 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76976512818688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44886475 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72963268424704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚4249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73123804 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76976668396544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28237329 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76702536287744 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28237327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12167325 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71150294076832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7858076104425893e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73136095638528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74880393312072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3730116274600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28901972 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28901702 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73136095638528 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74880393312072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3730116274600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28901972 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28901702 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76702536287744 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28237327 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1888.734803397 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72963268424704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16734645 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76976512818688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44886475 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72963268424704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚4249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73123804 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76976668396544 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28237329 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76702536287744 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28237327 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12167325 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71150294076832 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7858076104425893e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936ÑoÞ³T2‰ - _wandbruntime‚1889  _runtime‚1889”FÈ&Š ÂM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3731079551058 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚1889.199582184 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72980435187712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858076109073315e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12169830 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44897012 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73140984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3740147482624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28243970 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72980452227072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28243972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12170061 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28908645 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3841350629888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28243972 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28908412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73153302755328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44897012 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73140984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3740147482624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28243970 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72980452227072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28243972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12170061 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28908645 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3841350629888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28243972 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28908412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73153302755328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72980435187712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858076109073315e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12169830 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77604870573568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73821623614464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45404821 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45404821 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28569477 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12309687 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3774090541232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _runtime‚1911.680958045 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28569479 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73821606575104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3885749733888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73996426574848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3783351428096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28569479 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77604870573568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77882201212416 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3885612768256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73983956 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75761290391580 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130'2ÐœÃ'˜ ½O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3783351428096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28569479 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚77604870573568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77882201212416 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3885612768256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73983956 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚75761290391580 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16931495 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16931497 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75761393234064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71987285653472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28569477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7858076333858647e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚77604870573568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73821623614464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45404821 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45404821 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28569477 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12309687 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3774090541232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _runtime‚1911.680958045 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28569479 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73821606575104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3885749733888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73996426574848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024Õ´¬2™ - _wandbruntime‚1912  _runtime‚1912aIÊgš ÂM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45415344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72004042865152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚77623021432320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3784231054848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45425014 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73838773338112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45415344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73838773338112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74001136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28576120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74001136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3886519791104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚342 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73838790377472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚77623021432320 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77622916956160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28576122 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75778908037122 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29247746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72004025825792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16935432 WDDeviceStats‘ÿaMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77900153482752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74013633691648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3774968014566 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3774882211330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45425014 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3784231054848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 _runtime‚1912.037353565 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74001136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3886519791104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚342 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73838790377472 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚77623021432320 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚77622916956160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28576122 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75778908037122 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29247746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72004025825792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16935432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77900153482752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74013633691648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3774968014566 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3774882211330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45425014 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3784231054848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 _runtime‚1912.037353565 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3784231054848 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45425014 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73838773338112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45415344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73838773338112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251Ö(æË2œ - _wandbruntime‚1912  _runtime‚1912Q=Ãá ý 1loss/SRqqt1/assignment_loss‚1.501792399957776 2loss/FBt2/assignment_loss‚0.021164522290070764 /loss/FRt2/detection_loss‚2.4522743794051083 1loss/SRqqt1/detection_loss‚1.2442191301828927 /loss/FBt1/detection_loss‚2.1243178129196165 /loss/FBt2/detection_loss‚1.5146649973466992 % _timestamp‚1.7858076337466562e+09 _runtime‚1912.038136394 0loss/FRt2/assignment_loss‚3.4183958660472524 .loss/FRt1/detection_loss‚2.783311702634977 epoch‚0 trainer/global_step‚4299 0loss/FBt1/assignment_loss‚0.5922651886477979 $loss/total_loss‚23.4047937412241 _step‚343 0loss/FRt1/assignment_loss‚3.9070810200511543 2loss/SRqqt2/assignment_loss‚1.4661495880833988 0loss/SRqqt2/detection_loss‚2.379157133657357:¢Ïž ‚ /loss/FBt1/detection_loss‚2.1243178129196165 /loss/FBt2/detection_loss‚1.5146649973466992 % _timestamp‚1.7858076337466562e+09 _runtime‚1912.038136394 0loss/FRt2/assignment_loss‚3.4183958660472524 .loss/FRt1/detection_loss‚2.783311702634977 epoch‚0 trainer/global_step‚4299 0loss/FBt1/assignment_loss‚0.5922651886477979 $loss/total_loss‚23.4047937412241 _step‚343 0loss/FRt1/assignment_loss‚3.9070810200511543 2loss/SRqqt2/assignment_loss‚1.4661495880833988 0loss/SRqqt2/detection_loss‚2.379157133657357 1loss/SRqqt1/assignment_loss‚1.501792399957776 2loss/FBt2/assignment_loss‚0.021164522290070764 /loss/FRt2/detection_loss‚2.4522743794051083 1loss/SRqqt1/detection_loss‚1.2442191301828927×óªTœŸ ‚(:‘ ™†ÅÓâ²ÙÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚125.397' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚69.66499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô‹¬ÁŠ  ‚(:ÿ ™†ÅÓ¤“ŸÕ disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 94.67578125% disk./.usageGB‚1240.6044845581055 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.193293571472168 network.recv‚275132 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.dm-0.in‚ 28.33984375 disk.rbd0.out‚0) disk./.usagePercent‚83.25800789519043 disk.sda3.in‚0 disk.sda3.out‚ 94.67578125 network.sent‚ 4.007102e+06 disk.dm-0.out‚7258.115234375! disk.nvme0c0n1.in‚ 28.33984375% memory_percent‚30.066466331481934 disk.rbd3.in‚0 disk.nvme0n1.in‚ 28.33984375 disk.rbd5.out‚ 0.919921875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚7258.115234375 disk.rbd2.out‚ 0.919921875" proc.memory.rssMB‚ 11267.796875 disk.drbd1743.out‚ 2.12890625# disk.nvme0n1.out‚7258.115234375 disk.sda1.in‚0 disk.rbd4.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 45831.640625 cpu‚13.941321338869715 proc.cpu.threads‚29 disk.rbd1.out‚ 0.919921875 disk.rbd5.in‚0 disk.rbd4.out‚ 0.919921875 disk.dm-1.out‚ 94.67578125H–[¡ ‚(:„ ¨†ÅÓ”ÌðÒ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.57' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚63.65 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò,­ˆ¢ ‚(:ý ¨†ÅÓ°¼†× disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 0.919921875 disk.dm-1.out‚ 96.0234375 disk.sda.in‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚7323.2236328125 disk.rbd5.out‚ 0.919921875 disk.sda3.in‚0* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 2.1484375 disk.dm-0.in‚ 28.33984375 disk.rbd5.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 45831.640625 disk.sda1.in‚0 disk.rbd2.out‚ 0.919921875" proc.memory.rssMB‚ 11267.796875 disk.sda.out‚ 96.0234375 network.recv‚277221! disk.nvme0c0n1.in‚ 28.33984375 disk.sda3.out‚ 96.0234375% memory_percent‚30.066466331481934 cpu‚13.939454220763274 network.sent‚ 4.05193e+06) disk./.usagePercent‚83.26221358696293% disk./.usageGB‚1240.6671524047852& disk.nvme0c0n1.out‚7323.2236328125 disk.rbd1.out‚ 0.919921875 disk.nvme0n1.in‚ 28.33984375 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚7323.2236328125 disk.rbd3.out‚0Äxé«2£ - _wandbruntime‚1935  _runtime‚1935£}Pɤ à ' lr-AdamW/pg2‚0.00011208323902459621 trainer/global_step‚4349 % _timestamp‚1.7858076567585042e+09 _runtime‚1935.050175931 _step‚344 ' lr-AdamW/pg1‚0.00011208323902459621c!ÕKÎ¥ È ' lr-AdamW/pg1‚0.00011208323902459621 ' lr-AdamW/pg2‚0.00011208323902459621 trainer/global_step‚4349 % _timestamp‚1.7858076567585042e+09 _runtime‚1935.050175931 _step‚344ØXáBt2¦ - _wandbruntime‚1935  _runtime‚1935 3àN¾'§ ¸O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74856807318528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78507204859392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45942479 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74679961764864 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3817979004740 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76642187471088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28901627 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3817893201616 trainer/global_step‚4349 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3827347563520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29582464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3930898245632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858076567595332e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74834450 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74679944725504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28901629 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _step‚345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74856782414848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3827260133888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3930761280000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78507204859392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28901629 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12453884 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78787705564160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28901627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17128347 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17128345 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74679961764864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3827347563520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78507309328384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72824311308832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72824294269472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45932823 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76642290313572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45932823 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3827260133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1935.054983078 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45942479 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74679944725504 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3827347563520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29582464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3930898245632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858076567595332e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74834450 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74679944725504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28901629 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _step‚345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74856782414848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3827260133888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3930761280000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78507204859392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28901629 TGDeviceStatsMonitor.on_train_batc¼£œh_start/inactive_split.small_pool.freed‚12453884 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78787705564160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28901627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17128347 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17128345 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74679961764864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3827347563520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78507309328384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72824311308832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72824294269472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45932823 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76642290313572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45932823 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3827260133888 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1935.054983078 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45942479 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74679944725504 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130Ù½MÏÏ2© - _wandbruntime‚1935  _runtime‚1935Ϲ±È&ª ÂM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74861288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28908272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28908270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72841034441792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3931816244736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78805666720768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76659907959226 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12457040 trainer/global_step‚4349 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28908270 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74851616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3828227190272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74873989531648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45943346 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78525355718144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _runtime‚1935.514460383 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78805830680064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3818770674838 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74861288 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74851616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28908272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74697111488512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3828227190272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74874014435328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3828139753472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3828139753472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45953016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74697128527872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29589324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29589059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17132284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76659805116630 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78525355718144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45953016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78525251241984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12456777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136ú4:ÐÍ&« ÇM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3931816244736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78805666720768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76659907959226 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12457040 trainer/global_step‚4349 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28908270 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74851616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3828227190272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74873989531648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45943346 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78525355718144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _runtime‚1935.514460383 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78805830680064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3818770674838 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74861288 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74851616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28908272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74697111488512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3828227190272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74874014435328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3828139753472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3828139753472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45953016 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74697128527872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29589324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29589059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17132284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76659805116630 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78525355718144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45953016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78525251241984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12456777 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74861288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28908272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28908270 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72841034441792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ú!Ôd2¬ - _wandbruntime‚1935  _runtime‚1935OTúØ­ û 1loss/SRqqt1/detection_loss‚0.7262341185346447 0loss/SRqqt2/detection_loss‚2.110504688591593 _step‚347 1loss/SRqqt2/assignment_loss‚1.600684832678073 .loss/FRt1/detection_loss‚2.265652567650495 /loss/FBt2/detection_loss‚1.8358680444673217 %loss/total_loss‚30.58706922668645 _runtime‚1935.515052293 /loss/FRt2/assignment_loss‚3.284600705228826 1loss/FBt1/assignment_loss‚0.48452574679807486 1loss/FBt2/assignment_loss‚0.02343964407025161 /loss/FRt2/detection_loss‚2.4363685772624066 epoch‚0 trainer/global_step‚4349 /loss/FRt1/assignment_loss‚3.378507575961981 1loss/SRqqt1/assignment_loss‚1.344950866036945 /loss/FBt1/detection_loss‚11.095731859405836 % _timestamp‚1.7858076572235212e+09ú†‹ê® € 1loss/SRqqt2/assignment_loss‚1.600684832678073 .loss/FRt1/detection_loss‚2.265652567650495 /loss/FBt2/detection_loss‚1.8358680444673217 %loss/total_loss‚30.58706922668645 _runtime‚1935.515052293 /loss/FRt2/assignment_loss‚3.284600705228826 1loss/FBt1/assignment_loss‚0.48452574679807486 1loss/FBt2/assignment_loss‚0.02343964407025161 /loss/FRt2/detection_loss‚2.4363685772624066 epoch‚0 trainer/global_step‚4349 /loss/FRt1/assignment_loss‚3.378507575961981 1loss/SRqqt1/assignment_loss‚1.344950866036945 /loss/FBt1/detection_loss‚11.095731859405836 % _timestamp‚1.7858076572235212e+09 1loss/SRqqt1/detection_loss‚0.7262341185346447 0loss/SRqqt2/detection_loss‚2.110504688591593 _step‚347ÛöæóüŽ¯ ‚(:ƒ ·†ÅÓˆëÐÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚154.98' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚86.1 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cv‰Š° ‚(:ÿ ·†ÅÓÁçŠÕ) disk./.usagePercent‚83.26221896312114! disk.dm-0.out‚7323.4697265625 disk.rbd5.out‚ 0.919921875& disk.nvme0c0n1.out‚7323.4697265625 cpu‚13.93882687344083 network.recv‚279228 disk.rbd3.in‚0 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚ 0.919921875 disk.sda3.in‚0 disk.sda3.out‚ 97.0234375 disk.dm-1.out‚ 97.0234375 disk.rbd3.out‚0 disk.rbd2.in‚0$ memory_percent‚30.06550669670105 proc.cpu.threads‚29 disk.rbd4.in‚0* proc.memory.availableMB‚45832.26953125 network.sent‚ 4.085843e+06$ disk.nvme0n1.out‚7323.4697265625 disk.rbd4.out‚ 0.919921875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875* proc.memory.percent‚17.193293571472168 disk.drbd1743.in‚0 disk.drbd1743.out‚ 2.17578125 disk.nvme0n1.in‚ 28.33984375 disk.sda1.out‚0% disk./.usageGB‚1240.6672325134277 disk.dm-0.in‚ 28.33984375 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 28.33984375 disk.sda.out‚ 97.0234375 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 11267.796875>I0r2± - _wandbruntime‚1957  _runtime‚1957vÄvÿɲ à % _timestamp‚1.7858076796794534e+09 _runtime‚1957.971137898 _step‚348 ' lr-AdamW/pg1‚0.00011337184834886152 ' lr-AdamW/pg2‚0.00011337184834886152 trainer/global_step‚4399×Àúγ È ' lr-AdamW/pg1‚0.00011337184834886152 ' lr-AdamW/pg2‚0.00011337184834886152 trainer/global_step‚4399 % _timestamp‚1.7858076796794534e+09 _runtime‚1957.971137898 _step‚348ܢЛ­2´ - _wandbruntime‚1957  _runtime‚1957ëâ:êþµ ¸O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75694602 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29233779 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77523187393080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858076796804512e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3871256269312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29923400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77523084550596 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3861781665124 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29233777 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3975915151360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861867468248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75717163158528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29923646 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75538282875904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46470481 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46460825 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3976047922688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29233777 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75704260 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75717138254848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79409539145216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79409643614208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75538282875904 \ODeviceStatsMoni²WϽÀtor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17325197 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79409539145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29233779 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12598449 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚4399 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79693211081216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73661302885472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚1957.975982166 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46470481 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79409539145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29233779 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12598449 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚4399 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79693211081216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73661302885472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _runtime‚1957.975982166 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46470481 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29233777 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3975915151360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861867468248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75717163158528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29923646 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75538282875904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46470481 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46460825 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3976047922688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29233777 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75704260 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75717138254848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79409539145216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79409643614208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75538282875904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130ÝfÊ" 2· - _wandbruntime‚1958  _runtime‚1958éàÉÈ&¸ ÂM ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75555466678272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75555449638912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79427585527808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73678060097152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73678043057792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79427690003968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3872223325696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75721440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17329132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12601015 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3872135888896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29240422 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3862659138346 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3872223325696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75711768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75721440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17329134 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3862744941582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75555466678272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75734345371648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3976819034624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _runtime‚1958.330182029 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29240420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚350 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79711164406272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29240422 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75555449638912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858076800380964e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3872135888896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29930147 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29930377 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77540805038734 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75711768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640Y.Ž`¹ ÇM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46471348 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79427690003968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46481018 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79711319976960 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3872135888896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29930147 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29930377 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77540805038734 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75711768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75555466678272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75555449638912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79427585527808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73678060097152 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73678043057792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79427690003968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3872223325696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75721440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17329132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12601015 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3872135888896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29240422 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3862659138346 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3872223325696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:Devicè+!Im eStatsMonitor.on_train_batch_end/allocation.all.freed‚75711768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75721440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17329134 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3862744941582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75555466678272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75734345371648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3976819034624 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 _runtime‚1958.330182029 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29240420 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚350 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79711164406272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29240422 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75555449638912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858076800380964e+09 Fnu2º - _wandbruntime‚1958  _runtime‚1958Ó 7C» ú 0loss/SRqqt2/detection_loss‚2.282109191183184 epoch‚0 $ _timestamp‚1.785807680039409e+09 2loss/SRqqt2/assignment_loss‚1.6376951004638047 0loss/FRt2/assignment_loss‚3.3283328589271095 0loss/FBt1/assignment_loss‚0.5802200060097675 -loss/FBt1/detection_loss‚3.36367130279541 %loss/total_loss‚23.59394539952173 trainer/global_step‚4399 _runtime‚1958.330877123 _step‚351 0loss/FRt1/assignment_loss‚3.3195164722554824 2loss/SRqqt1/assignment_loss‚1.4403536226600409 1loss/FBt2/assignment_loss‚0.05731537770797904 .loss/FRt1/detection_loss‚2.290600427459268 1loss/SRqqt1/detection_loss‚1.0351706185026894 /loss/FBt2/detection_loss‚1.7547627889917754 .loss/FRt2/detection_loss‚2.504197632565218…ÒÞœ¼ ÿ 0loss/FRt2/assignment_loss‚3.3283328589271095 0loss/FBt1/assignment_loss‚0.5802200060097675 -loss/FBt1/detection_loss‚3.36367130279541 %loss/total_loss‚23.59394539952173 trainer/global_step‚4399 _runtime‚1958.330877123 _step‚351 0loss/FRt1/assignment_loss‚3.3195164722554824 2loss/SRqqt1/assignment_loss‚1.4403536226600409 1loss/FBt2/assignment_loss‚0.05731537770797904 .loss/FRt1/detection_loss‚2.290600427459268 1loss/SRqqt1/detection_loss‚1.0351706185026894 /loss/FBt2/detection_loss‚1.7547627889917754 .loss/FRt2/detection_loss‚2.504197632565218 0loss/SRqqt2/detection_loss‚2.282109191183184 epoch‚0 $ _timestamp‚1.785807680039409e+09 2loss/SRqqt2/assignment_loss‚1.6376951004638047ߺ˜<µœ½ ‚(:‘ ƆÅÓ¸ŒÈÒ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚130.927' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚72.73722222222221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼™qª¾ ‚(:„  ƆÅÓáÛÕ disk.rbd0.out‚0 disk.rbd4.in‚0 disk.sda3.out‚ 97.7265625 disk.dm-1.out‚ 97.7265625 disk.rbd1.in‚0 disk.sda2.out‚0) disk./.usagePercent‚83.26231291788622% disk.nvme0n1.out‚7323.91259765625' disk.nvme0c0n1.out‚7323.91259765625 disk.rbd2.out‚ 0.919921875% memory_percent‚30.065983533859253% disk./.usageGB‚1240.6686325073242 disk.nvme0n1.in‚ 28.33984375 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1743.out‚ 2.17578125" proc.memory.rssMB‚ 11267.796875 cpu‚13.938544110445878 disk.rbd4.out‚ 0.919921875" disk.dm-0.out‚7323.91259765625 proc.cpu.threads‚29 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 97.7265625 disk.drbd1743.in‚0 disk.dm-0.in‚ 28.33984375* proc.memory.availableMB‚45831.95703125 network.sent‚ 4.120219e+06 network.recv‚281096 disk.rbd5.out‚ 0.919921875! disk.nvme0c0n1.in‚ 28.33984375 disk.rbd3.in‚0 disk.rbd0.in‚03Í…Üœ¿ ‚(:‘ Õ†ÅÓÞÎäÒ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚107.951' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚59.97277777777777 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OB­ŽÀ ‚(:ƒ  Õ†ÅÓ£¨¡× disk.sda.out‚ 98.41796875% memory_percent‚30.065631866455078 disk.drbd1743.in‚0" disk.dm-0.out‚7386.54443359375 disk.sda.in‚0 network.sent‚ 4.165248e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.nvme0n1.in‚ 28.33984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 0.919921875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 98.41796875" proc.memory.rssMB‚ 11267.796875 disk.dm-0.in‚ 28.33984375 disk.rbd5.in‚0! disk.nvme0c0n1.in‚ 28.33984375 disk.sda3.in‚0 cpu‚13.937214034451381) disk./.usagePercent‚83.27069895668532 disk.sda2.out‚0 disk.drbd1743.out‚ 2.21484375% disk.nvme0n1.out‚7386.54443359375 disk.rbd4.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd5.out‚ 0.919921875' disk.nvme0c0n1.out‚7386.54443359375 network.recv‚283340 disk.rbd3.out‚0 disk.sda3.out‚ 98.41796875 proc.cpu.threads‚29% disk./.usageGB‚1240.7935905456543 disk.rbd4.out‚ 0.919921875& proc.memory.availableMB‚ 45832.1875A„rI2Á - _wandbruntime‚1982  _runtime‚1982·™Ð\É Ã trainer/global_step‚4449 % _timestamp‚1.7858077037730813e+09 _runtime‚1982.064733051 _step‚352 ' lr-AdamW/pg1‚0.00011466045767312683 ' lr-AdamW/pg2‚0.00011466045767312683Ôg,ÆÎà È ' lr-AdamW/pg1‚0.00011466045767312683 ' lr-AdamW/pg2‚0.00011466045767312683 trainer/global_step‚4449 % _timestamp‚1.7858077037730813e+09 _runtime‚1982.064733051 _step‚352àÝÙ–2Ä - _wandbruntime‚1982  _runtime‚1982ˆ…½'Å ·O SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚1982.066184075 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80311873431040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17522047 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4021094370304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78404084472588 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46988827 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80598588465152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46998483 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76396638065664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4021231335936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46988827 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78403981630104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76396621026304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29565929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76396638065664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46998483 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3905755931756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76554754 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3915252404736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80311873431040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76564412 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29565927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74498311501472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30262723 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12740432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29565929 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76577494094848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30262477 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12740676 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905670128632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80311873431040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17522047 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4021094370304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78404084472588 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46988827 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80598588465152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46998483 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76396638065664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4021231335936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46988827 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78403981630104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76396621026304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29565929 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76396638065664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46998483 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3905755931756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76554754 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3915252404736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80311873431040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76564412 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29565927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74498311501472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30262723 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12740432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29565929 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76577494094848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30262477 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12740676á©j@Ü2Ç - _wandbruntime‚1982  _runtime‚1982@âì`ëÈ ÂM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚4449 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46999350 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74515068713152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3916132024320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17525984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47009020 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80330024289792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80616857427968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76581592 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29572570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3916219461120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80330024289792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76581592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29572570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3916219461120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚1982.523108029 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30269473 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78421702118242 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29572572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12743489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29572572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚354 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76413787789312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17525982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4021996451328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30269219 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76571920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76571920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47009020 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130*"$[Í&É ÇM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76413804828672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29572570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3916219461120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80330024289792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76581592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29572570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3916219461120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚1982.523108029 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30269473 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78421702118242 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29572572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12743489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29572572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚354 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76413787789312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17525982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4021996451328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30269219 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76571920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76571920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47009020 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚4449 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46999350 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74515068713152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3916132024320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17525984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47009020 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80330024289792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80616857427968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76581592 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47516829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚79284878709612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81214312185856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29898079 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29898077 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30602014 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75335337156832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3959248540160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47526485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47516829 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12883375 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81504125272064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81214312185856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚2004.620276909 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81214207716864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12883119 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75335320117472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3959335969792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77254976216064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47526485 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81214207716864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77424564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚4499 % _timestamp‚1.7858077263283522e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17718897 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3949558592140 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77437874838528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚79284981552096 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29898079 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29898077 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30602272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77254959176704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3949644395264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4066275337216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17718895 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77414906 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3959248540160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77254959176704 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4066406011392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77254976216064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚77254976216064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47526485 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚81214207716864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77424564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚4499 % _timestamp‚1.7858077263283522e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17718897 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3949558592140 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77437874838528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚79284981552096 O@DeviceStatsMonitor.on_-%ïëtrain_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29898079 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29898077 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30602272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77254959176704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3949644395264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4066275337216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17718895 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77414906 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3959248540160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚77254959176704 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4066406011392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77254976216064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47516829 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚79284878709612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81214312185856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29898079 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29898077 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30602014 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75335337156832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3959248540160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47526485 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47516829 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12883375 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81504125272064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81214312185856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚2004.620276909 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81214207716864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12883119 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75335320117472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121屺¿Ä2Õ - _wandbruntime‚2004  _runtime‚2004ð¾ÌÇ&Ö ÁM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3960128159744 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142987776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4499 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29904722 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77441744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77432072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3950521868598 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29904722 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77272125939712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30609013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29904720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118084096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75352077329152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77272142979072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81232254099456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77272142979072 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81232358575616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29904720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17722834 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77455081955328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81522242268672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77455057051648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚358 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47527352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3960215596544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77441744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47537022 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30608777 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75352060289792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29904722 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77272125939712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30609013 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29904720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 118084096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75352077329152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77272142979072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81232254099456 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77272142979072 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81232358575616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29904720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17722834 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77455081955328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81522242268672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77455057051648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚358 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47527352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3960215596544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77441744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47537022 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30608777 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75352060289792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0æ3Nóº2Ø - _wandbruntime‚2004  _runtime‚2004ƒÔŒ‚Ù û -loss/FRt2/detection_loss‚2.47182555431905 1loss/SRqqt2/detection_loss‚1.9928268093458363 _step‚359 0loss/FRt2/assignment_loss‚3.5059461982353874 2loss/SRqqt1/assignment_loss‚1.5494659593105315 0loss/SRqqt1/detection_loss‚1.099532869890662 /loss/FBt2/detection_loss‚1.6077791289581607 &loss/total_loss‚28.221348974434843 % _timestamp‚1.7858077266914504e+09 _runtime‚ 2004.98296261 1loss/SRqqt2/assignment_loss‚1.656816337519968 0loss/FBt1/assignment_loss‚0.4997527400652568 epoch‚0 trainer/global_step‚4499 /loss/FRt1/assignment_loss‚4.136383629706969 1loss/FBt2/assignment_loss‚0.03551516796863628 /loss/FRt1/detection_loss‚2.3734932386551995 /loss/FBt1/detection_loss‚7.2920113404591875Ü6Ú € &loss/total_loss‚28.221348974434843 % _timestamp‚1.7858077266914504e+09 _runtime‚ 2004.98296261 1loss/SRqqt2/assignment_loss‚1.656816337519968 0loss/FBt1/assignment_loss‚0.4997527400652568 epoch‚0 trainer/global_step‚4499 /loss/FRt1/assignment_loss‚4.136383629706969 1loss/FBt2/assignment_loss‚0.03551516796863628 /loss/FRt1/detection_loss‚2.3734932386551995 /loss/FBt1/detection_loss‚7.2920113404591875 -loss/FRt2/detection_loss‚2.47182555431905 1loss/SRqqt2/detection_loss‚1.9928268093458363 _step‚359 0loss/FRt2/assignment_loss‚3.5059461982353874 2loss/SRqqt1/assignment_loss‚1.5494659593105315 0loss/SRqqt1/detection_loss‚1.099532869890662 /loss/FBt2/detection_loss‚1.6077791289581607çEœÛ ‚(:‘ ó†ÅÓ±ÛÝÒ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚152.974' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.98555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹/SuÜ ‚(:ê ó†ÅÓ…·Õ disk.rbd4.in‚0" proc.memory.rssMB‚ 11267.796875) disk./.usagePercent‚83.27059168952849 disk.rbd2.in‚0 disk.rbd2.out‚ 0.919921875 disk.sda1.in‚0 cpu‚13.938075800253166 disk.sda3.out‚99.75% disk.nvme0n1.out‚7388.53955078125 proc.cpu.threads‚29 network.recv‚286500 disk.rbd3.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 0.919921875 disk.rbd0.in‚0$ memory_percent‚29.37568426132202* proc.memory.percent‚17.193293571472168" disk./.usageGB‚1240.7919921875 disk.dm-1.out‚99.75" disk.dm-0.out‚7388.53955078125 network.sent‚ 4.212564e+06 disk.sda.out‚99.75 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 28.33984375 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 28.33984375 disk.sda.in‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1743.in‚0 disk.drbd1743.out‚2.25 disk.sda1.out‚0 disk.rbd4.out‚ 0.919921875) proc.memory.availableMB‚ 46284.3515625' disk.nvme0c0n1.out‚7388.53955078125 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 28.33984375`àŒ'›Ý ‚(: ‚‡ÅÓ²ÌãÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚130.99' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚72.77222222222223 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~Ù¤#ˆÞ ‚(:ý ‚‡ÅÓµÒÈ× disk.sda1.out‚0 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 29.40234375 disk.dm-0.out‚7451.783203125 disk.sda1.in‚0% memory_percent‚28.532785177230835 disk.sda.in‚0 disk.rbd2.out‚ 0.919921875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd5.out‚ 0.919921875 disk.rbd4.in‚0* proc.memory.percent‚17.193293571472168 disk.rbd3.out‚0 disk.rbd0.in‚0* proc.memory.availableMB‚46836.75390625 proc.cpu.threads‚29) disk./.usagePercent‚83.27060039378465! disk.nvme0c0n1.in‚ 29.40234375# disk.nvme0n1.out‚7451.783203125" proc.memory.rssMB‚ 11267.796875$ disk./.usageGB‚1240.792121887207% disk.nvme0c0n1.out‚7451.783203125 disk.sda.out‚ 100.41015625 disk.rbd2.in‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1743.out‚2.25 cpu‚13.937598679851396 disk.sda3.in‚0 disk.sda3.out‚ 100.41015625 disk.rbd4.out‚ 0.919921875 network.sent‚ 4.257333e+06 network.recv‚288523 disk.dm-1.out‚ 100.41015625 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 29.40234375GðË@2ß - _wandbruntime‚2027  _runtime‚2027¬8¡½Éà à ' lr-AdamW/pg1‚0.00011723767632165744 ' lr-AdamW/pg2‚0.00011723767632165744 trainer/global_step‚4549 % _timestamp‚1.7858077488577943e+09 _runtime‚2027.149512755 _step‚360€Td‚Îá È _step‚360 ' lr-AdamW/pg1‚0.00011723767632165744 ' lr-AdamW/pg2‚0.00011723767632165744 trainer/global_step‚4549 % _timestamp‚1.7858077488577943e+09 _runtime‚2027.149512755èf6L2â - _wandbruntime‚2027  _runtime‚202743ö'xã ¸O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82116646471680 ^LDeviceStatsMonitor.oÊä1œF'n_train_batch_start/allocated_bytes.small_pool.allocated‚ 4003332105216 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚361 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82116542002688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30230229 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80165775789120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4003332105216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78113314366464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3993532858772 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82116542002688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78275058 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48044831 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17915745 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76172345772832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17915747 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80165878631604 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78113297327104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78113297327104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76172328733472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858077488589432e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48054487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 trainer/global_step‚4549 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4003332105216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78113314366464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3993532858772 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82116542002688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78275058 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48044831 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17915745 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76172345772832 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17915747 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80165878631604 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78113297327104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78113297327104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76172328733472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858077488589432e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48054487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 trainer/global_step‚4549 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30236872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48065024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3994324528870 _step‚362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4004124295168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48055354 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78301896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17919684 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 trainer/global_step‚4549 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82134692861440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82427778539008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78130481129472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78315412891648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76189068905792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13029740 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80183496277258 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30236870 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78130464090112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30949422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17919682 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78292224 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82134588385280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76189085945152 _runtime‚2027.551737564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 €¡FÚ ç ÇM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78130464090112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78130481129472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4112365647360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30236872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48065024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3994324528870 _step‚362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4004124295168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48055354 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78301896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17919684 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 trainer/global_step‚4549 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82134692861440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82427778539008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78130481129472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78315412891648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76189068905792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13029740 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80183496277258 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30236870 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78130464090112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30949422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17919682 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78292224 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82134588385280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76189085945152 _runtime‚2027.551737564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856êzS(2è - _wandbruntime‚2027  _runtime‚2027†áP”é ý 2loss/SRqqt2/assignment_loss‚1.4339469919602077 .loss/FRt2/detection_loss‚2.629208083215513 1loss/SRqqt1/detection_loss‚0.9250977220821938 % _timestamp‚1.7858077492614486e+09 _runtime‚2027.553143591 _step‚363 2loss/SRqqt1/assignment_loss‚1.3921790689457463 0loss/FBt1/assignment_loss‚0.5609887561627797 /loss/FBt1/detection_loss‚1.9529272403035844 trainer/global_step‚4549 0loss/FRt2/assignment_loss‚3.6463770772281445 -loss/FRt1/detection_loss‚2.30878194926799 /loss/FBt2/detection_loss‚1.5825443800178267 &loss/total_loss‚22.374339015590536 epoch‚0 /loss/FRt1/assignment_loss‚3.763896633838785 1loss/FBt2/assignment_loss‚0.02141094675114859 1loss/SRqqt2/detection_loss‚2.1569801658166177®òéêê ‚ 2loss/SRqqt1/assignment_loss‚1.3921790689457463 0loss/FBt1/assignment_loss‚0.5609887561627797 /loss/FBt1/detection_loss‚1.9529272403035844 trainer/global_step‚4549 0loss/FRt2/assignment_loss‚3.6463770772281445 -loss/FRt1/detection_loss‚2.30878194926799 /loss/FBt2/detection_loss‚1.5825443800178267 &loss/total_loss‚22.374339015590536 epoch‚0 /loss/FRt1/assignment_loss‚3.763896633838785 1loss/FBt2/assignment_loss‚0.02141094675114859 1loss/SRqqt2/detection_loss‚2.1569801658166177 2loss/SRqqt2/assignment_loss‚1.4339469919602077 .loss/FRt2/detection_loss‚2.629208083215513 1loss/SRqqt1/detection_loss‚0.9250977220821938 % _timestamp‚1.7858077492614486e+09 _runtime‚2027.553143591 _step‚363ë÷¡aœë ‚(:‘ ‘‡ÅÓ¼€äÒ gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚149.188' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.88222222222221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„N‚ì ‚(:÷ ‘‡ÅÓ†´ÈÖ disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 0.919921875% disk.nvme0c0n1.out‚7452.130859375 network.sent‚ 4.25981e+06# disk.nvme0n1.out‚7452.130859375 disk.dm-1.in‚0 network.recv‚289726) disk./.usagePercent‚83.27061396218396 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 11267.796875 disk.rbd1.out‚ 0.919921875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 29.453125 disk.sda.in‚0 disk.rbd2.out‚ 0.919921875$ disk./.usageGB‚1240.792324066162 disk.dm-0.out‚7452.130859375 disk.nvme0n1.in‚ 29.453125 disk.rbd2.in‚0 disk.sda3.out‚ 101.0546875* proc.memory.availableMB‚46836.71484375* proc.memory.percent‚17.193293571472168 disk.rbd3.out‚0 disk.rbd4.out‚ 0.919921875 disk.sda.out‚ 101.0546875 disk.drbd1743.out‚ 2.29296875 disk.dm-0.in‚ 29.453125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 101.0546875$ memory_percent‚28.53284478187561 cpu‚13.93676377071018 proc.cpu.threads‚29 disk.sda2.out‚0Æœ©2í - _wandbruntime‚2049  _runtime‚2049åDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18112597 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13170394 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79144868 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77009337349472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48572833 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31282991 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚4599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79135210 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83018980757504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30562377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81046672868628 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78971652516864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78971635477504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83018876288512 $ _timestamp‚1.785807771554598e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4047240811008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156775778816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48582489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4047328240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79158586518528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚2049.849957966 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13170129 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79158561614848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚365 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31282724 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4037421322280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83018876288512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83315206719488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48572833 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83018980757504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30562379 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81046775711112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48582489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4037335519156 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30562379 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156645104640{c«Â'ò ¼O [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4047328240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18112597 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13170394 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79144868 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77009337349472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48572833 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31282991 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚4599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79135210 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83018980757504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30562377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81046672868628 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78971652516864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78971635477504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83018876288512 $ _timestamp‚1.785807771554598e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4047240811008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156775778816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48582489 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4047328240640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79158586518528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚2049.849957966 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13170129 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79158561614848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚365 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31282724 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4037421322280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83018876288512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚83315206719488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48572833 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83018980757504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30562379 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81046775711112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48582489 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4037335519156 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30562379 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156645104640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18112595 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78971652516864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78971635477504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4047240811008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130í¶W2ó - _wandbruntime‚2050  _runtime‚2050õ™2¦ô ÁM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30569022 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81064393356766 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48593026 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31289678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.smaT†'F!"ll_pool.freed‚13172899 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4048207867392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18116534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4038298795614 _step‚366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13173144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48583356 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2050.21648892 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79152376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157684152320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83036922671104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78988802240512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83333315925504 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81064290514170 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79152376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4048120430592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18116532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83333477787648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79162048 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31289431 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858077719244633e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83037027147264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78988819279872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79175768731648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78988819279872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48593026 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83037027147264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79162048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30569022 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4599 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4048120430592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4038212992378 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4048207867392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77026094561152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30569020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79175793635328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0BŽÌ&õ ÆM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79162048 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31289431 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858077719244633e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83037027147264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78988819279872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79175768731648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78988819279872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48593026 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83037027147264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79162048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30569022 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4599 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4048120430592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4038212992378 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4048207867392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77026094561152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30569020 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79175793635328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30569022 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81064393356766 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48593026 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31289678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13172899 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4048207867392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18116534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4038298795614 _step‚366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13173144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48583356 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2050.21648892 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79152376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157684152320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83036922671104 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78988802240512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83333315925504 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81064290514170 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79152376 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4048120430592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18116532 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83333477787648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0î^/üÝ2ö - _wandbruntime‚2050  _runtime‚2050О<÷ ü 0loss/FBt1/assignment_loss‚0.5143777611311424 /loss/FBt2/detection_loss‚1.1911584349981224 $ _timestamp‚1.785807771925883e+09 2loss/SRqqt2/assignment_loss‚1.3211863643052626 2loss/FBt2/assignment_loss‚0.043450365071193014 1loss/SRqqt2/detection_loss‚2.3921500712024923 .loss/FBt1/detection_loss‚1.479692397802616 _runtime‚2050.217483396 0loss/FRt1/assignment_loss‚3.7272766656947858 1loss/SRqqt1/detection_loss‚1.0047132319768004 epoch‚0 &loss/total_loss‚22.101630952853007 /loss/FRt2/assignment_loss‚3.598656682225017 1loss/SRqqt1/assignment_loss‚1.511471174265209 .loss/FRt1/detection_loss‚2.649157553911209 .loss/FRt2/detection_loss‚2.668340250269159 trainer/global_step‚4599 _step‚367s1bø  0loss/FRt1/assignment_loss‚3.7272766656947858 1loss/SRqqt1/detection_loss‚1.0047132319768004 epoch‚0 &loss/total_loss‚22.101630952853007 /loss/FRt2/assignment_loss‚3.598656682225017 1loss/SRqqt1/assignment_loss‚1.511471174265209 .loss/FRt1/detection_loss‚2.649157553911209 .loss/FRt2/detection_loss‚2.668340250269159 trainer/global_step‚4599 _step‚367 0loss/FBt1/assignment_loss‚0.5143777611311424 /loss/FBt2/detection_loss‚1.1911584349981224 $ _timestamp‚1.785807771925883e+09 2loss/SRqqt2/assignment_loss‚1.3211863643052626 2loss/FBt2/assignment_loss‚0.043450365071193014 1loss/SRqqt2/detection_loss‚2.3921500712024923 .loss/FBt1/detection_loss‚1.479692397802616 _runtime‚2050.217483396ïníf‹œù ‚(:‘  ‡ÅÓìŸàÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.682' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.04555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eëm…ú ‚(:ú  ‡ÅÓû„£Õ' disk.nvme0c0n1.out‚7452.38232421875 network.sent‚ 4.304628e+06 disk.sda2.in‚0 disk.drbd1743.out‚ 2.29296875" disk.dm-0.out‚7452.38232421875 disk.rbd0.out‚0% disk./.usageGB‚1240.7923049926758 disk.rbd1.out‚ 0.919921875 disk.rbd4.in‚0 network.recv‚291541 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.919921875 disk.rbd0.in‚0 disk.sda.out‚ 101.609375 disk.sda3.out‚ 101.609375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 29.453125 disk.rbd2.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 11267.796875 cpu‚13.93681528807282 disk.drbd1743.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd2.out‚ 0.919921875 disk.dm-1.out‚ 101.609375$ memory_percent‚28.53376865386963 disk.rbd4.out‚ 0.919921875 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 29.453125% disk.nvme0n1.out‚7452.38232421875( proc.memory.availableMB‚ 46836.109375) disk./.usagePercent‚83.27061268214628 disk.nvme0c0n1.in‚ 29.453125* proc.memory.percent‚17.193293571472168 proc.cpu.threads‚29$ ¹Pœû ‚(:‘ ¯‡ÅÓ¹íæÒ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚140.258' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚77.92111111111112 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ8Ab‰ü ‚(:þ ¯‡ÅÓðó¡× network.sent‚ 4.349251e+06 disk.dm-0.in‚ 29.53515625% memory_percent‚28.503704071044922& proc.memory.availableMB‚ 46855.8125" proc.memory.rssMB‚ 11267.796875$ disk./.usageGB‚1241.042324066162 disk.rbd1.out‚ 0.919921875 disk.drbd1743.in‚0$ disk.nvme0n1.out‚7515.2451171875 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd4.out‚ 0.919921875 disk.sda3.in‚0 disk.sda3.out‚ 102.5390625 cpu‚13.935438782541535 disk.drbd1743.out‚ 2.3203125 disk.sda1.in‚0& disk.nvme0c0n1.out‚7515.2451171875 disk.dm-1.out‚ 102.5390625 proc.cpu.threads‚29 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 29.53515625 disk.rbd5.out‚ 0.919921875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 29.53515625 disk.rbd2.in‚0 disk.sda.out‚ 102.5390625 network.recv‚293528 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.193293571472168) disk./.usagePercent‚83.28739167194793 disk.rbd1.in‚0! disk.dm-0.out‚7515.2451171875 disk.rbd0.out‚0 disk.rbd2.out‚ 0.919921875»JÐÖ2ý - _wandbruntime‚2073  _runtime‚2073ÆœNÛÉþ à _runtime‚2073.411887452 _step‚368 ' lr-AdamW/pg1‚0.00011981489497018804 ' lr-AdamW/pg2‚0.00011981489497018804 trainer/global_step‚4649 % _timestamp‚1.7858077951198177e+09±OöÎÿ È ' lr-AdamW/pg1‚0.00011981489497018804 ' lr-AdamW/pg2‚0.00011981489497018804 trainer/global_step‚4649 % _timestamp‚1.7858077951198177e+09 _runtime‚2073.411887452 _step‚368ðKðè2€ - _wandbruntime‚2073  _runtime‚2073žp-P5" ¸O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79829973627904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30894527 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49110491 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49110491 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4091236946432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49100835 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18309445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4201960795136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77846363004832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4091236946432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4091324376064 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83921210574336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4091324376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80005020 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79829990667264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31622236 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13312791 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13313031 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79829973627904 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83921210574336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49100835 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚2073.413704982 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4081309785788 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30894529 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83921315043328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18309447 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81927569948136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79995362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 % _timestamp‚1.7858077951215193e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4201825926656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81927672790620 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚369 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640óðôÃ'‚ ½O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4081309785788 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30894529 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83921315043328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18309447 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚81927569948136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79995362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 % _timestamp‚1.7858077951215193e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4201825926656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81927672790620 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚369 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79829973627904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30894527 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49110491 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49110491 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4091236946432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49100835 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18309445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4201960795136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77846363004832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4091236946432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4091324376064 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83921210574336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4091324376064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80005020 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79829990667264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31622236 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13312791 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13313031 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79829973627904 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83921210574336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49100835 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _runtime‚2073.413704982 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392ñ`DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80022200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30901170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18313384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80036124571648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30901172 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4092116566016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18313382 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49121028 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77863086137792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082187259122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79847140390912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31628890 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30901172 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 trainer/global_step‚4649 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79847157430272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4202741860352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13315775 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 _runtime‚2073.881577743 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83939361433088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77863103177152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4082101455886 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80022200 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79847140390912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4092204002816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4202874624512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80012528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49111358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 $ _timestamp‚1.785807795588826e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30901170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83939361433088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80012528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81945187593678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4092204002816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49111358 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80012528 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49111358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 $ _timestamp‚1.785807795588826e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30901170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83939361433088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80012528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81945187593678 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4092204002816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49111358 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80022200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30901170 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18313384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80036124571648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30901172 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4092116566016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18313382 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49121028 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77863086137792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082187259122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚79847140390912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31628890 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30901172 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 trainer/global_step‚4649 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79847157430272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4202741860352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13315775 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 _runtime‚2073.881577743 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83939361433088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77863103177152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856òü¥Â}2† - _wandbruntime‚2073  _runtime‚2073mÀK“‡ € 0loss/FRt1/assignment_loss‚3.4182548792738663 0loss/FRt2/assignment_loss‚3.5828395514261153 /loss/FBt2/detection_loss‚1.5429885065458055 &loss/total_loss‚22.115776587105977 epoch‚0 % _timestamp‚1.7858077955910826e+09 _step‚371 1loss/FBt1/assignment_loss‚0.44427994315758823 /loss/FRt2/detection_loss‚2.5398871146497273 1loss/SRqqt1/detection_loss‚0.8831694084367493 0loss/SRqqt2/detection_loss‚2.026690396524611 2loss/SRqqt1/assignment_loss‚1.3780947865497681 /loss/FRt1/detection_loss‚2.1494368090441354 .loss/FBt1/detection_loss‚2.349218027932303 _runtime‚2073.882766604 2loss/SRqqt2/assignment_loss‚1.7330935119636475 1loss/FBt2/assignment_loss‚0.06782365160165682 trainer/global_step‚46498çꪈ … 1loss/SRqqt1/detection_loss‚0.8831694084367493 0loss/SRqqt2/detection_loss‚2.026690396524611 2loss/SRqqt1/assignment_loss‚1.3780947865497681 /loss/FRt1/detection_loss‚2.1494368090441354 .loss/FBt1/detection_loss‚2.349218027932303 _runtime‚2073.882766604 2loss/SRqqt2/assignment_loss‚1.7330935119636475 1loss/FBt2/assignment_loss‚0.067823651601e ûŒ65682 trainer/global_step‚4649 0loss/FRt1/assignment_loss‚3.4182548792738663 0loss/FRt2/assignment_loss‚3.5828395514261153 /loss/FBt2/detection_loss‚1.5429885065458055 &loss/total_loss‚22.115776587105977 epoch‚0 % _timestamp‚1.7858077955910826e+09 _step‚371 1loss/FBt1/assignment_loss‚0.44427994315758823 /loss/FRt2/detection_loss‚2.5398871146497273óvy'xŽ‰ ‚(:ƒ ¾‡ÅÓÓúßÒ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚117.72' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚65.4 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Ð6“Š ‚(:„  ¾‡ÅÓ¤ÁìÖ cpu‚13.934056015790594 disk.drbd1743.in‚0% disk.nvme0n1.out‚7515.58056640625 disk.rbd4.in‚0! disk.nvme0c0n1.in‚ 29.53515625' disk.nvme0c0n1.out‚7515.58056640625 disk.dm-0.in‚ 29.53515625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda3.in‚0 disk.rbd5.out‚ 0.919921875 disk.rbd4.out‚ 0.919921875 disk.dm-1.out‚ 103.109375 network.recv‚294401% disk./.usageGB‚1241.0423049926758 disk.drbd1743.out‚ 2.32421875" disk.dm-0.out‚7515.58056640625 disk.nvme0n1.in‚ 29.53515625 disk.rbd5.in‚0" proc.memory.rssMB‚ 11267.609375 disk.rbd0.out‚0 disk.rbd2.in‚0* proc.memory.percent‚17.193007469177246) disk./.usagePercent‚83.28739039191026 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 103.109375 disk.rbd1.out‚ 0.919921875 disk.sda2.in‚0 proc.cpu.threads‚29 network.sent‚ 4.351726e+06 disk.rbd0.in‚0 disk.rbd2.out‚ 0.919921875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0% memory_percent‚28.503984212875366* proc.memory.availableMB‚46855.62890625 disk.sda.in‚0 disk.sda3.out‚ 103.109375_B2‹ - _wandbruntime‚2097  _runtime‚2097,‹~LÉŒ à trainer/global_step‚4699 % _timestamp‚1.7858078193621166e+09 _runtime‚2097.653889722 _step‚372 ' lr-AdamW/pg1‚0.00012110350429445335 ' lr-AdamW/pg2‚0.00012110350429445335ʧÉüÎ È _runtime‚2097.653889722 _step‚372 ' lr-AdamW/pg1‚0.00012110350429445335 ' lr-AdamW/pg2‚0.00012110350429445335 trainer/global_step‚4699 % _timestamp‚1.7858078193621166e+09ôœÆéØ2Ž - _wandbruntime‚2097  _runtime‚2097 ²ë½' ·O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78683371620832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _step‚373 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80688311778304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80879273294848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78683354581472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13456193 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4125112446172 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80855514 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80688311778304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18506297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49638493 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4135233081856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4125198249296 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84823649329152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31962228 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84823544860160 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18506295 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85126327293952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49638493 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49628837 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 trainer/global_step‚4699 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13455933 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858078193632097e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84823649329152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121¬!`Â' ¼O F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84823544860160 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18506295 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85126327293952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49638493 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49628837 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 trainer/global_step‚4699 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13455933 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858078193632097e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84823649329152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78683371620832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 _step‚373 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80688311778304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80879273294848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78683354581472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13456193 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4125112446172 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80855514 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80688311778304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18506297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49638493 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4135233081856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4125198249296 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84823649329152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31962228õ´Ûû2‘ - _wandbruntime‚2098  _runtime‚2098ð‹ÜO^&’ ÂM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80872680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85144441783808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80896505315328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4125989919394 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82826187515782 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84841591242752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31969286 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49639360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80705495580672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80882352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31969032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80872680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31233322 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49649030 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31233320 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84841695718912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80705495580672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚2098.022705881 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80705478541312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13458800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31233322 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4136200138240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82826084673186 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4247961372160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13459052 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80896480411648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80705478541312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4248092039168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4699 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78700111793152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18510234 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4126075722630 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78700094753792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84841695718912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18510232 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80882352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49649030 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49639360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85144597354496 JBDeviceStatsMonitor.on_train_batch_endë¯ô,j/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121í´ Í&“ ÇM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80896480411648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80705478541312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4248092039168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4699 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78700111793152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18510234 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4126075722630 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78700094753792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84841695718912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18510232 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80882352 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49649030 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49639360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85144597354496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80872680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85144441783808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80896505315328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4125989919394 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82826187515782 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84841591242752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31969286 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49639360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80705495580672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80882352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31969032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80872680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31233322 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49649030 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31233320 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84841695718912 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80705495580672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚2098.022705881 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80705478541312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13458800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31233322 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 _step‚374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4136200138240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82826084673186 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4247961372160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13459052ö(BPÈ2” - _wandbruntime‚2098  _runtime‚2098Éwäè• ü 1loss/FBt1/assignment_loss‚0.36548007966503065 1loss/FBt2/assignment_loss‚0.06761220111250599 epoch‚0 % _timestamp‚1.7858078197320395e+09 _step‚375 .loss/FRt1/assignment_loss‚3.99644688634496 2loss/SRqqt1/assignment_loss‚1.4761267518767944 /loss/FRt2/detection_loss‚2.5628785567228185 1loss/SRqqt2/detection_loss‚2.6674587612456464 /loss/FRt2/assignment_loss‚3.503436081631239 2loss/SRqqt2/assignment_loss‚1.7558314578330263 1loss/SRqqt1/detection_loss‚1.0492134454233457 /loss/FBt2/detection_loss‚1.3378749838212152 trainer/global_step‚4699 _runtime‚ 2098.02352597 .loss/FRt1/detection_loss‚2.593654856477913 .loss/FBt1/detection_loss‚1.362888765335083 &loss/total_loss‚22.738902827489575w€Z½–  .loss/FRt1/assignment_loss‚3.99644688634496 2loss/SRqqt1/assignment_loss‚1.4761267518767944 /loss/FRt2/detection_loss‚2.5628785567228185 1loss/SRqqt2/detection_loss‚2.6674587612456464 /loss/FRt2/assignment_loss‚3.503436081631239 2loss/SRqqt2/assignment_loss‚1.7558314578330263 1loss/SRqqt1/detection_loss‚1.0492134454233457 /loss/FBt2/detection_loss‚1.3378749838212152 trainer/global_step‚4699 _runtime‚ 2098.02352597 .loss/FRt1/detection_loss‚2.593654856477913 .loss/FBt1/detection_loss‚1.362888765335083 &loss/total_loss‚22.738902827489575 1loss/FBt1/assignment_loss‚0.36548007966503065 1loss/FBt2/assignment_loss‚0.06761220111250599 epoch‚0 % _timestamp‚1.7858078197320395e+09 _step‚375÷囚— ‚(: ͇ÅÓ÷ÈæÒ gpu.0.gpu‚26.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚140.42' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚78.0111111111111 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Âö˜ ‚(:„  ͇ÅÓð±ØÕ% memory_percent‚28.504449129104614* proc.memory.availableMB‚46855.32421875! disk.nvme0c0n1.in‚ 29.53515625' disk.nvme0c0n1.out‚7515.83056640625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.rbd5.out‚ 0.919921875 disk.rbd0.in‚0 disk.sda3.out‚ 103.8046875% disk./.usageGB‚1241.0422439575195 disk.rbd2.in‚0 disk.rbd1.in‚0 network.sent‚ 4.396548e+06 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd1.out‚ 0.919921875 cpu‚13.933554658793774 disk.rbd3.in‚0 disk.sda.out‚ 103.8046875 disk.rbd2.out‚ 0.919921875 disk.sda3.in‚0 network.recv‚296552" disk.dm-0.out‚7515.83056640625 disk.nvme0n1.in‚ 29.53515625 disk.dm-1.out‚ 103.8046875* proc.memory.percent‚17.193031311035156) disk./.usagePercent‚83.28738629578972 disk.drbd1743.out‚ 2.34765625 disk.dm-0.in‚ 29.53515625 disk.rbd3.out‚0% disk.nvme0n1.out‚7515.83056640625 disk.sda2.out‚0 proc.memory.rssMB‚ 11267.625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 0.919921875 disk.sda.in‚0 disk.dm-1.in‚0œ·›™ ‚(: ܇ÅÓ°õîÒ gpu.0.gpu‚25.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚133.796' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚74.3311111111111 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ3>쌚 ‚(:  ܇ÅÓé²»Ö disk.rbd2.out‚ 0.919921875 disk.rbd1.out‚ 0.919921875$ proc.memory.rssMB‚11267.62890625 cpu‚13.93314784475396 disk.sda.out‚ 104.359375 network.recv‚298651 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚83.28739064791779 disk.rbd0.out‚0 disk.dm-0.in‚ 29.53515625$ memory_percent‚28.50414514541626" disk.dm-0.out‚7578.60791015625 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 29.53515625' disk.nvme0c0n1.out‚7578.60791015625 disk.sda3.out‚ 104.359375 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.rbd5.in‚0 disk.rbd2.in‚0 network.sent‚ 4.441312e+06 disk.rbd0.in‚0 disk.rbd4.out‚ 0.919921875% disk.nvme0n1.out‚7578.60791015625 disk.sda1.in‚0$ disk./.usageGB‚1241.042308807373 disk.dm-1.out‚ 104.359375 disk.rbd1.in‚0 disk.drbd1743.out‚ 2.3671875 disk.rbd3.out‚0 disk.rbd5.out‚ 0.919921875* proc.memory.percent‚17.193037271499634 disk.rbd4.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 29.53515625) proc.memory.availableMB‚ 46855.5234375 proc.cpu.threads‚29äì/K2› - _wandbruntime‚2120  _runtime‚2120•|“þÉœ à trainer/global_step‚4749 % _timestamp‚1.7858078422055871e+09 _runtime‚2120.497322196 _step‚376 ' lr-AdamW/pg1‚0.00012239211361871866 ' lr-AdamW/pg2‚0.00012239211361871866$³¼Î È _step‚376 ' lr-AdamW/pg1‚0.00012239211361871866 ' lr-AdamW/pg2‚0.00012239211361871866 trainer/global_step‚4749 % _timestamp‚1.7858078422055871e+09 _runtime‚2120.497322196øxu„”2ž - _wandbruntime‚2120  _runtime‚2120rùVV½'Ÿ ·O _runtime‚2120.499296971 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50156839 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85725983614976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85725879145984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4292365026816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4179229217280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83689466949636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31558827 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79520363197472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4292228061184 _step‚377 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50166495 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13598822 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32302221 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81739654038528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79520380236832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81546666968064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86031857196032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85725879145984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81546649928704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81546666968064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50166495 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86032019065344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4179316646912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32301967 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4169086712804 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50156839 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4749 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85725983614976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18703145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13599074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81715666 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31558829 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121Ø̳¨  ¼O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50166495 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13598822 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32302221 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81739654038528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79520380236832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81546666968064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86031857196032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85725879145984 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81546649928704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81546666968064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50166495 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86032019065344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4179316646912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32301967 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4169086712804 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50156839 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4749 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85725983614976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18703145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13599074 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81715666 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31558829 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚2120.499296971 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50156839 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85725983614976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85725879145984 cQDþ⦧¥ eviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4292365026816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4179229217280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83689466949636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31558827 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79520363197472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4292228061184 _step‚377ù[»¸]2¡ - _wandbruntime‚2120  _runtime‚2120$F8È&¢ ÂM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50177032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50167362 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81742504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50177032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4293265585152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81563816691712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18707084 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚2120.942234489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31565470 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85743925528576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81732832 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81756861155328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4180108836864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83706981752694 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85744030004736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32309018 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86050126740480 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4293130723840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81563833731072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13601697 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81563816691712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81742504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32308779 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0kcqyÍ&£ ÇM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81756836251648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4180108836864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79537103369792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81742504 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50177032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4293265585152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81563816691712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18707084 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚2120.942234489 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31565470 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85743925528576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81732832 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81756861155328 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4180108836864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83706981752694 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85744030004736 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32309018 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86050126740480 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4293130723840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81563833731072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13601697 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81563816691712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81742504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32308779 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50177032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50167362 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _runtime‚2142.740260373 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86628213431808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86937539887616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50684841 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86628317900800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18899997 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82585476 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50694497 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4223312782336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚381 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80357371813472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4212889373188 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82404988079104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13743453 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82599984974848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84570261186660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32643704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚4799 $ _timestamp‚1.785807864444584e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18899995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.cur¬b„™ôrent‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32643448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86937384309760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31890977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86628317900800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4223225352704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31890979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31890977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4337530009088 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50694497 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82575818 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82600009878528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84570364029144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82405005118464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4223312782336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13743707 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80357388852832 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86628213431808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86937539887616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50684841 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86628317900800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18899997 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82585476 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50694497 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4223312782336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚381 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚80357371813472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4212889373188 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82404988079104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13743453 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82599984974848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84570261186660 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32643704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚4799 $ _timestamp‚1.785807864444584e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18899995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32643448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86937384309760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31890977 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86628317900800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4223225352704 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31890979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31890977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4337530009088 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50694497 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82575818 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82600009878528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84570364029144 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82405005118464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4223312782336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13743707 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80357388852832 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 _runtime‚2142.740260373 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ý†ªÿ2¯ - _wandbruntime‚2143  _runtime‚2143è\Y\È&° ÂM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13746284 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50695364 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86646364290560 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4224104972288 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚4799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50705034 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86646364290560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80374111985792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31897620 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82422154842112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82602656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82602656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13746526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4338298204672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4224192409088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82422154842112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50695364 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86955652158464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32650460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82422171881472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32650216 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31897622 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18903934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18903932 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82617216995328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31897620 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4338435163136 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84587981674798 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4213852649646 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82422171881472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4213766846410 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86646259814400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4224104972288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86955490296320 _runtime‚2143.101146757 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31897622 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670×ú}†-!± ÇM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50705034 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86646364290560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80374111985792 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31897620 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82422154842112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82602656 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82602656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13746526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4338298204672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4224192409088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82422154842112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50695364 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86955652158464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32650460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82422171881472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32650216 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31897622 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18903934 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18903932 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82617216995328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚382 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31897620 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4338435163136 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84587981674798 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4213852649646 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82422171881472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4213766846410 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86646259814400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4224104972288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86955490296320 _runtime‚2143.101146757 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31897622 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13746284 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50695364 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86646364290560 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated}‡2- _bytes.small_pool.freed‚ 4224104972288 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 trainer/global_step‚4799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0þb®Y2² - _wandbruntime‚2143  _runtime‚2143Žþíp³ ý 2loss/SRqqt2/assignment_loss‚1.3409649901092053 /loss/FRt1/detection_loss‚2.2513207213746176 .loss/FBt1/detection_loss‚3.785260319709778 epoch‚0 _runtime‚2143.101978096 .loss/FRt2/assignment_loss‚3.57862507063767 1loss/FBt1/assignment_loss‚0.48588947205394106 .loss/FRt2/detection_loss‚2.581530385318844 trainer/global_step‚4799 2loss/SRqqt1/assignment_loss‚1.5301760208250872 2loss/FBt2/assignment_loss‚0.022383922510115895 1loss/SRqqt1/detection_loss‚1.1129611965490398 1loss/SRqqt2/detection_loss‚2.9583233445882797 /loss/FBt2/detection_loss‚1.6193857042725943 $ _timestamp‚1.785807864810493e+09 _step‚383 0loss/FRt1/assignment_loss‚3.6105569269922047 %loss/total_loss‚24.87737807494138¦¬´ ‚ 2loss/SRqqt1/assignment_loss‚1.5301760208250872 2loss/FBt2/assignment_loss‚0.022383922510115895 1loss/SRqqt1/detection_loss‚1.1129611965490398 1loss/SRqqt2/detection_loss‚2.9583233445882797 /loss/FBt2/detection_loss‚1.6193857042725943 $ _timestamp‚1.785807864810493e+09 _step‚383 0loss/FRt1/assignment_loss‚3.6105569269922047 %loss/total_loss‚24.87737807494138 2loss/SRqqt2/assignment_loss‚1.3409649901092053 /loss/FRt1/detection_loss‚2.2513207213746176 .loss/FBt1/detection_loss‚3.785260319709778 epoch‚0 _runtime‚2143.101978096 .loss/FRt2/assignment_loss‚3.57862507063767 1loss/FBt1/assignment_loss‚0.48588947205394106 .loss/FRt2/detection_loss‚2.581530385318844 trainer/global_step‚4799ÿC/w)›µ ‚(: ú‡ÅÓíðÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.011' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.67277777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YœyÜŠ¶ ‚(:ÿ ú‡ÅÓðî²Õ disk.rbd4.in‚0) proc.memory.availableMB‚ 46869.2265625 network.recv‚301919 disk.sda3.out‚ 105.921875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 29.57421875 disk.rbd5.in‚0 cpu‚13.932105355132734) disk./.usagePercent‚83.28747282633628 disk.rbd2.in‚0 disk.drbd1743.in‚0 disk.dm-0.out‚7579.380859375 disk.rbd4.out‚ 0.919921875! disk.nvme0c0n1.in‚ 29.57421875 network.sent‚ 4.488625e+06% disk./.usageGB‚1241.0435333251953 disk.dm-1.in‚0 disk.dm-1.out‚ 105.921875 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚7579.380859375 disk.rbd2.out‚ 1.068359375 disk.dm-0.in‚ 29.57421875# disk.nvme0n1.out‚7579.380859375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 105.921875% memory_percent‚28.483235836029053$ proc.memory.rssMB‚11267.62890625 disk.sda3.in‚0 disk.rbd1.out‚ 1.068359375 disk.rbd3.in‚0* proc.memory.percent‚17.193037271499634 proc.cpu.threads‚29 disk.drbd1743.out‚ 2.39453125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd5.out‚ 0.919921875¤”ùš· ‚(: ‰ˆÅÓ¨ÙßÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚99.185' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚55.10277777777778 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Zâ‘q‘¸ ‚(:†  ‰ˆÅÓ°¼ÛÖ% disk.nvme0c0n1.out‚7580.111328125 disk.rbd5.in‚0% memory_percent‚28.483325242996216$ proc.memory.rssMB‚11267.62890625 disk.rbd0.out‚0 disk.rbd4.out‚ 0.919921875 disk.rbd1.out‚ 1.068359375 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd5.out‚ 0.919921875 proc.cpu.threads‚29 disk.rbd2.out‚ 1.1005859375 disk.drbd1743.out‚ 2.42578125 disk.rbd0.in‚0 disk.dm-1.in‚0 cpu‚13.931377188712524% disk./.usageGB‚1241.0335121154785 disk.rbd4.in‚0! disk.nvme0c0n1.in‚ 29.57421875 disk.sda.in‚0 disk.sda.out‚ 106.57421875 disk.sda2.out‚0 disk.dm-0.out‚7580.111328125 disk.nvme0n1.in‚ 29.57421875) disk./.usagePercent‚83.28680029454384 disk.sda3.in‚0 disk.dm-0.in‚ 29.57421875# disk.nvme0n1.out‚7580.111328125* proc.memory.percent‚17.193037271499634 network.recv‚303887 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.sda1.in‚0 network.sent‚ 4.53345e+06 disk.sda3.out‚ 106.57421875 disk.dm-1.out‚ 106.57421875 disk.rbd3.in‚0* proc.memory.availableMB‚46869.16796875gó52¹ - _wandbruntime‚2166  _runtime‚2166¨ÜZɺ à ' lr-AdamW/pg1‚0.00012496933226724928 ' lr-AdamW/pg2‚0.00012496933226724928 trainer/global_step‚4849 % _timestamp‚1.7858078879160984e+09 _runtime‚2166.207863975 _step‚384ªùñλ È % _timestamp‚1.7858078879160984e+09 _runtime‚2166.207863975 _step‚384 ' lr-AdamW/pg1‚0.00012496933226724928 ' lr-AdamW/pg2‚0.00012496933226724928 trainer/global_step‚4849€ûjž 2¼ - _wandbruntime‚2166  _runtime‚2166îìý”¾'½ ¸O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81194380429472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32223129 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83263343268864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4267221488128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4382582206464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32223127 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83445628 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83435970 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4267308917760 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85451158266168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19096847 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51222499 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4256777836696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87842923021312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚385 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87530547717632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚4849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚87530652186624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81194397468832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83263326229504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85451261108652 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83263326229504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886735 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19096845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87842923021312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚385 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87530547717632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚4849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚87530652186624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81194397468832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83263326229504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85451261108652 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83263326229504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886735 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19096845 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32223127 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83445628 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856ê2¿ - _wandbruntime‚2166  _runtime‚2166z;ÂеÀ ÂM O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83453136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32990349 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81211120601792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚4849 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32229772 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4257741113154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257655309918 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83462808 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51223366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19100784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83477572835328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83280492992512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚386 % _timestamp‚1.7858078883200133e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51223366 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32229770 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51233036 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83280492992512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83453136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VCDeviceSL#˜øtatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85468878754306 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4268101107712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83280510031872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32990112 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4268101107712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4268188544512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83477547931648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87548594100224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4383492069888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0÷ u=Í&Á ÇM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4268101107712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83280510031872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32990112 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4268101107712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4268188544512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83477547931648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87548594100224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4383492069888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83453136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32990349 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81211120601792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚4849 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32229772 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4257741113154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257655309918 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83462808 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51223366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19100784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83477572835328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83280492992512 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚386 % _timestamp‚1.7858078883200133e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51223366 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32229770 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51233036 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83280492992512 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83453136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85468878754306 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032‚üð–é2 - _wandbruntime‚2166  _runtime‚2166<Ó ÿà ù 1loss/SRqqt2/detection_loss‚2.3570371846909874 .loss/FBt1/detection_loss‚1.941981132213886 epoch‚0 .loss/FRt2/assignment_loss‚3.46759992973371 1loss/SRqqt1/assignment_loss‚1.432074838797703 0loss/FBt1/assignment_loss‚0.6117209812439867 1loss/FBt2/assignment_loss‚0.07230894228552064 .loss/FRt2/detection_loss‚2.512787240134044 1loss/SRqqt1/detection_loss‚1.0689576685677686 trainer/global_step‚4849 % _timestamp‚1.7858078883217795e+09 0loss/FRt1/assignment_loss‚3.7247320564487314 2loss/SRqqt2/assignment_loss‚1.4756484687052391 .loss/FRt1/detection_loss‚2.429100564952138 &loss/total_loss‚22.654075062118455 _runtime‚2166.613398179 _step‚387 -loss/FBt2/detection_loss‚1.56012605434474ˆ±5¼Ä þ 1loss/SRqqt1/detection_loss‚1.0689576685677686 trainer/global_step‚4849 % _timestamp‚1.7858078883217795e+09 0loss/FRt1/assignment_loss‚3.7247320564487314 2loss/SRqqt2/assignment_loss‚1.4756484687052391 .loss/FRt1/detection_loss‚2.429100564952138 &loss/total_loss‚22.654075062118455 _runtime‚2166.613398179 _step‚387 -loss/FBt2/detection_loss‚1.56012605434474 1loss/SRqqt2/detection_loss‚2.3570371846909874 .loss/FBt1/detection_loss‚1.941981132213886 epoch‚0 .loss/FRt2/assignment_loss‚3.46759992973371 1loss/SRqqt1/assignment_loss‚1.432074838797703 0loss/FBt1/assignment_loss‚0.6117209812439867 1loss/FBt2/assignment_loss‚0.07230894228552064 .loss/FRt2/detection_loss‚2.512787240134044ƒçm›Å ‚(: ˜ˆÅÓ®îÒ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚146.06' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.14444444444445 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|5©mÆ ‚(:…  ˜ˆÅÓ©•½Ö disk.sda3.out‚ 107.109375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 29.58984375% memory_percent‚28.483378887176514$ disk./.usageGB‚1241.033058166504 disk.dm-1.out‚ 107.109375 network.sent‚ 4.535936e+06 disk.rbd4.in‚0 disk.rbd2.out‚ 1.1005859375" disk.dm-0.out‚7642.87939453125 disk.drbd1743.out‚ 2.4453125 disk.rbd5.in‚0! disk.nvme0c0n1.in‚ 29.58984375 disk.sda.in‚0 disk.sda.out‚ 107.109375 disk.rbd1.out‚ 1.1005859375 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd4.out‚ 0.919921875 network.recv‚304767) disk./.usagePercent‚83.28676982964727 disk.rbd0.in‚0 proc.cpu.threads‚29 disk.sda3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 46869.1328125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0$ proc.memory.rssMB‚11267.62890625* proc.memory.percent‚17.193037271499634 cpu‚13.930403579340645 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 29.58984375% disk.nvme0n1.out‚7642.87939453125 disk.drbd1743.in‚0' disk.nvme0c0n1.out‚7642.87939453125 disk.rbd5.out‚ 1.005859375î%Q—2Ç - _wandbruntime‚2189  _runtime‚2189ömL›ÈÈ Â trainer/global_step‚4899 % _timestamp‚1.7858079109623291e+09 _runtime‚ 2189.25486463 _step‚388 ' lr-AdamW/pg1‚0.00012625794159151456 ' lr-AdamW/pg2‚0.00012625794159151456ò›­;ÍÉ Ç % _timestamp‚1.7858079109623291e+09 _runtime‚ 2189.25486463 _step‚388 ' lr-AdamW/pg1‚0.00012625794159151456 ' lr-AdamW/pg2‚0.00012625794159151456 trainer/global_step‚4899„lÑé2Ê - _wandbruntime‚2189  _runtime‚2189³Rݾ'Ë ¸O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32555277 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88432882003456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4311305053184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14030328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84121681419264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88432882003456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4427807605760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82031406084832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51750501 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88748504260608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4427940377088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19293695 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84320696654848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84296122 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84320721558528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84121664379904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4300666300204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4311217623552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84121681419264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33323765 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33324025 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88432986472448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 _runtime‚2189.255105763 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84320696654848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84296122 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84320721558528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 [HDeviceStatsMonitor.on_train_bat" Ì6ch_start/allocated_bytes.large_pool.freed‚84121664379904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4300666300204 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4311217623552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84121681419264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33323765 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33324025 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88432986472448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 _runtime‚2189.255105763 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88432882003456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4311305053184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14030328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84121681419264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88432882003456…—wó2Í - _wandbruntime‚2189  _runtime‚2189DÈÄÈ&Î ÂM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858079113329268e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32561922 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4312097243136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84313288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82048146257152 _runtime‚2189.625029903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84138831142912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 trainer/global_step‚4899 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88451032862208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84138848182272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4312097243136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4428710391296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84138831142912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14033191 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4301629576662 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88766771830784 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19297634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14032940 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32561920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33330572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84322960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51761038 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84138848182272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84337903771648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82048129217792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84322960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32561922 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19297632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0EئÍ&Ï ÇM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4312097243136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84313288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82048146257152 _runtime‚2189.625029903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84138831142912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 trainer/global_step‚4899 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88451032862208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84138848182272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4312097243136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4428710391296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84138831142912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14033191 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4301629576662 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88766771830784 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19297634 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14032940 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32561920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33330572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84322960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51761038 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84138848182272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84337903771648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82048129217792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84322960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32561922 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19297632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858079113329268e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32561922 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0†h8„2Ð - _wandbruntime‚2189  _runtime‚21894±˜üÑ ý /loss/FRt2/assignment_loss‚3.284724176599738 0loss/FBt1/assignment_loss‚0.8615987962457439 1loss/SRqqt2/detection_loss‚2.4656122887264127 .loss/FBt2/detection_loss‚1.201156916256462 epoch‚0 2loss/SRqqt2/assignment_loss‚1.1099784757294084 1loss/FBt2/assignment_loss‚0.06345276588815198 /loss/FRt1/assignment_loss‚3.678191852875245 /loss/FRt1/detection_loss‚2.4981030537149844 /loss/FRt2/detection_loss‚2.5002160638235926 /loss/FBt1/detection_loss‚1.7863445083337348 trainer/global_step‚4899 _runtime‚2189.625714217 2loss/SRqqt1/assignment_loss‚1.3685724589007873 0loss/SRqqt1/detection_loss‚1.003144250928157 &loss/total_loss‚21.821095608022418 % _timestamp‚1.7858079113342211e+09 _step‚391–Ù“šÒ ‚ /loss/FRt2/assignment_loss‚3.284724176599738 0loss/FBt1/assignment_loss‚0.8615987962457439 1loss/SRqqt2/detection_loss‚2.4656122887264127 .loss/FBt2/detection_loss‚1.201156916256462 epoch‚0 2loss/SRqqt2/assignment_loss‚1.1099784757294084 1loss/FBt2/assignment_loss‚0.06345276588815198 /loss/FRt1/assignment_loss‚3.678191852875245 /loss/FRt1/detection_loss‚2.4981030537149844 /loss/FRt2/detection_loss‚2.5002160638235926 /loss/FBt1/detection_loss‚1.7863445083337348 trainer/global_step‚4899 _runtime‚2189.625714217 2loss/SRqqt1/assignment_loss‚1.3685724589007873 0loss/SRqqt1/detection_loss‚1.003144250928157 &loss/total_loss‚21.821095608022418 % _timestamp‚1.7858079113342211e+09 _step‚391‡j­§îœÓ ‚(:‘ §ˆÅÓ—ˆŽÓ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.70055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—EŸ;‘Ô ‚(:†  §ˆÅÓ¢ˆõÕ! disk.nvme0c0n1.in‚ 29.58984375* proc.memory.availableMB‚46869.07421875 disk.dm-1.out‚ 108.15625 disk.rbd1.out‚ 1.1005859375 disk.sda2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚83.28677136569247% disk./.usageGB‚1241.0330810546875 disk.drbd1743.out‚ 2.47265625 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda.in‚0* proc.memory.percent‚17.193037271499634 cpu‚13.930375107731255" disk.dm-0.out‚7643.14501953125 disk.nvme0n1.in‚ 29.58984375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚7643.14501953125 disk.rbd2.out‚ 1.16357421875$ proc.memory.rssMB‚11267.62890625 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0% memory_percent‚28.483468294143677 network.sent‚ 4.580793e+06 network.recv‚306829 disk.drbd1743.in‚0% disk.nvme0n1.out‚7643.14501953125 disk.sda.out‚ 108.15625 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 108.15625 disk.sda1.in‚0 disk.rbd5.out‚ 1.068359375 disk.sda2.out‚0 disk.dm-0.in‚ 29.58984375 disk.rbd5.in‚0 disk.rbd4.out‚ 0.919921875Bz¿ ›Õ ‚(: ¶ˆÅÓ°üÞÒ gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚69.2088888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D½˜Ö ‚(:  ¶ˆÅÓ²ù‰Ö% disk.nvme0n1.out‚7644.46630859375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda.out‚ 108.67578125 network.sent‚ 4.625559e+06% disk./.usageGB‚1241.0331726074219 disk.dm-0.in‚ 29.58984375 disk.sda1.out‚0 disk.rbd5.out‚ 1.068359375 cpu‚13.92990420778673" disk.dm-0.out‚7644.46630859375 disk.rbd0.out‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.16357421875 disk.dm-1.out‚ 108.67578125% memory_percent‚28.450322151184082 disk.drbd1743.in‚0 disk.rbd5.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 46890.796875 network.recv‚308729 disk.sda1.in‚0 disk.rbd4.out‚ 1.005859375! disk.nvme0c0n1.in‚ 29.58984375 disk.sda3.in‚0 disk.sda3.out‚ 108.67578125 disk.sda2.out‚0) disk./.usagePercent‚83.28677750987329 disk.rbd4.in‚0' disk.nvme0c0n1.out‚7644.46630859375 disk.rbd1.out‚ 1.16357421875 disk.sda2.in‚0$ proc.memory.rssMB‚11267.62890625* proc.memory.percent‚17.193037271499634 proc.cpu.threads‚29 disk.drbd1743.out‚ 2.47265625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 29.58984375Û™šú2× - _wandbruntime‚2212  _runtime‚2212cìøÏÉØ Ã ' lr-AdamW/pg1‚0.00012754655091577988 ' lr-AdamW/pg2‚0.00012754655091577988 trainer/global_step‚4949 % _timestamp‚1.7858079339245532e+09 _runtime‚2212.216308097 _step‚392·›—ÎÙ È _runtime‚2212.216308097 _step‚392 ' lr-AdamW/pg1‚0.00012754655091577988 ' lr-AdamW/pg2‚0.00012754655091577988 trainer/global_step‚4949 % _timestamp‚1.7858079339245532e+09ˆYmÖ„2Ú - _wandbruntime‚2212  _runtime‚2212Nýò“Û ¸O KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4355213758976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚393 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19490547 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89335320758272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85181077398528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82868414700832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚P Òú¢  2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32887429 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85181052494848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89335216289280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4355213758976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858079339259655e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84980019569664 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87213055267668 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32887427 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52278503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4472954139136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89335320758272 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89654168503296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85165932 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4473091104768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355301188608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14172770 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19490545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89654006633984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82868397661472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33663570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14173023 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87212952425184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89335216289280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4949*¤â•Ã'Ü ½O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85165932 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4473091104768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4355301188608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14172770 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19490545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89654006633984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82868397661472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33663570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14173023 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87212952425184 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89335216289280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4949 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4355213758976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚393 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19490547 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89335320758272 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85181077398528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82868414700832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32887429 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85181052494848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89335216289280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4355213758976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858079339259655e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84980019569664 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87213055267668 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32887427 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52278503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4472954139136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89335320758272 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89654168503296‰¢¬¼ž2Ý - _wandbruntime‚2212  _runtime‚2212vÍwúÆ&Þ ÀM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84997169293312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356093378560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19494484 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89672113792000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚394 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 $ _timestamp‚1.785807934451102e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85183112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19494482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32894072 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52289040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4356093378560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 2212.74359816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32894072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87230672913322 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89353367148032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4356180815360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85173440 trainer/global_step‚4949 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87230570070726 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84997186332672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85198259611648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89353262671872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85183112 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89672277751296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356180815360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚89353367148032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85173440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33670095 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82885137833792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4345432236934Ù%5€ß ÅM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89353262671872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85183112 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89672277751296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356180815360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚89353367148032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85173440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33670095 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82885137833792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4345432236934 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84997169293312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356093378560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19494484 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89672113792000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚394 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 $ _timestamp‚1.785807934451102e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85183112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19494482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_ÐŽ‹Ksegments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32894072 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52289040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4356093378560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 2212.74359816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32894072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87230672913322 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89353367148032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4356180815360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85173440 trainer/global_step‚4949 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87230570070726 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84997186332672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85198259611648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Šæ:A’2à - _wandbruntime‚2212  _runtime‚2212Í_?üá ý /loss/FRt2/assignment_loss‚3.421374572948976 1loss/SRqqt1/detection_loss‚1.2800144626173389 .loss/FBt2/detection_loss‚1.771571830785098 &loss/total_loss‚22.384676273747637 epoch‚0 _runtime‚2212.745295475 /loss/FRt1/assignment_loss‚3.333918969572326 1loss/SRqqt1/assignment_loss‚1.444511036642573 2loss/FBt2/assignment_loss‚0.056878879400546555 /loss/FRt2/detection_loss‚2.3763819242065605 1loss/SRqqt2/detection_loss‚2.4085992174843947 % _timestamp‚1.7858079344536989e+09 _step‚395 /loss/FRt1/detection_loss‚2.4083180407315123 trainer/global_step‚4949 1loss/SRqqt2/assignment_loss‚1.676616653179129 0loss/FBt1/assignment_loss‚0.3536258138536306 /loss/FBt1/detection_loss‚1.8528648723255505]7Aâ ‚ .loss/FBt2/detection_loss‚1.771571830785098 &loss/total_loss‚22.384676273747637 epoch‚0 _runtime‚2212.745295475 /loss/FRt1/assignment_loss‚3.333918969572326 1loss/SRqqt1/assignment_loss‚1.444511036642573 2loss/FBt2/assignment_loss‚0.056878879400546555 /loss/FRt2/detection_loss‚2.3763819242065605 1loss/SRqqt2/detection_loss‚2.4085992174843947 % _timestamp‚1.7858079344536989e+09 _step‚395 /loss/FRt1/detection_loss‚2.4083180407315123 trainer/global_step‚4949 1loss/SRqqt2/assignment_loss‚1.676616653179129 0loss/FBt1/assignment_loss‚0.3536258138536306 /loss/FBt1/detection_loss‚1.8528648723255505 /loss/FRt2/assignment_loss‚3.421374572948976 1loss/SRqqt1/detection_loss‚1.2800144626173389‹U\H¢œã ‚(:‘ ňÅÓ¿ìÚÒ gpu.0.gpu‚13.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚136.603' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚75.89055555555557 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„[jÖ‘ä ‚(:†  ňÅÓ‡¹¹Ö disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚29% disk./.usageGB‚1241.0331382751465% disk.nvme0n1.out‚7707.11865234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 cpu‚13.929220875142514 network.recv‚310025 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7707.11865234375 disk.rbd2.out‚ 1.2265625* proc.memory.percent‚17.193037271499634 disk.drbd1743.out‚ 2.50390625 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.dm-1.out‚ 109.44140625 disk.rbd1.out‚ 1.16357421875% memory_percent‚28.450405597686768 network.sent‚ 4.628053e+06 disk.sda2.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 109.44140625) proc.memory.availableMB‚ 46890.7421875$ proc.memory.rssMB‚11267.62890625" disk.dm-0.out‚7707.11865234375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 1.005859375) disk./.usagePercent‚83.28677520580548 disk.sda3.out‚ 109.44140625 disk.drbd1743.in‚0 disk.dm-0.in‚ 31.078125 disk.rbd5.in‚0 disk.nvme0c0n1.in‚ 31.078125 disk.sda2.in‚0 disk.nvme0n1.in‚ 31.078125 disk.rbd5.out‚ 1.1005859375 s±¯œå ‚(:‘ ÔˆÅÓö™àÒ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚151.374' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.09666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0··µÝ€æ ‚(:õ ÔˆÅÓ„åÎÕ disk.nvme0n1.in‚ 31.078125 disk.rbd0.out‚0 disk.rbd4.out‚ 1.068359375 disk.sda.in‚0% disk./.usageGB‚1241.0331420898438 disk.dm-1.out‚ 110.17578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd4.in‚0 network.recv‚311964) disk./.usagePercent‚83.28677546181301 disk.dm-0.out‚ 7707.3828125 disk.rbd0.in‚0 disk.sda3.out‚ 110.17578125 network.sent‚ 4.672889e+06 disk.sda2.in‚0 disk.rbd5.in‚0$ memory_percent‚28.45078706741333 disk.drbd1743.in‚0 disk.drbd1743.out‚ 2.50390625 disk.rbd5.out‚ 1.1005859375* proc.memory.percent‚17.193037271499634 proc.cpu.threads‚29 disk.rbd1.out‚ 1.2265625 disk.sda.out‚ 110.17578125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.2265625 disk.sda3.in‚0) proc.memory.availableMB‚ 46890.4921875 cpu‚13.928668215940366 disk.dm-1.in‚0 disk.dm-0.in‚ 31.078125! disk.nvme0n1.out‚ 7707.3828125 disk.nvme0c0n1.in‚ 31.078125# disk.nvme0c0n1.out‚ 7707.3828125$ proc.memory.rssMB‚11267.62890625 disk.rbd3.out‚0Ò«ù!2ç - _wandbruntime‚2235  _runtime‚2235œ´¸™Çè Á & lr-AdamW/pg1‚0.0001288351602400452 & lr-AdamW/pg2‚0.0001288351602400452 trainer/global_step‚4999 % _timestamp‚1.7858079575514657e+09 _runtime‚2235.843170731 _step‚396Nð©Ìé Æ & lr-AdamW/pg1‚0.0001288351602400452 & lr-AdamW/pg2‚0.0001288351602400452 trainer/global_step‚4999 % _timestamp‚1.7858079575514657e+09 _runtime‚2235.843170731 _step‚396ŒP_µ_2ê - _wandbruntime‚2235  _runtime‚2235ãqÞ¹¾'ë ¸O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14315915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52796849 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4388443227220 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚4999 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85838340680704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83705423316832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88093849504692 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚2235.848015049 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90237655044096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14315680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52796849 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52806505 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33219579 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4518122947072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90237550575104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85838357720064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33219579 _step‚397 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19687397 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4399297324032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86026084 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33219577 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86041433238528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19687395 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4388529030344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4399209894400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4518253621248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52806505 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86016426 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34003075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85838357720064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4399297324032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33219577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86041408334848 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90237550575104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88093849504692 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚2235.848015049 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90237655044096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14315680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52796849 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52806505 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33219579 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4518122947072 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90237550575104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85838357720064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33219579 _step‚397 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19687397 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4399297324032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86026084 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33219577 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86041433238528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19687395 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4388529030344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4399209894400 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4518253621248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52806505 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86016426 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34003075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85838357720064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4399297324032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Û‚s2í - _wandbruntime‚2236  _runtime‚2236ÔµÊî ÂM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚90255701433856 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88111467150234 D>Devicëõ°Ý³!eStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86043264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34010085 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33226220 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14318751 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88111569992830 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4389406503678 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85855524483072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4400089513984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 % _timestamp‚1.7858079579151642e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33226222 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4400176950784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19691332 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86058615451648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4400176950784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4389320700442 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83722163489152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52807372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85855507443712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14318501 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90577798712832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85855524483072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚4999 _step‚398 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83722146449792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33226220 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90255596957696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4400089513984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90577634753536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52817042 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52807372 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34009833 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19691334 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2236.207122684[ /*Í&ï ÇM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85855524483072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚4999 _step‚398 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83722146449792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33226220 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚90255596957696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4400089513984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90577634753536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52817042 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52807372 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34009833 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19691334 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚2236.207122684 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚90255701433856 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88111467150234 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86043264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34010085 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33226220 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14318751 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88111569992830 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4389406503678 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85855524483072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4400089513984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 % _timestamp‚1.7858079579151642e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33226222 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4400176950784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19691332 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86058615451648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4400176950784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4389320700442 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83722163489152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52807372 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚85855507443712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14318501 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90577798712832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596Ž$ªŒ•2ð - _wandbruntime‚2236  _runtime‚2236'_”±ñ ü /loss/FRt2/detection_loss‚2.5293922784887712 0loss/FBt1/assignment_loss‚0.4509554861315842 1loss/SRqqt1/detection_loss‚1.0927127094872024 .loss/FBt2/detection_loss‚4.449933346360922 $loss/total_loss‚25.0327434531864 _step‚399 0loss/FRt1/assignment_loss‚3.9021419278089553 2loss/SRqqt1/assignment_loss‚1.4159980227768019 2loss/FBt2/assignment_loss‚0.023728015163903313 /loss/FRt1/detection_loss‚2.6957008713397426 _runtime‚2236.207988889 .loss/FRt2/assignment_loss‚3.09501709614271 1loss/SRqqt2/detection_loss‚2.1459698397379654 /loss/FBt1/detection_loss‚1.7949216100904677 epoch‚0 trainer/global_step‚4999 % _timestamp‚1.7858079579164855e+09 1loss/SRqqt2/assignment_loss‚1.436272249657374¶û”³ò  1loss/SRqqt2/assignment_loss‚1.436272249657374 /loss/FRt2/detection_loss‚2.5293922784887712 0loss/FBt1/assignment_loss‚0.4509554861315842 1loss/SRqqt1/detection_loss‚1.0927127094872024 .loss/FBt2/detection_loss‚4.449933346360922 $loss/total_loss‚25.0327434531864 _step‚399 0loss/FRt1/assignment_loss‚3.9021419278089553 2loss/SRqqt1/assignment_loss‚1.4159980227768019 2loss/FBt2/assignment_loss‚0.023728015163903313 /loss/FRt1/detection_loss‚2.6957008713397426 _runtime‚2236.207988889 .loss/FRt2/assignment_loss‚3.09501709614271 1loss/SRqqt2/detection_loss‚2.1459698397379654 /loss/FBt1/detection_loss‚1.7949216100904677 epoch‚0 trainer/global_step‚4999 % _timestamp‚1.7858079579164855e+09Mgdœó ‚(:‘ ãˆÅÓð‡‹Ó gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚150.016' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.34222222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x̉~ô ‚(:ó ãˆÅÓ…Å¢×% disk./.usageGB‚1241.1586799621582 disk.rbd2.out‚ 1.2265625 disk.sda3.in‚0 disk.dm-0.out‚ 7707.640625 disk.rbd4.out‚ 1.068359375 disk.nvme0c0n1.in‚ 31.078125$ proc.memory.rssMB‚11267.64453125 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0 disk.drbd1743.out‚ 2.5390625 disk.rbd3.out‚0* proc.memory.percent‚17.193061113357544 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.out‚ 7707.640625 disk.sda1.out‚0 disk.sda.out‚ 110.97265625% memory_percent‚28.450477123260498) proc.memory.availableMB‚ 46890.6953125 disk.sda2.out‚0 network.recv‚313211 disk.dm-1.in‚0 disk.rbd4.in‚0) disk./.usagePercent‚83.29520041375733 disk.rbd3.in‚0 disk.nvme0n1.in‚ 31.078125 disk.rbd5.in‚0 disk.rbd5.out‚ 1.16357421875 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 7707.640625 proc.cpu.threads‚29 network.sent‚ 4.675318e+06 disk.sda3.out‚ 110.97265625 disk.dm-0.in‚ 31.078125 disk.rbd0.out‚0 cpu‚13.928198565664614 disk.dm-1.out‚ 110.97265625 disk.sda.in‚0˜÷2õ - _wandbruntime‚2258  _runtime‚2258À¶òÇö Á & lr-AdamW/pg1‚0.0001301237695643105 & lr-AdamW/pg2‚0.0001301237695643105 trainer/global_step‚5049 % _timestamp‚1.7858079802771225e+09 _runtime‚2258.568922148 _step‚400 AÌ÷ Æ & lr-AdamW/pg2‚0.0001301237695643105 trainer/global_step‚5049 % _timestamp‚1.7858079802771225e+09 _runtime‚2258.568922148 _step‚400 & lr-AdamW/pg1‚0.0001301237695643105Lj2ø - _wandbruntime‚2258  _runtime‚2258Pî]½'ù ·O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86901764174848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚5049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86696678831104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34343380 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33551727 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53324851 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4432331690728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33551727 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19884247 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.785807980278315e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19884245 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91139989329920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14459135 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4563296753664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91465060928512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86886236 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33551729 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84542414893472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4432417493852 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53334507 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91139884860928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86696695870464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2•‹b€ú ¼O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19884247 ?9DeviceStatsMonitor.on_train_batch_start˜—^‹¦&/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.785807980278315e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19884245 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91139989329920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14459135 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4563296753664 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91465060928512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86886236 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33551729 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84542414893472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4432417493852 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53334507 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91139884860928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86696695870464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86901764174848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚5049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86696678831104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34343380 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33551727 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53324851 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4432331690728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33551727 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0‘ntÕ 2û - _wandbruntime‚2258  _runtime‚2258¬îxÇ&ü ÁM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86893744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84559155065792 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4433209163950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33558370 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53335374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14462248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86893744 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚2258.974553486 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34350432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91483175303680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86903416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91158035719680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4564332581888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4444085649408 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91483328777216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34350195 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4444173086208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86713862633472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84559172105152 $ _timestamp‚1.785807980682209e+09 _step‚402 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86903416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53345044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86713862633472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88992364229742 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53345044 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0d(ƒðÌ&ý ÆM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19888184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91157931243520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86918996195328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34350432 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91483175303680 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86903416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91158035719680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4564332581888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4444085649408 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91483328777216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34350195 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4444173086208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86713862633472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84559172105152 $ _timestamp‚1.785807980682209e+09 _step‚402 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86903416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53345044 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86713862633472 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88992364229742 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53345044 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86893744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84559155065792 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4433209163950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33558370 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53335374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14462248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86893744 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚2258.974553486 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87555034020864 _runtime‚2282.657979147 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92370805568000 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4608521586688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785808004362231e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33883877 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34683546 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4608660649472 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4476220154236 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487289594880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20081095 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87555016981504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚5099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33883879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14602451 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92042219146752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33883877 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53852853 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85379423509472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487202165248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53862509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4476305957360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487289594880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92370641601536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87762144918528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89855746506192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87746388 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92042219146752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33883877 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53852853 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85379423509472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487202165248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53862509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4476305957360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487289594880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92370641601536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87762144918528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89855746506192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87746388 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87555034020864 _runtime‚2282.657979147 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92370805568000 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4608521586688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 $ _timestamp‚1.785808004362231e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33883877 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34683546 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4608660649472 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4476220154236 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487289594880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20081095 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87555016981504 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 trainer/global_step‚5099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33883879 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14602451•ê£2‹- _wandbruntime‚2283  _runtime‚2283êe€Ç&ŒÁM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92060370005504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488169221632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20085034 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33890522 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4477183430694 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87763568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87779352035328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4609562125312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89873364151846 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34690313 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4609418875392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33890520 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87572183744512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2283.019286291 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87753896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53863376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85396163681792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53863376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87572200783872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488169221632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92060265529344 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488081784832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14605281 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33890520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92388914160640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85396180721152 trainer/global_step‚5099 $ _timestamp‚1.785808004727272e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89873261309250 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92060370005504v™ÅÆM >8DeviceStatsMonitor.onÚ}¿­&_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488169221632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92060265529344 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488081784832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14605281 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33890520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92388914160640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85396180721152 trainer/global_step‚5099 $ _timestamp‚1.785808004727272e+09 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89873261309250 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92060370005504 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92060370005504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488169221632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20085034 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33890522 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4477183430694 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87763568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87779352035328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4609562125312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89873364151846 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34690313 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4609418875392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168153600 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33890520 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87572183744512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2283.019286291 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87753896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53863376 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85396163681792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53863376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚87572200783872 E#œ‘‚(:‘ ‰ÅÓèú×Ò gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚62.00666666666667 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Ã>n’‚(:ã ‰ÅÓ«ãÂÖ disk.sda.in‚0) disk./.usagePercent‚83.29516713277788% disk./.usageGB‚1241.1581840515137 disk.rbd5.out‚ 1.2265625 disk.nvme0c0n1.in‚ 31.078125 disk.nvme0c0n1.out‚ 7770.9375 disk.drbd1743.out‚ 2.58984375 disk.dm-0.out‚ 7770.9375 cpu‚13.92637234626889 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.out‚0 disk.dm-0.in‚ 31.078125* proc.memory.availableMB‚46890.34765625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 31.078125 disk.sda1.in‚0 disk.rbd2.out‚ 1.2265625 disk.dm-1.in‚0 network.sent‚ 4.767337e+06 network.recv‚318376 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 1.16357421875 disk.sda.out‚ 112.765625 disk.sda3.out‚ 112.765625 disk.rbd2.in‚0* proc.memory.percent‚17.193061113357544 proc.cpu.threads‚29 disk.sda1.out‚0 disk.dm-1.out‚ 112.765625 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.out‚ 7770.9375 disk.sda3.in‚0" memory_percent‚28.451007604599$ proc.memory.rssMB‚11267.64453125îÜ 2“- _wandbruntime‚2305  _runtime‚2305SÿÇÇ”Á & lr-AdamW/pg2‚0.0001327009882128411 trainer/global_step‚5149 % _timestamp‚1.7858080274845052e+09 _runtime‚2305.776214461 _step‚408 & lr-AdamW/pg1‚0.0001327009882128411?Í <Ì•Æ _runtime‚2305.776214461 _step‚408 & lr-AdamW/pg1‚0.0001327009882128411 & lr-AdamW/pg2‚0.0001327009882128411 trainer/global_step‚5149 % _timestamp‚1.7858080274845052e+09˜ƒwwÓ2–- _wandbruntime‚2305  _runtime‚2305ÝÉäü¾'—¸O JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚5149 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92944553432576 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88596882 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54390511 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93276187832832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88413355131904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4531198300672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚2305.780993737 _step‚409 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88413372171264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653711977984 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92944657901568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35023524 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90736540743216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90736643585700 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4531285730304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88606540 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35023277 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88413355131904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4531198300672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20277945 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86216432125472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4520108617744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34216027 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54390511 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92944553432576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200€é/€#˜½O F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20277947 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93276343410688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 % _timestamp‚1.7858080274857063e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34216029 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88622475854848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4520194420868 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88413372171264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34216029 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚88413355131904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4531198300672 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚2305.780993737 _step‚409 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88413372171264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653711977984 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92944657901568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35023524 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90736540743216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90736643585700 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4531285730304 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88606540 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35023277 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88413355131904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4531198300672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20277945 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86216432125472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4520108617744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34216027 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54390511 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92944553432576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚5149 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92944553432576 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88596882 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54390511 YFDeviceStatsMonitor.on_train_bê´ •¼atch_start/inactive_split_bytes.all.freed‚93276187832832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35030048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88430521894912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34222670 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4532077920256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88639682971648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86233189337152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88430521894912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 % _timestamp‚1.7858080278613455e+09 _step‚410 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4532165357056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35030286 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54391378 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34222672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34222672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654745659904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86233172297792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88614048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90754158388758 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20281882 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93294302158848 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚5149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54401048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654619187200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88614048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93294453535232 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14748166 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90754261231354 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88623720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92962704291328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936}ƒ–Í&›ÇM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88614048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90754158388758 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20281882 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93294302158848 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚5149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54401048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654619187200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88614048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93294453535232 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14748166 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90754261231354 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88623720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92962704291328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4521071894202 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚2306.153420232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14748402 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92962704291328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35030048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88430521894912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34222670 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4532077920256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88639682971648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86233189337152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88430521894912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 % _timestamp‚1.7858080278613455e+09 _step‚410 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4532165357056 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35030286 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54391378 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34222672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34222672 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654745659904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86233172297792šŠ^ñ2œ- _wandbruntime‚2306  _runtime‚2306ï3û .loss/FBt1/detection_loss‚2.441759393635776 /loss/FBt2/detection_loss‚0.9611870393014963 $ _timestamp‚1.785808027863134e+09 epoch‚0 1loss/SRqqt1/assignment_loss‚1.455213121044031 0loss/FBt1/assignment_loss‚0.4270353206419589 2loss/FBt2/assignment_loss‚0.009426481110547617 .loss/FRt1/detection_loss‚2.474686749102104 -loss/FRt2/detection_loss‚2.69867369118664 trainer/global_step‚5149 _runtime‚2306.154666761 _step‚411 &loss/total_loss‚21.741492819823595 0loss/FRt1/assignment_loss‚3.4216458854221163 2loss/SRqqt2/assignment_loss‚1.3451787518958251 1loss/SRqqt2/detection_loss‚2.0856150945550516 /loss/FRt2/assignment_loss‚3.390131723549631 1loss/SRqqt1/detection_loss‚1.0309395683784188¼z`ក /loss/FRt2/assignment_loss‚3.390131723549631 1loss/SRqqt1/detection_loss‚1.0309395683784188 .loss/FBt1/detection_loss‚2.441759393635776 /loss/FBt2/detection_loss‚0.9611870393014963 $ _timestamp‚1.785808027863134e+09 epoch‚0 1loss/SRqqt1/assignment_loss‚1.455213121044031 0loss/FBt1/assignment_loss‚0.4270353206419589 2loss/FBt2/assignment_loss‚0.009426481110547617 .loss/FRt1/detection_loss‚2.474686749102104 -loss/FRt2/detection_loss‚2.69867369118664 trainer/global_step‚5149 _runtime‚2306.154666761 _step‚411 &loss/total_loss‚21.741492819823595 0loss/FRt1/assignment_loss‚3.4216458854221163 2loss/SRqqt2/assignment_loss‚1.3451787518958251 1loss/SRqqt2/detection_loss‚2.0856150945550516›âædžšŸ‚(: Ÿ‰ÅÓÑÉÐÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚99.6' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚55.333333333333336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bÒîDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93846887718400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89271710321664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14889730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93846992187392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87053457780832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89482856598528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54908857 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4563997081252 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89457034 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89271710321664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89466692 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚5199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4698949891584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4564082884376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89271693282304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4575194436096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4699091051520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93846992187392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35364527 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93846887718400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34548179 % _timestamp‚1.7858080500578845e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.cÖ¤sÉurrent‚87429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91617540665208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4575281865728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89271693282304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20474795 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94181947650048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34548179 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14889473 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 _step‚413 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93846887718400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89271710321664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14889730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93846992187392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87053457780832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89482856598528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54908857 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4563997081252 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89457034 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89271710321664 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89466692 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚5199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4698949891584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4564082884376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89271693282304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4575194436096 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4699091051520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93846992187392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35364527 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93846887718400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34548179 % _timestamp‚1.7858080500578845e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91617540665208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4575281865728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89271693282304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20474795 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94181947650048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0O Y2©- _wandbruntime‚2328  _runtime‚2328è¾n3È&ªÂM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94200048615936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35371132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89288860045312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91635055468266 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858080504199512e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54929050 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94199893045248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14892653 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4699984900608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34554820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54919380 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35371387 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34554822 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576161492480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89288860045312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89483872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20478732 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14892400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚2328.712060351 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4564960357710 trainer/global_step‚5199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20478734 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89500038811648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87070180913792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34554820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91635158310862 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89288877084672 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89288860045312 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89483872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20478732 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14892400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 _runtime‚2328.712060351 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4564960357710 trainer/global_step‚5199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20478734 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89500038811648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87070180913792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34554820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91635158310862 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89288877084672 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35371132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89288860045312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91635055468266 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858080504199512e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54929050 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ODeviceStatsMonitor.on_train_batch_end/segmentäTƒy¿.small_pool.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚94199893045248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14892653 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4699984900608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34554820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54919380 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35371387 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34554822 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576161492480žÓ9òÛ2¬- _wandbruntime‚2328  _runtime‚2328³öô¤­þ 2loss/SRqqt1/assignment_loss‚1.2965160104305409 2loss/SRqqt2/assignment_loss‚1.4006572040142837 1loss/FBt2/assignment_loss‚0.13143466191833772 &loss/total_loss‚31.998252611079565 epoch‚0 _step‚415 0loss/FBt1/assignment_loss‚0.5403742752969265 /loss/FRt1/detection_loss‚2.3690259797232494 /loss/FRt1/assignment_loss‚3.514277826462473 /loss/FRt2/detection_loss‚2.6673643733892174 0loss/SRqqt2/detection_loss‚2.271911419928074 /loss/FBt1/detection_loss‚12.228699994087219 /loss/FBt2/detection_loss‚1.2919834700794461 trainer/global_step‚5199 % _timestamp‚1.7858080504213376e+09 _runtime‚2328.712825467 /loss/FRt2/assignment_loss‚3.303875371813774 1loss/SRqqt1/detection_loss‚0.98213202393602673µ“" ®ƒ /loss/FRt1/assignment_loss‚3.514277826462473 /loss/FRt2/detection_loss‚2.6673643733892174 0loss/SRqqt2/detection_loss‚2.271911419928074 /loss/FBt1/detection_loss‚12.228699994087219 /loss/FBt2/detection_loss‚1.2919834700794461 trainer/global_step‚5199 % _timestamp‚1.7858080504213376e+09 _runtime‚2328.712825467 /loss/FRt2/assignment_loss‚3.303875371813774 1loss/SRqqt1/detection_loss‚0.9821320239360267 2loss/SRqqt1/assignment_loss‚1.2965160104305409 2loss/SRqqt2/assignment_loss‚1.4006572040142837 1loss/FBt2/assignment_loss‚0.13143466191833772 &loss/total_loss‚31.998252611079565 epoch‚0 _step‚415 0loss/FBt1/assignment_loss‚0.5403742752969265 /loss/FRt1/detection_loss‚2.3690259797232494ŸN½Êœ¯‚(:‘ ½‰ÅÓÕÌüÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚113.129' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚62.849444444444444 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)”G}°‚(:ò ½‰ÅÓì´ëÖ disk.sda2.out‚0 disk.sda.out‚114.875% memory_percent‚28.093039989471436 cpu‚13.925523761925144 network.sent‚ 4.859204e+06" disk.dm-0.out‚7834.72607421875 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚17.193061113357544 network.recv‚323306 disk.rbd1.in‚0 proc.cpu.threads‚29 disk.rbd5.out‚ 1.2265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd1743.in‚0% disk.nvme0n1.out‚7834.72607421875 disk.sda1.in‚0' disk.nvme0c0n1.out‚7834.72607421875 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625$ proc.memory.rssMB‚11267.64453125% disk./.usageGB‚1241.1589012145996 disk.dm-0.in‚ 32.01953125 disk.nvme0n1.in‚ 32.01953125 disk.rbd4.out‚ 1.2265625 disk.sda.in‚0 disk.sda3.out‚114.875 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.sda2.in‚0 disk.drbd1743.out‚ 2.6484375) proc.memory.availableMB‚ 47124.9453125( disk./.usagePercent‚83.2952152621943 disk.rbd3.in‚0 disk.rbd5.in‚0! disk.nvme0c0n1.in‚ 32.01953125 disk.rbd2.out‚ 1.2265625 disk.sda3.in‚0 disk.dm-1.out‚114.8753_§ˆ2±- _wandbruntime‚2350  _runtime‚2350¥'° DzÁ ' lr-AdamW/pg1‚0.00013527820686137172 ' lr-AdamW/pg2‚0.00013527820686137172 trainer/global_step‚5249 $ _timestamp‚1.785808072099098e+09 _runtime‚ 2350.39090036 _step‚416lø®pÌ³Æ _step‚416 ' lr-AdamW/pg1‚0.00013527820686137172 ' lr-AdamW/pg2‚0.00013527820686137172 trainer/global_step‚5249 $ _timestamp‚1.785808072099098e+09 _runtime‚ 2350.39090036 ¯Æ W2´- _wandbruntime‚2350  _runtime‚2350è?Dv¾'µ¸O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4619190571520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87890466396832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4619278001152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20671645 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92498437744716 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20671647 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34880327 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4619278001152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35705160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4744135716352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15033515 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90317186 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94749326473216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55446515 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4619190571520 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34880327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4607885544760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744262196224 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90343212438528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94749222004224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94749222004224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90343187534848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94749326473216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55446515 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34880329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚5249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858080721003425e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90326844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87890449357472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55436859 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95087474634752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90130048472064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95087323251200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90343187534848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94749326473216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55446515 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34880329 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 trainer/global_step‚5249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858080721003425e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90326844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87890449357472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55436859 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95087474634752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90130048472064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95087323251200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34880327 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4607885544760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744262196224 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90343212438528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94749222004224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94749222004224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4620157627904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55457052 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92515952547774 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34886972 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95105428582400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20675582 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94767372862976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4608848821218 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55447382 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92516055390370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87907206569152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 $ _timestamp‚1.785808072514773e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90344024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34886970 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55457052 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35711884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90147215235072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87907189529792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90344024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15036547 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90334352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90147198195712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95105590444544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34886972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚418 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4620070191104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚5249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35712131 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMŸ´–ê onitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90334352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15036302 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90360394651648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚2350.807028872 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4620157627904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55457052 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92515952547774 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34886972 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95105428582400 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20675582 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94767372862976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4608848821218 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55447382 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚92516055390370 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87907206569152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 $ _timestamp‚1.785808072514773e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90344024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34886970 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55457052 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35711884 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90147215235072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87907189529792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90344024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15036547 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90334352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90147198195712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95105590444544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34886972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚418 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4620070191104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 trainer/global_step‚5249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35712131 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90334352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15036302 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90360394651648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚2350.807028872 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4663274136576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91203543374848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993044282880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4663274136576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91203568278528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55964861 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95651556290048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚2372.989763167 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90988386622464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90988369583104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35212477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15176177 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95992897093632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55964861 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4789353718784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55974517 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90988386622464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4663186706944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15176406 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95651660759040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95651556290048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35212479 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36044903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93379231981740 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4789476004352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95651660759040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚88727457973472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20868497 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20868495 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4663274136576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91203543374848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993044282880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4663274136576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91203568278528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55964861 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95651556290048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚2372.989763167 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90988386622464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90988369583104 =6DeviceStatsMonitor.on_train_batch_sb¼°âtart/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35212477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15176177 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95992897093632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55964861 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4789353718784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55974517 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90988386622464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4663186706944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15176406 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95651660759040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95651556290048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35212479 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36044903 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93379231981740 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4789476004352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚95651660759040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚88727457973472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20868497 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20868495 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚422 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20872434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91005553385472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15179228 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _runtime‚ 2373.35793905 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4664066326528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4652737284726 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96011165581312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91220750491648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚5299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15178978 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88744215185152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36051410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91005536346112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91194504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91005536346112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95669707148800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95669602672640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20872432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95669707148800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55985054 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93396952469878 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35219120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55975384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91204176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91194504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4664153763328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91220775395328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4790390185984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91005553385472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35219122 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55985054 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4790257421824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36051410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91005536346112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91194504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91005536346112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95669707148800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚95669602672640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20872432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95669707148800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55985054 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93396952469878 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35219120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55975384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91204176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91194504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4664153763328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91220775395328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4790390185984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91005553385472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35219122 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55985054 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4790257421824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚422 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20872434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91005553385472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15179228 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _runtime‚ 2373.35793905 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4664066326528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4652737284726 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96011165581312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91220750491648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚5299 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15178978 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88744215185152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4707182842368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4695748274900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56492863 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4834527831040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56502519 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707270272000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96553995044864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4707270272000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21065345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89564483628832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ °Åà+2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚425 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92063924118528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96553890575872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858081155570276e+09 _runtime‚ 2393.85193179 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35544627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4834664796672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96553995044864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92037490 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92047148 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4707182842368 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89564466589472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35544629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36385033 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35544629 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92063899214848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94260129061248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15319416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96553890575872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4707182842368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4695748274900 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56492863 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4834527831040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56502519 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707270272000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96553995044864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4707270272000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21065345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89564483628832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚425 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92063924118528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96553890575872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858081155570276e+09 _runtime‚ 2393.85193179 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35544627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4834664796672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96553995044864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92037490 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92047148 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4707182842368 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89564466589472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35544629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36385033 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35544629 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92063899214848 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94260129061248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15319416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96553890575872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15322318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91863874496512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4835568852992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4696625748234 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36391600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96916700088320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2394.190524886 _step‚426 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92064328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92081106331648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35551270 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91863874496512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35551270 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚5349 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56503386 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4708062461952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92081131235328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92064328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92054656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91863891535872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35551272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4835429797376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21069284 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94277849549386 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89581223801152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92054656 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56513056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4708149898752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858081158985286e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96572041434624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94277746706790 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36391855 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15322571 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4708062461952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56513056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96572041434624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91863891535872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89581206761792QÐBö×ÇM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92064328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92081106331648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35551270 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91863874496512 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35551270 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚5349 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56503386 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4708062461952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92081131235328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92064328 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92054656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91863891535872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35551272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4835429797376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21069284 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94277849549386 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89581223801152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92054656 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56513056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4708149898752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858081158985286e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96572041434624 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94277746706790 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36391855 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15322571 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4708062461952 REDeviceStatsMonitor.on_train_bat5Ö×ch_end/allocation.small_pool.allocated‚56513056 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96572041434624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91863891535872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89581206761792 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35551272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708149898752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4696539944998 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15322318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚91863874496512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4835568852992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4696625748234 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36391600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96916700088320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2394.190524886 _step‚426 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255ª3µD‘2Ø- _wandbruntime‚2394  _runtime‚2394yµvÉÙþ 0loss/FBt1/assignment_loss‚0.4015934355510353 epoch‚0 $ _timestamp‚1.785808115899867e+09 0loss/FRt1/assignment_loss‚3.2397910301501933 /loss/FRt1/detection_loss‚2.8660589846280904 1loss/SRqqt1/detection_loss‚1.2288978871161556 &loss/total_loss‚22.035237227235935 trainer/global_step‚5349 _step‚427 1loss/SRqqt2/assignment_loss‚1.321836480643691 1loss/FBt2/assignment_loss‚0.04809885991605698 .loss/FRt2/detection_loss‚2.727028436757423 /loss/FBt1/detection_loss‚1.9140726340224938 /loss/FBt2/detection_loss‚1.2789275197714838 _runtime‚2394.191514582 0loss/FRt2/assignment_loss‚3.2053941891000077 2loss/SRqqt1/assignment_loss‚1.3868251115083694 1loss/SRqqt2/detection_loss‚2.4167126580709364œ:Ëf Úƒ 0loss/FRt1/assignment_loss‚3.2397910301501933 /loss/FRt1/detection_loss‚2.8660589846280904 1loss/SRqqt1/detection_loss‚1.2288978871161556 &loss/total_loss‚22.035237227235935 trainer/global_step‚5349 _step‚427 1loss/SRqqt2/assignment_loss‚1.321836480643691 1loss/FBt2/assignment_loss‚0.04809885991605698 .loss/FRt2/detection_loss‚2.727028436757423 /loss/FBt1/detection_loss‚1.9140726340224938 /loss/FBt2/detection_loss‚1.2789275197714838 _runtime‚2394.191514582 0loss/FRt2/assignment_loss‚3.2053941891000077 2loss/SRqqt1/assignment_loss‚1.3868251115083694 1loss/SRqqt2/detection_loss‚2.4167126580709364 0loss/FBt1/assignment_loss‚0.4015934355510353 epoch‚0 $ _timestamp‚1.785808115899867e+09«”ôœÛ‚(:‘ ù‰ÅÓâñÒ gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.51166666666667 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠXä™Ü‚(:‚  ù‰ÅÓî³ÆÖ cpu‚13.92846534331698 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.02734375 disk.sda.out‚ 117.71484375 disk.rbd2.out‚ 1.2265625% disk./.usageGB‚1241.1590881347656 disk.rbd0.in‚0 network.sent‚ 4.996255e+06$ proc.memory.rssMB‚11267.64453125" disk.dm-0.out‚7960.44873046875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 32.02734375 disk.sda1.in‚0 disk.rbd4.out‚ 1.2265625 network.recv‚330585 disk.dm-1.in‚0 disk.sda.in‚0% memory_percent‚28.093230724334717) disk./.usagePercent‚83.29522780656349 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0% disk.nvme0n1.out‚7960.44873046875 disk.rbd4.in‚0 disk.sda3.out‚ 117.71484375 proc.cpu.threads‚29 disk.dm-0.in‚ 32.02734375 disk.dm-1.out‚ 117.71484375 disk.drbd1743.in‚0 disk.drbd1743.out‚ 2.69921875 disk.rbd5.out‚ 1.2265625 disk.rbd2.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 47124.8203125 disk.sda1.out‚0 disk.rbd5.in‚0' disk.nvme0c0n1.out‚7960.44873046875* proc.memory.percent‚17.193061113357544 x›Ý‚(: ˆŠÅÓžÈýÒ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚151.97' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.42777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á,׃ނ(:ø ˆŠÅÓÚíÎÕ disk.rbd0.in‚0 disk.rbd4.in‚0 disk.dm-1.in‚0$ proc.memory.rssMB‚11267.64453125% disk.nvme0n1.out‚7961.08544921875 disk.rbd5.out‚ 1.2265625 disk.sda.in‚0% disk./.usageGB‚1241.1590309143066" disk.dm-0.out‚7961.08544921875 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 32.02734375' disk.nvme0c0n1.out‚7961.08544921875 disk.rbd1.in‚0$ memory_percent‚28.09324860572815 disk.drbd1743.out‚ 2.7421875 disk.dm-0.in‚ 32.02734375 disk.rbd3.out‚0 disk.sda.out‚ 118.28125 disk.rbd2.out‚ 1.2265625* proc.memory.availableMB‚47124.80859375 disk.nvme0n1.in‚ 32.02734375 disk.sda1.out‚0 disk.rbd5.in‚0 disk.dm-1.out‚ 118.28125* proc.memory.percent‚17.193061113357544 cpu‚13.92770929548842 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda2.in‚0 proc.cpu.threads‚29 disk.rbd4.out‚ 1.2265625 disk.sda3.out‚ 118.28125) disk./.usagePercent‚83.29522396645048 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.out‚0 network.sent‚ 5.041021e+06 network.recv‚332472ÓÈS"2ß- _wandbruntime‚2416  _runtime‚2416+G8ÏÉàà % _timestamp‚1.7858081383514209e+09 _runtime‚2416.643076561 _step‚428 ' lr-AdamW/pg1‚0.00013914403483416762 ' lr-AdamW/pg2‚0.00013914403483416762 trainer/global_step‚5399PL.ÎáÈ ' lr-AdamW/pg1‚0.00013914403483416762 ' lr-AdamW/pg2‚0.00013914403483416762 trainer/global_step‚5399 % _timestamp‚1.7858081383514209e+09 _runtime‚2416.643076561 _step‚428¬—^®.2â- _wandbruntime‚2416  _runtime‚2416óq¥¾'ã¸O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92705045883904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4751266407424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36726173 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90401492244832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92897642 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92907300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21262195 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4751178977792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92705062923264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97804174133760 _runtime‚2416.644549203 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57030521 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35876777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15463744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858081383523774e+09 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚95141128983240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚429 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739550935284 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21262197 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35876779 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97456224861696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97804018555904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57030521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4879894175232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4751266407424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57020865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15463976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4739636738408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95141026140756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35876779 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97456329330688 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97456329330688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4879763501056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57020865 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92705062923264;eØNÃ'ä½O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92705045883904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35876777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15463744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 % _timestamp‚1.7858081383523774e+09 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚95141128983240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚429 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739550935284 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21262197 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35876779 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97456224861696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97804018555904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57030521 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4879894175232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4751266407424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57020865 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15463976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4739636738408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95141026140756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35876779 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97456329330688 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚97456329330688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4879763501056 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57020865 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92705062923264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92705045883904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4751266407424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36726173 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90401492244832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92897642 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92907300 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21262195 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4751178977792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92705062923264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97804174133760 _runtime‚2416.644549203 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57030521 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36732771 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21266132 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95158746628894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15466899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97474375720448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97822300314624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97822138452480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36733033 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4740514211742 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90418215377792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57041058 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57031388 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92914808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57041058 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740428408506 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35883420 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97474271244288 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92722229686272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4880676280832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21266134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35883420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92941462171648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4880813239296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚430 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92722229686272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15466639 _runtime‚2417.001003624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92722212646912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92722212646912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90418232417152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92924480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4752058597376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97474375720448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4752146034176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92924480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92914808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4752146034176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95158643786298 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57031388 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92941487075328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640r¯pýÌ&çÆM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97474375720448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97822300314624 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97822138452480 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36733033 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4740514211742 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90418215377792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57041058 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57031388 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92914808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57041058 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740428408506 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35883420 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97474271244288 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92722229686272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4880676280832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21266134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35883420 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92941462171648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4880813239296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚430 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92722229686272 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15466639 _runtime‚2417.001003624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92722212646912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92722212646912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90418232417152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92924480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4752058597376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97474375720448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4752146034176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92924480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92914808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4752146034176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95158643786298 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57031388 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92941487075328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35883422 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36732771 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21266132 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95158746628894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15466899 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236®>`èX2è- _wandbruntime‚2417  _runtime‚2417vz—éý 1loss/SRqqt1/detection_loss‚0.9534565727674792 /loss/FBt2/detection_loss‚1.0040174868085343 epoch‚0 trainer/global_step‚5399 _step‚431 /loss/FRt2/assignment_loss‚3.166857454968595 2loss/SRqqt2/assignment_loss‚1.4027035985444043 2loss/SRqqt1/assignment_loss‚1.5877673682328817 0loss/FBt1/assignment_loss‚0.1571473805230994 2loss/FBt2/assignment_loss‚0.051965328858083616 .loss/FRt1/detection_loss‚2.341294366505838 .loss/FBt1/detection_loss‚3.637427447947175 0loss/FRt1/assignment_loss‚3.3346937311592924 /loss/FRt2/detection_loss‚2.6222172859071313 /loss/SRqqt2/detection_loss‚2.33977107824506 &loss/total_loss‚22.599319100467575 % _timestamp‚1.7858081387103333e+09 _runtime‚2417.001867664á]Žê‚ .loss/FBt1/detection_loss‚3.637427447947175 0loss/FRt1/assignment_loss‚3.3346937311592924 /loss/FRt2/detection_loss‚2.6222172859071313 /loss/SRqqt2/detection_loss‚2.33977107824506 &loss/total_loss‚22.599319100467575 % _timestamp‚1.7858081387103333e+09 _runtime‚2417.001867664 1loss/SRqqt1/detection_loss‚0.9534565727674792 /loss/FBt2/detection_loss‚1.0040174868085343 epoch‚0 trainer/global_step‚5399 _step‚431 /loss/FRt2/assignment_loss‚3.166857454968595 2loss/SRqqt2/assignment_loss‚1.4027035985444043 2loss/SRqqt1/assignment_loss‚1.5877673682328817 0loss/FBt1/assignment_loss‚0.1571473805230994 2loss/FBt2/assignment_loss‚0.051965328858083616 .loss/FRt1/detection_loss‚2.341294366505838¯¥Ã_œë‚(:‘ —ŠÅÓËŠñÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚153.334' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.18555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª«»Žì‚(:ƒ  —ŠÅÓ¯øÀÖ disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 32.02734375 network.recv‚333545 disk.sda1.out‚0 disk.rbd5.out‚ 1.2265625 disk.rbd0.out‚0 disk.rbd4.out‚ 1.2265625% memory_percent‚28.093308210372925) disk./.usagePercent‚83.29522524648814" disk.dm-0.out‚7961.35888671875 disk.rbd0.in‚0 disk.sda.out‚ 119.06640625* proc.memory.percent‚17.193061113357544 disk.dm-0.in‚ 32.02734375 disk.nvme0n1.in‚ 32.02734375% disk.nvme0n1.out‚7961.35888671875 disk.rbd5.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd4.in‚0 disk.rbd2.in‚0$ proc.memory.rssMB‚11267.64453125$ disk./.usageGB‚1241.159049987793 disk.sda3.out‚ 119.06640625 disk.sda1.in‚0 proc.cpu.threads‚29 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0' disk.nvme0c0n1.out‚7961.35888671875 disk.sda.in‚0 disk.rbd2.out‚ 1.2265625* proc.memory.availableMB‚47124.76953125 disk.drbd1743.out‚ 2.76171875 disk.rbd3.in‚0 cpu‚13.928128029182975 network.sent‚ 5.043447e+06 disk.dm-1.out‚ 119.06640625 disk.rbd1.out‚ 1.2265625]š±Q2í- _wandbruntime‚2437  _runtime‚2437gªöoÇîÁ % _timestamp‚1.7858081594451993e+09 _runtime‚2437.738942213 _step‚432 & lr-AdamW/pg1‚0.0001404326441584329 & lr-AdamW/pg2‚0.0001404326441584329 trainer/global_step‚5449CÆþ‚ÌïÆ & lr-AdamW/pg1‚0.0001404326441584329 & lr-AdamW/pg2‚0.0001404326441584329 trainer/global_step‚5449 % _timestamp‚1.7858081594451993e+09 _runtime‚2437.738942213 _step‚432°¢²B·2ð- _wandbruntime‚2437  _runtime‚2437ÛóÑ6¾'ñ¸O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96021923220264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15608106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36208927 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4795175113216 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98358663616512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4924983068672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4783525201916 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93563384034304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4795262542848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4795175113216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98709753735680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57548867 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21459045 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36208927 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858081594458833e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37067151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21459047 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91238483821472 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36208929 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98358663616512 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98358559147520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36208929 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93757794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93563401073664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37067396 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57558523 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57558523 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2‡Ðpë{ò½O SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36208927 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4795175113216 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98358663616512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4924983068672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4783525201916 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93563384034304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4795262542848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4795175113216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98709753735680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57548867 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21459045 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36208927 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858081594458833e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37067151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21459047 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91238483821472 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36208929 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98358663616512 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98358559147520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36208929 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93757794 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93563401073664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37067396 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57558523 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57558523 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96021923220264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15608106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856±f¨û2ó- _wandbruntime‚2438  _runtime‚2438R3yÈ&ôÂM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93784632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57569060 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15610814 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚434 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93580550797312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93580550797312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93784632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4796142169600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57559390 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36215570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15611059 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57569060 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36215572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98376605530112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98376710006272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93774960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4926017866752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21462982 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96039643708402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37073796 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93801842915328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4925891394048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858081597815995e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21462984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98727709405696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚5449 _runtime‚2438.073527842 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37074043 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96039540865806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36215572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4796142169600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36215570 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91255223993792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57559390 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93801818011648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98376710006272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93774960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4784402675250 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93784632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4796142169600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57559390 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36215570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15611059 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57569060 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36215572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98376605530112 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98376710006272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93774960 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4926017866752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21462982 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96039643708402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37073796 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93801842915328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4925891394048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858081597815995e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21462984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98727709405696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚5449 _runtime‚2438.073527842 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37074043 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96039540865806 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36215572 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4796142169600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36215570 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚91255223993792 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57559390 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93801818011648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98376710006272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93774960 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4784402675250 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15610814 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚434 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93580550797312 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚93580550797312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858081790249095e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36541079 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21655897 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4827327862300 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4970202947584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94617946 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58086525 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92075492437472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99260997902336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58076869 trainer/global_step‚5499 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58086525 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94627604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21655895 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15750453 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96902820299772 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37406348 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4839258678272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96902923142256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4970331524608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99260893433344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484$¢U}Ã'€½O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36541079 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4839258678272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4839171248640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94627604 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21655895 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15750453 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96902820299772 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37406348 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4839258678272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96902923142256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4970331524608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99260893433344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36541077 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94421739224064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839171248640 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99260893433344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37406594 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94644991638528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58076869 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99260997902336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94421739224064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94421722184704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858081790249095e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36541079 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21655897 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4827327862300 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4970202947584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94617946 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58086525 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92075492437472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99260997902336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58076869 trainer/global_step‚5499 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58086525 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94662198755328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4840138305024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36547722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99279044292096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21659834 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99633441020928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94644784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4971099015680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36547722 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37413165 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4828205335522 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92092249649152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94635112 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4828291138758 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36547720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4840050868224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99633272867328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99279044292096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94438905987072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94438905987072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58087392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58097062 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4971242265600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15753582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94438888947712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96920437945314 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99278939815936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58087392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4840138305024 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94644784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4971099015680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36547722 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37413165 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4828205335522 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92092249649152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94635112 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4828291138758 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36547720 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4840050868224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99633272867328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_íV4î;batch_end/segment.large_pool.allocated‚130 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99279044292096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94438905987072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94438905987072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58087392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58097062 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4971242265600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15753582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94438888947712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96920437945314 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99278939815936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58087392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4840138305024 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94662198755328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 143249920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4840138305024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36547722¶@Sº2„- _wandbruntime‚2457  _runtime‚2457gŽ@<…ý /loss/FRt2/detection_loss‚2.4928673779548602 1loss/SRqqt2/detection_loss‚2.1816249390443168 _runtime‚2457.652374265 _step‚439 epoch‚0 2loss/SRqqt1/assignment_loss‚1.3020413632576282 1loss/SRqqt2/assignment_loss‚1.338869489538364 0loss/FBt1/assignment_loss‚1.0271608953674634 .loss/FRt1/detection_loss‚2.720819183037831 1loss/SRqqt1/detection_loss‚1.0751293422087285 $loss/total_loss‚49.6579473020333 /loss/FRt1/assignment_loss‚3.484631479703463 0loss/FRt2/assignment_loss‚3.3638394322506215 2loss/FBt2/assignment_loss‚0.001280111683444472 % _timestamp‚1.7858081793608913e+09 /loss/FBt1/detection_loss‚28.953880437809858 /loss/FBt2/detection_loss‚1.7158032501767246 trainer/global_step‚5499€Vd®†‚ _step‚439 epoch‚0 2loss/SRqqt1/assignment_loss‚1.3020413632576282 1loss/SRqqt2/assignment_loss‚1.338869489538364 0loss/FBt1/assignment_loss‚1.0271608953674634 .loss/FRt1/detection_loss‚2.720819183037831 1loss/SRqqt1/detection_loss‚1.0751293422087285 $loss/total_loss‚49.6579473020333 /loss/FRt1/assignment_loss‚3.484631479703463 0loss/FRt2/assignment_loss‚3.3638394322506215 2loss/FBt2/assignment_loss‚0.001280111683444472 % _timestamp‚1.7858081793608913e+09 /loss/FBt1/detection_loss‚28.953880437809858 /loss/FBt2/detection_loss‚1.7158032501767246 trainer/global_step‚5499 /loss/FRt2/detection_loss‚2.4928673779548602 1loss/SRqqt2/detection_loss‚2.1816249390443168 _runtime‚2457.652374265·?…h.›‡‚(: µŠÅÓ¦Þ‡Ó gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚151.44' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.13333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç:m\‰ˆ‚(:þ µŠÅÓÁä°Õ disk.sda2.out‚0 cpu‚13.928983410562566 disk.dm-0.in‚ 32.02734375 disk.rbd0.out‚0 disk.dm-1.out‚ 120.49609375 disk.rbd3.in‚0 network.sent‚ 5.133077e+06) disk./.usagePercent‚83.29287688937657 disk.drbd1743.out‚ 2.80078125 disk.rbd2.out‚ 1.2265625 proc.cpu.threads‚29 disk.dm-0.out‚8256.294921875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 1.2265625 disk.rbd2.in‚0$ proc.memory.rssMB‚11267.64453125* proc.memory.percent‚17.193061113357544% disk./.usageGB‚1241.1240577697754 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0* proc.memory.availableMB‚47124.93359375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.02734375# disk.nvme0n1.out‚8256.294921875% disk.nvme0c0n1.out‚8256.294921875 disk.sda.in‚0% memory_percent‚28.093057870864868 disk.rbd5.out‚ 1.2265625! disk.nvme0c0n1.in‚ 32.02734375 disk.sda.out‚ 120.49609375 network.recv‚337757 disk.drbd1743.in‚0 disk.sda3.out‚ 120.49609375 disk.dm-1.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0<éÝ›‰‚(: ÄŠÅÓÌìÓ gpu.0.gpu‚9.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚146.301' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚81.27833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÊ-…ŽŠ‚(:ƒ  ÄŠÅÓ€ªÖ disk.sda3.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0* proc.memory.availableMB‚47124.43359375 network.sent‚ 5.177636e+06 network.recv‚339654 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.dm-1.in‚0$ proc.memory.rssMB‚11267.64453125 disk.rbd3.in‚0% disk.nvme0n1.out‚8805.93603515625 disk.rbd0.in‚0 disk.sda.out‚ 121.10546875* proc.memory.percent‚17.193061113357544 disk.nvme0n1.in‚ 32.02734375 disk.sda1.out‚0 disk.rbd5.out‚ 1.2265625' disk.nvme0c0n1.out‚8805.93603515625% disk./.usageGB‚1241.1241149902344 disk.rbd3.out‚0 disk.rbd2.out‚ 1.2265625 disk.sda3.out‚ 121.10546875 disk.dm-1.out‚ 121.10546875 disk.rbd1.in‚0 proc.cpu.threads‚29 disk.rbd5.in‚0) disk./.usagePercent‚83.29288072948958 disk.dm-0.in‚ 32.02734375" disk.dm-0.out‚8805.93603515625 disk.sda.in‚0% memory_percent‚28.093820810317993 disk.drbd1743.in‚0 disk.drbd1743.out‚ 2.80859375! disk.nvme0c0n1.in‚ 32.02734375 disk.sda2.out‚0 cpu‚13.92982370763956 disk.rbd4.in‚0 disk.rbd4.out‚ 1.2265625Š—¥x2‹- _wandbruntime‚2476  _runtime‚2476s/´ÉŒÃ ' lr-AdamW/pg2‚0.00014300986280696353 trainer/global_step‚5549 % _timestamp‚1.7858081984722373e+09 _runtime‚2476.763881714 _step‚440 ' lr-AdamW/pg1‚0.00014300986280696353VH´ÎÈ ' lr-AdamW/pg2‚0.00014300986280696353 trainer/global_step‚5549 % _timestamp‚1.7858081984722373e+09 _runtime‚2476.763881714 _step‚440 ' lr-AdamW/pg1‚0.00014300986280696353¸§2Ž- _wandbruntime‚2476  _runtime‚2476m+ýRÄ'¾O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858081984731581e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37745883 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15893396 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95478098 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4883254813696 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100520866122240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100520708447232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36873229 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4883254813696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4883167384064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚2476.767928681 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100163227719168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92912518092832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58604871 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97783820221764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36873227 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95280077374464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37746143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21852747 trainer/global_step‚5549 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5015385872384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4871302128932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100163332188160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95505322574848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36873229 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58614527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15893138y4vŸÉ'ÃO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4871302128932 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100163332188160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95505322574848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36873229 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58614527 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15893138 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858081984731581e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37745883 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15893396 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95478098 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4883254813696 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100520866122240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100520708447232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36873229 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4883254813696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4883167384064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _runtime‚2476.767928681 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100163227719168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92912518092832 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58604871 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97783820221764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36873227 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95280077374464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37746143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21852747 trainer/global_step‚5549 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5015385872384¹^u\–2‘- _wandbruntime‚2477  _runtime‚2477òMn]’ÇM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36879870 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95297244137472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4884047003648 F@DeviceStatsMonitor.on_tr“çô¼$ain_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97801437867418 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37752630 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36879872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100538827868672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95522554595328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4884047003648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95504936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5016298177024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92929258265152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4872179602266 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36879872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4884134440448 _step‚442 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785808198808781e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95504936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15895948 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4884134440448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58625064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4872093799030 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95495264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95297244137472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36879870 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58615394 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100181378577920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97801335024822 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92929241225792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58615394 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4884047003648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97801437867418 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37752630 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36879872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 142987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100538827868672 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95522554595328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4884047003648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95504936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5016298177024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚92929258265152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4872179602266 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36879872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4884134440448 _step‚442 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 $ _timestamp‚1.785808198808781e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95504936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15895948 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4884134440448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58625064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4872093799030 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95495264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95297244137472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36879870 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58615394 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100181378577920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97801335024822 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚92929241225792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58615394 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101065562004992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37205377 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4927250949120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37205377 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16037515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98664614458788 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38086870 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96338250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37205379 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96138415524864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37205377 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4927250949120 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37205377 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16037515 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98664614458788 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38086870 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96338250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37205379 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96138415524864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101065562004992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38093857 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4916068065774 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96355416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37212020 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101444506090496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96355416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2496.854020159 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96365088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96155582287872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37212022 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38093603 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59143396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101083712863744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37212020 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96365088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37212022 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59153066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5061595655168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5599 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96382910435328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59153066 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98682232104330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4928130575872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928043139072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96382885531648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98682334946926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5061456599552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101083712863744 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93766249841792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96155565248512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22053532 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96155582287872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101444342131200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928043139072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚170393608oÄÓ&¡ÍM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96355416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37212020 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101444506090496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96355416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚2496.854020159 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96365088 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96155582287872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37212022 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38093603 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59143396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101083712863744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37212020 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96365088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37212022 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59153066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5061595655168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5599 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96382910435328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59153066 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98682232104330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4928130575872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928043139072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚96382885531648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98682334946926 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5061456599552 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101083712863744 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93766249841792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96155565248512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22053532 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96155582287872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101444342131200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928043139072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4915982262538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38093857 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4916068065774 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101967896290816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99545511538296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5105850254336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94586535324832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4971159654912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59670531 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4971159654912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97198402 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101967896290816 % _timestamp‚1.7858082374198282e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37537527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4959079055948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22246447 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4958993252824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4971247084544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16180234 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102331909412864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97226034254848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4971247084544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚2515.711825256bWØÉ'¬ÃO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97198402 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101967896290816 % _timestamp‚1.7858082374198282e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37537527 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4959079055948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22246447 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4958993252824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4971247084544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16180234 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102331909412864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97226034254848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4971247084544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 _runtime‚2515.711825256 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59670531 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37537529 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101968000759808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22246445 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59660875 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101967896290816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99545511538296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5105850254336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94586535324832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _step‚449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4971159654912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59670531 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4971159654912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392Ánk±2­- _wandbruntime‚2516  _runtime‚2516Ú“ÀËÎ&®ÈM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97013920438272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5106624274432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4959956529282 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101986047149568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59671398 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97225240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4972126711296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858082377556808e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37544172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37544170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59681068 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4972039274496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37544170 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38433475 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16183093 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97243266275328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97013903398912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4972039274496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38433716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22250382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97013920438272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97225240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22250384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4972126711296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97243241371648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102350014925312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552ª>ƒÅð¯ÍM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5649 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4972126711296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97243241371648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102350014925312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97013920438272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5106624274432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4959956529282 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101986047149568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59671398 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97225240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4972126711296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858082377556808e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37544172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37544170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59681068 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4972039274496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37544170 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38433475 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16183093 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97243266275328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97013903398912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4972039274496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38433716 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22250382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97013920438272 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97225240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22250384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160‘CçW2°- _wandbruntime‚2516  _runtime‚2516…Ãœ±ý 0loss/FBt1/assignment_loss‚0.3972354150390558 0loss/SRqqt1/detection_loss‚0.991651247403804 /loss/FBt2/detection_loss‚1.3286235801673218 % _timestamp‚1.7858082377569413e+09 _runtime‚2516.048436209 0loss/FRt1/assignment_loss‚3.6160896078842444 1loss/SRqqt1/assignment_loss‚1.337313546235584 .loss/FRt2/detection_loss‚2.677527716130386 /loss/FBt1/detection_loss‚2.0244369758163683 trainer/global_step‚5649 _step‚451 0loss/FRt2/assignment_loss‚3.1677202339525574 2loss/SRqqt2/assignment_loss‚1.3678875248879194 0loss/SRqqt2/detection_loss‚2.284432385582477 epoch‚0 1loss/FBt2/assignment_loss‚0.10258034862847369 /loss/FRt1/detection_loss‚2.3555729185662617 &loss/total_loss‚21.651071500294453V»²‚ 1loss/FBt2/assignment_loss‚0.10258034862847369 /loss/FRt1/detection_loss‚2.3555729185662617 &loss/total_loss‚21.651071500294453 0loss/FBt1/assignment_loss‚0.3972354150390558 0loss/SRqqt1/detection_loss‚0.991651247403804 /loss/FBt2/detection_loss‚1.3286235801673218 % _timestamp‚1.7858082377569413e+09 _runtime‚2516.048436209 0loss/FRt1/assignment_loss‚3.6160896078842444 1loss/SRqqt1/assignment_loss‚1.337313546235584 .loss/FRt2/detection_loss‚2.677527716130386 /loss/FBt1/detection_loss‚2.0244369758163683 trainer/global_step‚5649 _step‚451 0loss/FRt2/assignment_loss‚3.1677202339525574 2loss/SRqqt2/assignment_loss‚1.3678875248879194 0loss/SRqqt2/detection_loss‚2.284432385582477 epoch‚0Ãd0á*›³‚(: ñŠÅÓ™¾þÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚156.048' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.69333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸG‡´‚(:ü ñŠÅÓîüÕ disk.sda1.out‚0 disk.sda.in‚0 disk.rbd2.in‚0 network.sent‚ 5.269819e+06 disk.nvme0n1.in‚ 32.02734375 disk.rbd5.in‚0* proc.memory.percent‚17.193061113357544 disk.sda.out‚ 123.15234375 disk.sda3.out‚ 123.15234375 disk.dm-1.out‚ 123.15234375 disk.rbd1.in‚0 disk.sda2.in‚0) disk./.usagePercent‚83.32643205289698 disk.dm-1.in‚0# memory_percent‚28.0939519405365& disk.nvme0c0n1.out‚9070.9970703125 disk.sda2.out‚0$ proc.memory.rssMB‚11267.64453125 cpu‚13.931797458048093 disk.rbd2.out‚ 1.2265625 disk.drbd1743.in‚0 disk.drbd1743.out‚2.84375 disk.dm-0.in‚ 32.02734375! disk.dm-0.out‚9070.9970703125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 1.2265625 network.recv‚344754 disk.rbd3.out‚0$ disk.nvme0n1.out‚9070.9970703125 disk.rbd5.out‚ 1.2265625 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.02734375 disk.rbd1.out‚ 1.2265625% disk./.usageGB‚1241.6240539550781 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0* proc.memory.availableMB‚47124.34765625 proc.cpu.threads‚29Ö=¾œµ‚(:‘ €‹ÅÓÍ·êÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚125.693' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚69.82944444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æß@¶‚(:‚  €‹ÅÓ鎤Ö disk.rbd0.in‚0 disk.rbd0.out‚0$ proc.memory.rssMB‚11267.64453125 disk.rbd1.out‚ 1.2265625 disk.rbd4.in‚0 proc.cpu.threads‚29 disk.sda3.out‚ 123.84765625 disk.sda2.in‚0 disk.rbd5.out‚ 1.2265625$ memory_percent‚28.09396982192993 cpu‚13.932657085539338! disk.nvme0c0n1.in‚ 32.02734375 disk.drbd1743.out‚ 2.87109375 disk.dm-0.in‚ 32.02734375 disk.rbd5.in‚0 disk.rbd4.out‚ 1.2265625) proc.memory.availableMB‚ 47124.3359375 network.recv‚346643 disk.rbd1.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 123.84765625 disk.sda2.out‚0 disk.sda1.in‚0% disk./.usageGB‚1241.6240539550781 disk.drbd1743.in‚0* proc.memory.percent‚17.193061113357544% disk.nvme0n1.out‚9106.11865234375) disk./.usagePercent‚83.32643205289698' disk.nvme0c0n1.out‚9106.11865234375 disk.rbd2.out‚ 1.2265625" disk.dm-0.out‚9106.11865234375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.02734375 disk.sda1.out‚0 network.sent‚ 5.314574e+06 disk.sda.in‚0 disk.sda.out‚ 123.84765625 disk.sda3.in‚0åýß­2·- _wandbruntime‚2535  _runtime‚2535ºèɺɸà _step‚452 ' lr-AdamW/pg1‚0.00014687569077975944 ' lr-AdamW/pg2‚0.00014687569077975944 trainer/global_step‚5699 % _timestamp‚1.7858082570630581e+09 _runtime‚2535.354707451iw4TÎ¹È % _timestamp‚1.7858082570630581e+09 _runtime‚2535.354707451 _step‚452 ' lr-AdamW/pg1‚0.00014687569077975944 ' lr-AdamW/pg2‚0.00014687569077975944 trainer/global_step‚5699Äg “Ó2º- _wandbruntime‚2535  _runtime‚2535‹ÙÎ[Æ'»ÀO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37869679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60198533 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103237310007808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2535.358953222 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97855074786304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38767414 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98086390094848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7858082570639877e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102870230576640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5015155790336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98068212 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102870230576640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95423526901472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102870335045632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22443297 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16323877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5150919912960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5015243219968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _step‚453 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98058554 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97855091825664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100426408617804 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97855074786304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5015155790336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16324117 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚5699P>Br¼ÅO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37869679 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60198533 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103237310007808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚2535.358953222 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97855074786304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38767414 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98086390094848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7858082570639877e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 187192320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102870230576640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5015155790336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98068212 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102870230576640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95423526901472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102870335045632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22443297 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16323877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5150919912960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936ÀÎnY ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5015243219968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _step‚453 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98058554 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97855091825664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100426408617804 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97855074786304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5015155790336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16324117 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚5699 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130ÅœÈùÉ2½- _wandbruntime‚2535  _runtime‚2535ËÊéÐ&¾ÊM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102888276959232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37876320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37876322 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97872258588672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60199400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102888381435392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5016122846720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5016035409920 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5003759189554 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60199400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5016035409920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5151954627072 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37876322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37876320 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103255576742400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38773922 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98085392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97872258588672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16326690 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100444026263346 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98085392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38774184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22447232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97872241549312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5016122846720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95440267073792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97872241549312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5016122846720 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95440267073792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37876322 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97872258588672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60199400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102888381435392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5016122846720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5016035409920 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5003759189554 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 187192320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60199400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5016035409920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5151954627072 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37876322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37876320 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103255576742400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38773922 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98085392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97872258588672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16326690 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100444026263346 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121Æï%hÊ2À- _wandbruntime‚2535  _runtime‚2535¶ªgÁþ 2loss/SRqqt1/assignment_loss‚1.2305490669772723 &loss/total_loss‚20.863921457632355 0loss/FRt1/assignment_loss‚3.1394058074269977 1loss/FBt2/assignment_loss‚0.07517484531467467 1loss/SRqqt1/detection_loss‚0.9578870318310205 1loss/SRqqt2/detection_loss‚2.3056878781801946 .loss/FBt2/detection_loss‚1.241513130058382 epoch‚0 trainer/global_step‚5699 $ _timestamp‚1.785808257402883e+09 1loss/SRqqt2/assignment_loss‚1.334720664733165 _runtime‚2535.694355799 0loss/FRt2/assignment_loss‚3.5133005204024137 0loss/FBt1/assignment_loss‚0.1716490139516864 /loss/FRt1/detection_loss‚2.5943110500063216 /loss/FRt2/detection_loss‚2.6365039436905473 /loss/FBt1/detection_loss‚1.6632185050596793 _step‚455ý✠ƒ .loss/FBt2/detection_loss‚1.241513130058382 epoch‚0 trainer/global_step‚5699 $ _timestamp‚1.785808257402883e+09 1loss/SRqqt2/assignment_loss‚1.334720664733165 _runtime‚2535.694355799 0loss/FRt2/assignment_loss‚3.5133005204024137 0loss/FBt1/assignment_loss‚0.1716490139516864 /loss/FRt1/detection_loss‚2.5943110500063216 /loss/FRt2/detection_loss‚2.6365039436905473 /loss/FBt1/detection_loss‚1.6632185050596793 _step‚455 2loss/SRqqt1/assignment_loss‚1.2305490669772723 &loss/total_loss‚20.863921457632355 0loss/FRt1/assignment_loss‚3.1394058074269977 1loss/FBt2/assignment_loss‚0.07517484531467467 1loss/SRqqt1/detection_loss‚0.9578870318310205 1loss/SRqqt2/detection_loss‚2.3056878781801946LJ%çv›Ã‚(: ‹ÅÓÜúÔÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚154.45' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.80555555555554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿƒ¦TvÄ‚(:ë ‹ÅÓÂÿã×% disk./.usageGB‚1241.6242599487305! disk.nvme0c0n1.in‚ 32.02734375 disk.rbd5.in‚0 disk.rbd0.out‚0% memory_percent‚28.094029426574707' disk.nvme0c0n1.out‚9106.44287109375 disk.sda.out‚124.5 disk.sda3.out‚124.5 disk.drbd1743.in‚0" disk.dm-0.out‚9106.44287109375 disk.rbd3.out‚0 disk.rbd5.out‚ 1.2265625 disk.rbd4.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0* proc.memory.percent‚17.193061113357544 network.recv‚347712) disk./.usagePercent‚83.32644587730383 disk.rbd2.out‚ 1.2265625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 1.2265625 cpu‚13.933210892184594 disk.rbd1.in‚0% disk.nvme0n1.out‚9106.44287109375 proc.cpu.threads‚29 disk.dm-1.out‚124.5 disk.rbd3.in‚0 network.sent‚ 5.317016e+06 disk.sda.in‚0 disk.rbd2.in‚0 disk.drbd1743.out‚2.890625 disk.dm-0.in‚ 32.02734375 disk.nvme0n1.in‚ 32.02734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 47124.296875$ proc.memory.rssMB‚11267.64453125ãħ2Å- _wandbruntime‚2554  _runtime‚255439ÀŸÉÆà % _timestamp‚1.7858082764577715e+09 _runtime‚2554.749411722 _step‚456 ' lr-AdamW/pg1‚0.00014816430010402475 ' lr-AdamW/pg2‚0.00014816430010402475 trainer/global_step‚5749(D-ÎÇÈ trainer/global_step‚5749 % _timestamp‚1.7858082764577715e+09 _runtime‚2554.749411722 _step‚456 ' lr-AdamW/pg1‚0.00014816430010402475 ' lr-AdamW/pg2‚0.00014816430010402475ÈaáˆÙ2È- _wandbruntime‚2554  _runtime‚2554S­èF#ÉÀO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60716879 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98713429976064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39107404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5196208011776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚101307305697312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98946770838528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39107666 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚101307408539796 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858082764587061e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38201827 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98946745934848 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103772564862464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104142978850304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚96260535517472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98713429976064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60716879 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38201829 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60726535 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22640145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104142816980992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5046855982964 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103772564862464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚5749 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38201829 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103772669331456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5059151925760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38201827 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5196071046144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98713412936704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2554.751850655I‘P)Ë'ÊÅO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98946745934848 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103772564862464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104142978850304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚96260535517472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98713429976064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60716879 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38201829 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60726535 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22640145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104142816980992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5046855982964 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103772564862464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚5749 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38201829 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103772669331456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5059151925760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38201827 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5196071046144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98713412936704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2554.751850655 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60716879 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98713429976064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39107404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5196208011776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚101307305697312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98946770838528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39107666 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚101307408539796 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858082764587061e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38201827 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392ÉÅq2Ë- _wandbruntime‚2555  _runtime‚2555#S~Ð&ÌÊM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38208472 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858082767937047e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39114181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16470099 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚101325026185450 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98945544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98935872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5060118982144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39114445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5197110757376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚101324923342854 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38208472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98935872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60727402 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚96277292729152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5060031545344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104160926850560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚458 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98730596739072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38208470 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60737072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98945544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60737072 trainer/global_step‚5749 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚2555.085653158 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103790611245056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22644082 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103790715721216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22644084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104161088712704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5047733456298ø­¥²m$ÍÏM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5060031545344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98963953051648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39114181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16470099 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚101325026185450 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98945544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98935872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5060118982144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39114445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5197110757376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚101324923342854 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38208472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98935872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60727402 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚96277292729152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5060031545344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104160926850560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚458 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98730596739072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38208470 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60737072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98945544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60737072 trainer/global_step‚5749 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚2555.085653158 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103790611245056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22644082 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103790715721216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22644084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104161088712704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5047733456298 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38208472 EDeviceStatsMonitor.on_train_batch_end/active_àÑÏhbytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858082767937047e+09 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104674899148288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5090658643348 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61254537 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104675003617280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104674899148288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99571751087104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102188305619304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99571768126464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61244881 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99571751087104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102188202776820 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61244881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105048510269952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38533977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5241383591424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5241250820096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 2574.18693481 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99778858 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97097561172832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104675003617280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97097544133472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61254537 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39447874Xc?Ê'ØÄO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5090658643348 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61254537 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104675003617280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104674899148288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99571751087104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102188305619304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99571768126464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61244881 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99571751087104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102188202776820 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61244881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105048510269952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38533977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5241383591424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5241250820096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚ 2574.18693481 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99778858 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97097561172832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104675003617280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97097544133472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61254537 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39447874 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16610625 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5103235490816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104674899148288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936Íðq2Ù- _wandbruntime‚2574  _runtime‚2574óR»ÚÉM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99805696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39454546 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102205820422362 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5104027680768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚5799 % _timestamp‚1.7858082962275188e+09 _runtime‚ 2574.51945444 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99796024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61255404 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5104115117568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5242154498560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5091621919806 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16613366 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105066621058048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99824333795328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61265074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99588917850112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16613612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5242287262720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5104027680768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38540620 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99796024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38540622 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104693050007040 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97114301345152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61265074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚99588934889472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38540622 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104693050007040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.lÖ>”¼arge_pool.peak‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22840932 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61255404 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104692945530880 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97114284305792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102205923264958 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39454298 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99805696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38540620 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0§\YÔ&ÛÎM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5104027680768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚5799 % _timestamp‚1.7858082962275188e+09 _runtime‚ 2574.51945444 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99796024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61255404 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5104115117568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5242154498560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5091621919806 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16613366 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105066621058048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99824333795328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61265074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99588917850112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16613612 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5242287262720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5104027680768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38540620 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99796024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38540622 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104693050007040 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97114301345152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61265074 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚99588934889472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38540622 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104693050007040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22840932 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61255404 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104692945530880 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97114284305792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102205923264958 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39454298 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99805696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38540620 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99805696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39454546 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102205820422362Îlè6Ç2Ü- _wandbruntime‚2574  _runtime‚2574‹ü!˜Ýþ 1loss/SRqqt2/detection_loss‚1.8491851579811838 _runtime‚2574.520266693 1loss/SRqqt1/assignment_loss‚1.194320818960247 .loss/FRt1/detection_loss‚2.291653701286206 1loss/SRqqt1/detection_loss‚1.1144785812439424 epoch‚0 % _timestamp‚1.7858082962287796e+09 /loss/FRt2/assignment_loss‚3.723783222366782 0loss/FBt1/assignment_loss‚0.6623784601688385 3loss/FBt2/assignment_loss‚0.0008727335522298977 .loss/FRt2/detection_loss‚2.589964174523073 /loss/FBt1/detection_loss‚2.0433731108903883 /loss/FBt2/detection_loss‚1.9801934387768962 trainer/global_step‚5799 1loss/SRqqt2/assignment_loss‚1.384631659421656 &loss/total_loss‚21.854432710801056 _step‚463 0loss/FRt1/assignment_loss‚3.0195976516296126äê3 Þƒ /loss/FRt2/assignment_loss‚3.723783222366782 0loss/FBt1/assignment_loss‚0.6623784601688385 3loss/FBt2/assignment_loss‚0.0008727335522298977 .loss/FRt2/detection_loss‚2.589964174523073 /loss/FBt1/detection_loss‚2.0433731108903883 /loss/FBt2/detection_loss‚1.9801934387768962 trainer/global_step‚5799 1loss/SRqqt2/assignment_loss‚1.384631659421656 &loss/total_loss‚21.854432710801056 _step‚463 0loss/FRt1/assignment_loss‚3.0195976516296126 1loss/SRqqt2/detection_loss‚1.8491851579811838 _runtime‚2574.520266693 1loss/SRqqt1/assignment_loss‚1.194320818960247 .loss/FRt1/detection_loss‚2.291653701286206 1loss/SRqqt1/detection_loss‚1.1144785812439424 epoch‚0 % _timestamp‚1.7858082962287796e+09Ϧ'Pœß‚(:‘ ­‹ÅÓ¼ÅèÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚161.373' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚89.65166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0' µ&Žà‚(:ƒ  ­‹ÅÓÉÕ‘Ö disk.dm-0.in‚ 32.02734375 disk.rbd5.out‚ 1.2265625' disk.nvme0c0n1.out‚9107.14306640625 disk.dm-1.in‚0 disk.dm-1.out‚ 125.99609375 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.rbd3.in‚0 disk.sda.out‚ 125.99609375 disk.sda3.out‚ 125.99609375 disk.nvme0n1.in‚ 32.02734375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 1.2265625* proc.memory.availableMB‚47124.35546875% disk.nvme0n1.out‚9107.14306640625 disk.rbd5.in‚0% memory_percent‚28.093940019607544$ proc.memory.rssMB‚11267.64453125 disk.rbd2.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 network.recv‚351832) disk./.usagePercent‚83.32647531817025 disk.drbd1743.out‚ 2.90234375 disk.rbd0.out‚0 disk.rbd4.in‚0$ disk./.usageGB‚1241.624698638916 disk.rbd4.out‚ 1.2265625 disk.sda.in‚0 disk.rbd1.in‚0* proc.memory.percent‚17.193061113357544 network.sent‚ 5.406571e+06 disk.drbd1743.in‚0" disk.dm-0.out‚9107.14306640625 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 32.02734375 disk.sda2.out‚0 cpu‚13.934723384047363&l§ù2á- _wandbruntime‚2593  _runtime‚2593y®xÉâà trainer/global_step‚5849 % _timestamp‚1.7858083155933402e+09 _runtime‚2593.884996393 _step‚464 ' lr-AdamW/pg1‚0.00015074151875255537 ' lr-AdamW/pg2‚0.00015074151875255537uÜÇRÎãÈ ' lr-AdamW/pg1‚0.00015074151875255537 ' lr-AdamW/pg2‚0.00015074151875255537 trainer/global_step‚5849 % _timestamp‚1.7858083155933402e+09 _runtime‚2593.884996393 _step‚464кõ &2ä- _wandbruntime‚2593  _runtime‚2593Vv¨ïÐ'åÊO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105577233434112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105577233434112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5147231626240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5147144196608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105577337903104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 % _timestamp‚1.7858083155943365e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97934569788832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39786335 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100430089237504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5147231626240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚5849 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5286422642688 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61782539 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61772883 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38866127 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100648668 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38866129 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5147144196608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61772883 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38866127 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103069099856328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5134632909980 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16752251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100667457614848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97934552749472 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100430106276864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2593.886237779 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100639010 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23033845 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392±ô¥¤áæÏO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38866129 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5147144196608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61772883 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38866127 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103069099856328 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5134632909980 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16752251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100667457614848 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97934552749472 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100430106276864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2593.886237779 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100639010 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23033845 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105577233434112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105577233434112 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5147231626240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5147144196608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105577337903104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 % _timestamp‚1.7858083155943365e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97934569788832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39786335 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100430089237504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5147231626240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚5849 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5286422642688 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61782539 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61772883 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38866127 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100648668 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448Ñþ±µ2ç- _wandbruntime‚2594  _runtime‚2594X‹‹Ú&èÔM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100656176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105595384292864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5148023816192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100447273039872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5148111252992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39792774 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5148023816192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100665848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100665848 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38872770 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103086820344466 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97951292921792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38872772 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23037784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97951309961152 _step‚466 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5135424580078 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100684664731648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5148111252992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚103086717501870 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100656176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38872770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858083159310443e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚2594.222997201 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100447256000512 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16755252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100684689635328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5135510383314 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ƒ^ß&éÙM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16755252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100684689635328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5135510383314 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100656176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105595384292864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5148023816192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100447273039872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5148111252992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39792774 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5148023816192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100665848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100665848 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38872770 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103086820344466 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97951292921792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38872772 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23037784 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97951309961152 _step‚466 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5135424580078 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100684664731648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5148111252992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚103086717501870 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100656176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38872770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858083159310443e+09 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚2594.222997201 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100447256000512Ò|Eùm2ê- _wandbruntime‚2594  _runtime‚2594š·Äÿëù .loss/FRt1/assignment_loss‚3.25465468757896 2loss/SRqqt1/assignment_loss‚1.2245036411638324 /loss/FRt1/detection_loss‚2.3803623953173236 0loss/SRqqt2/detection_loss‚2.314390808698677 .loss/FBt2/detection_loss‚8.522098809671707 _step‚467 1loss/SRqqt2/assignment_loss‚1.508807390573479 /loss/FRt2/detection_loss‚2.4342791869722564 /loss/FBt1/detection_loss‚2.4576303482055666 $loss/total_loss‚29.0553225524908 epoch‚0 _runtime‚2594.224206327 1loss/FBt1/assignment_loss‚0.38031938664421433 1loss/SRqqt1/detection_loss‚0.9500449460342919 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚3.479142686416363 0loss/FBt2/assignment_loss‚0.1490882652141267 % _timestamp‚1.7858083159323146e+09>ä‰Sìþ _step‚467 1loss/SRqqt2/assignment_loss‚1.508807390573479 /loss/FRt2/detection_loss‚2.4342791869722564 /loss/FBt1/detection_loss‚2.4576303482055666 $loss/total_loss‚29.0553225524908 epoch‚0 _runtime‚2594.224206327 1loss/FBt1/assignment_loss‚0.38031938664421433 1loss/SRqqt1/detection_loss‚0.9500449460342919 trainer/global_step‚5849 /loss/FRt2/assignment_loss‚3.479142686416363 0loss/FBt2/assignment_loss‚0.1490882652141267 % _timestamp‚1.7858083159323146e+09 .loss/FRt1/assignment_loss‚3.25465468757896 2loss/SRqqt1/assignment_loss‚1.2245036411638324 /loss/FRt1/detection_loss‚2.3803623953173236 0loss/SRqqt2/detection_loss‚2.314390808698677 .loss/FBt2/detection_loss‚8.522098809671707Óçæs±œí‚(:‘ ¼‹ÅÓשÜÒ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚142.975' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚79.43055555555554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÉNüî‚(:‚  ¼‹ÅÓ³¼Ö* proc.memory.availableMB‚47124.36328125 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0$ proc.memory.rssMB‚11267.64453125' disk.nvme0c0n1.out‚9107.52880859375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd5.out‚ 1.2265625 disk.sda3.in‚0 proc.cpu.threads‚29% disk.nvme0n1.out‚9107.52880859375 disk.sda1.in‚0 disk.dm-1.in‚0 network.sent‚ 5.451307e+06" disk.dm-0.out‚9107.52880859375! disk.nvme0c0n1.in‚ 32.02734375 disk.sda.in‚0 disk.sda.out‚ 126.57421875* proc.memory.percent‚17.193061113357544 disk.rbd0.out‚0$ memory_percent‚28.09392809867859 cpu‚13.935643832193104 disk.rbd4.out‚ 1.2265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1743.out‚ 2.93359375 disk.dm-0.in‚ 32.02734375 disk.rbd5.in‚0 disk.rbd2.out‚ 1.2265625 disk.sda3.out‚ 126.57421875) disk./.usagePercent‚83.32648504645655$ disk./.usageGB‚1241.624843597412 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.dm-1.out‚ 126.57421875 network.recv‚353387 disk.rbd1.out‚ 1.2265625 disk.nvme0n1.in‚ 32.02734375 disk.sda1.out‚0ƒçÚœï‚(:‘ Ë‹ÅÓ°–„Ó gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚128.213' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.22944444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éø£ÅŠð‚(:ÿ Ë‹ÅÓìë³Ö disk.sda3.in‚0 network.sent‚ 5.496005e+06 disk.drbd1743.out‚2.953125" disk.dm-0.out‚9366.85888671875 disk.sda1.out‚0 disk.rbd5.out‚ 1.2265625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 32.05859375 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 127.2421875$ proc.memory.rssMB‚11267.64453125 cpu‚13.936181841912836 proc.cpu.threads‚29 disk.sda.in‚0* proc.memory.availableMB‚47171.23046875 network.recv‚355599 disk.rbd3.in‚0 disk.rbd3.out‚0% memory_percent‚28.022414445877075* proc.memory.percent‚17.193061113357544% disk.nvme0n1.out‚9366.85888671875 disk.rbd2.out‚ 1.2265625% disk./.usageGB‚1241.8767204284668 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda2.in‚0 disk.nvme0n1.in‚ 32.05859375 disk.rbd5.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚83.34338871192737 disk.sda2.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 1.2265625' disk.nvme0c0n1.out‚9366.85888671875 disk.dm-1.out‚ 127.2421875 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.05859375 disk.rbd0.out‚0 disk.sda.out‚ 127.2421875< ·2ñ- _wandbruntime‚2613  _runtime‚2613Ç‹¸£Éòà _runtime‚2613.288433396 _step‚468 ' lr-AdamW/pg1‚0.00015203012807682068 ' lr-AdamW/pg2‚0.00015203012807682068 trainer/global_step‚5899 % _timestamp‚1.7858083349967923e+09­måÎóÈ ' lr-AdamW/pg1‚0.00015203012807682068 ' lr-AdamW/pg2‚0.00015203012807682068 trainer/global_step‚5899 % _timestamp‚1.7858083349967923e+09 _runtime‚2613.288433396 _step‚468Ô ¹¦h2ô- _wandbruntime‚2613  _runtime‚2613yžUÔ2õÊO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103949996935836 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23230697 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5331629620224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101288444427264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101288427387904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106479567719936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62310541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39198279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40125080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40125324 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62310541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106479567719936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5191227761664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5331760294400 _runtime‚2613.289717914 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5191140332032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16894627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2åEÕGÕ'öÏO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101527838358528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106479567719936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62310541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39198279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40125080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40125324 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62310541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106479567719936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5191227761664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5331760294400 _runtime‚2613.289717914 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5191140332032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16894627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5899 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103949996935836 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23230697 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5331629620224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101288444427264 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101288427387904 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39204922 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39204920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62311408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98788318577152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106497718578688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5192107388416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39204920 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101526000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62321078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16897227 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5332531217920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179398846822 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101545045475328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62321078 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5192019951616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106877711554560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62311408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101305594150912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5192019951616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101516328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5179313043586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚2613.625160085 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23234634 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚470 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5192107388416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5332666079232 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40131859 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101305611190272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5899 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39204922 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39204920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62311408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98788318577152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106497718578688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5192107388416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39204920 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101526000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62321078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16897227 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5332531217920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179398846822 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101545045475328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62321078 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5192019951616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 D>DeviceStatsMoni?€/ tor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106877711554560 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62311408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101305594150912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5192019951616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101516328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5179313043586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _runtime‚2613.625160085 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23234634 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _step‚470 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5192107388416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5332666079232 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40131859 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101305611190272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚5899 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17035967 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62838543 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102368972 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5222324033872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62828887 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107381902005760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5235136467456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5376762848256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99608587020832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5222409836996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5235136467456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17036242 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23427545 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107382006474752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 trainer/global_step‚5949 _runtime‚2632.762958573 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62838543 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107381902005760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5235223897088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107382006474752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102146765538304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104830996857828 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39530427 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130KÕÅ«Õ'„ÏO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102359314 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17035967 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62838543 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _step‚473 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102368972 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5222324033872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62828887 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107381902005760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5235136467456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5376762848256 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99608587020832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5222409836996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5235136467456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17036242 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23427545 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107382006474752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 trainer/global_step‚5949 _runtime‚2632.762958573 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62838543 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107381902005760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5235223897088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107382006474752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102146765538304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104830996857828 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39530427 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ÙJËƧ2…- _wandbruntime‚2633  _runtime‚2633¹HAê†ÓM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107783047419392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102386152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102405401315328 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5236103523840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62849080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99625310153792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62839410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107783196698624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102376480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17038803 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5377795383296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104848614503482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62849080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5223287310330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5223201507094 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102376480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17039057 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858083548037343e+09 _step‚474 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5236016087040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62839410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23431482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102163949340672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool~qwï.allocated‚99625327193152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39537070 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40470285 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40470541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102405376411648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 _runtime‚ 2633.09568117 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39537072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39537072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5377671007744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102386152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62849080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5223287310330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5223201507094 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102376480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17039057 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858083548037343e+09 _step‚474 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5236016087040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62839410 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23431482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚102163949340672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚99625327193152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39537070 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40470285 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40470541 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102405376411648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 _runtime‚ 2633.09568117 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39537072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39537072 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5377671007744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102386152 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5236103523840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62849080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99625310153792 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62839410 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107783196698624 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102376480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17038803 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5377795383296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640ÚÈ?ŠÙ2ˆ- _wandbruntime‚2633  _runtime‚2633v²òk‰ epoch‚0 _runtime‚2633.097023185 2loss/SRqqt2/assignment_loss‚1.9552336840525917 2loss/FBt2/assignment_loss‚0.027298592187506552 .loss/FRt1/detection_loss‚2.587805781786035 1loss/SRqqt1/detection_loss‚1.1464444606054214 trainer/global_step‚5949 _step‚475 /loss/FRt1/assignment_loss‚3.195827984228367 /loss/FBt1/detection_loss‚1.3299089956248054 0loss/FRt2/assignment_loss‚3.3991681017526765 1loss/FBt1/assignment_loss‚0.47088104052036783 /loss/FRt2/detection_loss‚2.6557246621062114 1loss/SRqqt2/detection_loss‚2.7383569568071677 &loss/total_loss‚22.184215390558585 % _timestamp‚1.7858083548049192e+09 2loss/SRqqt1/assignment_loss‚1.2739420009476523 /loss/FBt2/detection_loss‚1.4036231299397828‰ûD Š† 1loss/SRqqt2/detection_loss‚2.7383569568071677 &loss/total_loss‚22.184215390558585 % _timestamp‚1.7858083548049192e+09 2loss/SRqqt1/assignment_loss‚1.2739420009476523 /loss/FBt2/detection_loss‚1.4036231299397828 epoch‚0 _runtime‚2633.097023185 2loss/SRqqt2/assignment_loss‚1.9552336840525917 2loss/FBt2/assignment_loss‚0.027298592187506552 .loss/FRt1/detection_loss‚2.587805781786035 1loss/SRqqt1/detection_loss‚1.1464444606054214 trainer/global_step‚5949 _step‚475 /loss/FRt1/assignment_loss‚3.195827984228367 /loss/FBt1/detection_loss‚1.3299089956248054 0loss/FRt2/assignment_loss‚3.3991681017526765 1loss/FBt1/assignment_loss‚0.47088104052036783 /loss/FRt2/detection_loss‚2.6557246621062114Û¸JÚ¾›‹‚(: é‹ÅÓÿêÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚151.729' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚84.2938888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0У–‚Œ‚(:÷ é‹ÅÓÂ×Õ network.sent‚ 5.543458e+06 disk.rbd2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 32.0703125 disk.rbd3.in‚0 disk.rbd0.in‚0 network.recv‚358975) disk./.usagePercent‚83.36037737189956 disk.rbd1.in‚0 disk.drbd1743.out‚ 2.98828125$ memory_percent‚27.88073420524597' disk.nvme0c0n1.out‚9420.28662109375 disk.sda.in‚0 disk.sda.out‚ 128.671875 disk.sda3.out‚ 128.671875 disk.rbd1.out‚ 1.2265625 proc.cpu.threads‚29% disk./.usageGB‚1242.1298637390137" disk.dm-0.out‚9420.28662109375 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd2.out‚ 1.2265625 disk.sda3.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.nvme0c0n1.in‚ 32.0703125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.0703125 disk.rbd5.out‚ 1.2265625 disk.rbd4.in‚0 disk.rbd4.out‚ 1.2265625 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.availableMB‚47264.08203125 disk.dm-1.out‚ 128.671875 disk.sda2.in‚0 disk.drbd1743.in‚0% disk.nvme0n1.out‚9420.28662109375 disk.sda1.out‚0) proc.memory.percent‚17.19306707382202 cpu‚13.93762377519497Ä&´2- _wandbruntime‚2652  _runtime‚2652YhHsÉŽÃ ' lr-AdamW/pg1‚0.00015460734672535128 ' lr-AdamW/pg2‚0.00015460734672535128 trainer/global_step‚5999 % _timestamp‚1.7858083739938524e+09 _runtime‚2652.285549666 _step‚476Ç.ZÈÎÈ trainer/global_step‚5999 % _timestamp‚1.7858083739938524e+09 _runtime‚2652.285549666 _step‚476 ' lr-AdamW/pg1‚0.00015460734672535128 ' lr-AdamW/pg2‚0.00015460734672535128Ü;'ÕR2- _wandbruntime‚2652  _runtime‚2652â7 |Ñ'‘ËO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚ 2652.28995908 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63356889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚105711791094852 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100445595636832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17179533 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5999 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103005120728064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚105711893937336 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63366545 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5421966455808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5279220032512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5279220032512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7858083739948483e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103005103688704 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108670649265664 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23624397 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5266298300504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103248525134848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40804175 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103005120728064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39862579 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63366545 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5266212497380 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39862579 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40803928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108284236291584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23624395 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5422099227136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23624397 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5266298300504 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103248525134848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40804175 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103005120728064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39862579 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63366545 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5266212497380 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39862579 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40803928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitoÐ#,r.on_train_batch_start/active_bytes.all.freed‚108284236291584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23624395 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5422099227136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39869220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105729511582990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39869222 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103236632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103246304 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40810644 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5267089970602 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 $ _timestamp‚1.785808374342242e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23628334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17182581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105729408740394 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103246304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63367412 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108302387150336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108302387150336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108302282674176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23628332 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108688760803328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103265732251648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100462318769792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5267175773838 _step‚478 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103022287491072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108688603135488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63377082 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39869222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103022270451712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100462335809152 trainer/global_step‚5999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39869220 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5280012222464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236e Mà&•ÚM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 $ _timestamp‚1.785808374342242e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23628334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17182581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚105729408740394 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103246304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63367412 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108302387150336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108302387150336 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108302282674176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23628332 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108688760803328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103265732251648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100462318769792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5267175773838 _step‚478 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103022287491072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108688603135488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63377082 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39869222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103022270451712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100462335809152 trainer/global_step‚5999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39869220 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5280012222464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103022287491072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5280012222464 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39869220 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105729511582990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39869222 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103236632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103246304 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40810644 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5267089970602 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ÞÓÉçÇ2–- _wandbruntime‚2652  _runtime‚2652æÄN—þ &loss/total_loss‚20.591246058951317 0loss/FRt1/assignment_loss‚2.9773740721019832 0loss/FBt1/assignment_loss‚0.2921813014955474 /loss/FRt1/detection_loss‚2.5182850276204674 /loss/FBt2/detection_loss‚1.2844210978656958 _runtime‚2652.635012916 _step‚479 2loss/SRqqt2/assignment_loss‚1.3740743324160576 .loss/FRt2/detection_loss‚2.634415031951151 1loss/SRqqt1/detection_loss‚1.0207237539847631 epoch‚0 trainer/global_step‚5999 % _timestamp‚1.7858083743435295e+09 /loss/FRt2/assignment_loss‚3.107808017436369 1loss/SRqqt1/assignment_loss‚1.155920103973252 1loss/FBt2/assignment_loss‚0.03186684814453649 /loss/FBt1/detection_loss‚2.0495611971074883 1loss/SRqqt2/detection_loss‚2.1446152748540044PÓƒ ˜ƒ 0loss/FBt1/assignment_loss‚0.2921813014955474 /loss/FRt1/detection_loss‚2.5182850276204674 /loss/FBt2/detection_loss‚1.2844210978656958 _runtime‚2652.635012916 _step‚479 2loss/SRqqt2/assignment_loss‚1.3740743324160576 .loss/FRt2/detection_loss‚2.634415031951151 1loss/SRqqt1/detection_loss‚1.0207237539847631 epoch‚0 trainer/global_step‚5999 % _timestamp‚1.7858083743435295e+09 /loss/FRt2/assignment_loss‚3.107808017436369 1loss/SRqqt1/assignment_loss‚1.155920103973252 1loss/FBt2/assignment_loss‚0.03186684814453649 /loss/FBt1/detection_loss‚2.0495611971074883 1loss/SRqqt2/detection_loss‚2.1446152748540044 &loss/total_loss‚20.591246058951317 0loss/FRt1/assignment_loss‚2.9773740721019832ßÆ]ëýœ™‚(:‘ ø‹ÅÓÒÈüÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚157.115' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚87.28611111111113 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ¶>’š‚(:ö ø‹ÅÓ…îÍ× disk.rbd3.in‚0 cpu‚13.938256477359984 network.sent‚ 5.588219e+06% disk./.usageGB‚1242.6944160461426 disk.rbd2.in‚0 disk.nvme0n1.in‚ 32.0703125 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 1.2265625# proc.memory.rssMB‚ 11267.6484375 disk.sda3.out‚ 129.21875 disk.nvme0c0n1.in‚ 32.0703125 network.recv‚360964 disk.sda2.in‚0 disk.drbd1743.in‚0) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29 disk.rbd0.in‚0 disk.dm-1.out‚ 129.21875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625 disk.dm-0.in‚ 32.0703125% disk.nvme0n1.out‚9821.40576171875 disk.sda1.out‚0 disk.rbd5.in‚0) disk./.usagePercent‚83.39826495092191 disk.dm-1.in‚0 disk.drbd1743.out‚ 2.99609375% memory_percent‚27.673226594924927* proc.memory.availableMB‚47400.07421875 disk.sda2.out‚0" disk.dm-0.out‚9821.40576171875 disk.sda1.in‚0 disk.rbd5.out‚ 1.2265625' disk.nvme0c0n1.out‚9821.40576171875 disk.sda.in‚0 disk.sda.out‚ 129.21875 disk.rbd2.out‚ 1.2265625 disk.rbd3.out‚0 disk.sda3.in‚0GX‡›‚(:ü ‡ŒÅÓµ‘Ö network.recv‚363145 disk.drbd1743.out‚3.015625 disk.dm-0.in‚ 32.0703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 32.0703125 disk.rbd2.in‚0) disk./.usagePercent‚83.40234315094237 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 130.03515625' proc.memory.availableMB‚ 47400.40625# proc.memory.rssMB‚ 11267.6484375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.0703125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 130.03515625 disk.sda2.out‚0% memory_percent‚27.672719955444336 cpu‚13.938561020152257 network.sent‚ 5.633121e+06$ disk./.usageGB‚1242.755184173584 disk.rbd5.in‚0 disk.rbd5.out‚ 1.2265625 disk.sda.out‚ 130.03515625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.rbd3.in‚0& disk.nvme0n1.out‚10287.26123046875( disk.nvme0c0n1.out‚10287.26123046875 disk.rbd1.out‚ 1.2265625 disk.drbd1743.in‚0 disk.rbd2.out‚ 1.2265625 disk.sda.in‚0) proc.memory.percent‚17.19306707382202# disk.dm-0.out‚10287.26123046875 disk.rbd4.out‚ 1.2265625 disk.rbd1.in‚0 mUÄœœ‚(:‘ ‡ŒÅÓðº—Ý gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚156.874' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚87.15222222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F°kú2- _wandbruntime‚2672  _runtime‚2672ay¾ÇžÁ & lr-AdamW/pg1‚0.0001558959560496166 & lr-AdamW/pg2‚0.0001558959560496166 trainer/global_step‚6049 % _timestamp‚1.7858083942271504e+09 _runtime‚2672.518924666 _step‚480+’lCÌŸÆ & lr-AdamW/pg1‚0.0001558959560496166 & lr-AdamW/pg2‚0.0001558959560496166 trainer/global_step‚6049 % _timestamp‚1.7858083942271504e+09 _runtime‚2672.518924666 _step‚480à&–ö2 - _wandbruntime‚2672  _runtime‚26722ëºÐ¡ÌO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101282604252832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109186570577408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104108880974848 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106592791016844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103863441839104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104089276 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5310100960888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TUíoÑ"GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17323930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2672.523924097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109186675046400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _step‚481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5323216167936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5467311790592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103863458878464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5323128738304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17324180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109576217669120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40194727 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5323128738304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40194729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚6049 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40194727 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63894547 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109186570577408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63894547 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63884891 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103863458878464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109186570577408 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63894547 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63884891 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103863458878464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104108880974848 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106592791016844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103863441839104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104089276 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5310100960888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17323930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2672.523924097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109186675046400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 _step‚481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5323216167936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5467311790592 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103863458878464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5323128738304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17324180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109576217669120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40194727 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5323128738304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40194729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚6049 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40194727 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63894547 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41152004 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63905084 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63905084 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104106456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103880625641472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚2672.857969629 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40201370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109594332430336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 trainer/global_step‚6049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63895414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7858083945659618e+09 _step‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41152275 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104096784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23825182 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106610408662498 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109204721436160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63895414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101299344425152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109204721436160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109594174762496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5311064237346 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104106456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17327091 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104126088091648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23825184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5468086670848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5324008357888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5324095794688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109204616960000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5468219435008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104096784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40201372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5324095794688 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109204721436160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63895414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101299344425152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109204721436160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109594174762496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5311064237346 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚8£ÊIÛº7436800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104106456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17327091 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104126088091648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23825184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5468086670848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5324008357888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5324095794688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109204616960000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5468219435008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104096784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40201372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5324095794688 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41152004 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63905084 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63905084 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104106456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103880625641472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚2672.857969629 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40201370 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109594332430336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 trainer/global_step‚6049 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63895414 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7858083945659618e+09 _step‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41152275 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712âêȤc2¦- _wandbruntime‚2672  _runtime‚2672Mõ$ʧ€ 2loss/SRqqt2/assignment_loss‚1.1324707398958065 /loss/FRt2/detection_loss‚2.5232942958912217 epoch‚0 trainer/global_step‚6049 _step‚483 0loss/FRt2/assignment_loss‚3.0283298054373406 1loss/SRqqt1/assignment_loss‚1.369344817272023 .loss/FRt1/detection_loss‚2.459203102067113 /loss/FBt2/detection_loss‚1.3225076194997258 0loss/FBt1/assignment_loss‚0.4229989827314934 1loss/SRqqt2/detection_loss‚1.9493148872519241 /loss/FBt1/detection_loss‚1.7039056569337845 _runtime‚2672.860448506 2loss/FBt2/assignment_loss‚0.002543626975836107 1loss/SRqqt1/detection_loss‚1.1392867187294515 &loss/total_loss‚20.346725892607147 % _timestamp‚1.7858083945672214e+09 0loss/FRt1/assignment_loss‚3.2935256399214268Þâé¾ ¨… _step‚483 0loss/FRt2/assignment_loss‚3.0283298054373406 1loss/SRqqt1/assignment_loss‚1.369344817272023 .loss/FRt1/detection_loss‚2.459203102067113 /loss/FBt2/detection_loss‚1.3225076194997258 0loss/FBt1/assignment_loss‚0.4229989827314934 1loss/SRqqt2/detection_loss‚1.9493148872519241 /loss/FBt1/detection_loss‚1.7039056569337845 _runtime‚2672.860448506 2loss/FBt2/assignment_loss‚0.002543626975836107 1loss/SRqqt1/detection_loss‚1.1392867187294515 &loss/total_loss‚20.346725892607147 % _timestamp‚1.7858083945672214e+09 0loss/FRt1/assignment_loss‚3.2935256399214268 2loss/SRqqt2/assignment_loss‚1.1324707398958065 /loss/FRt2/detection_loss‚2.5232942958912217 epoch‚0 trainer/global_step‚6049ãõÇLœ©‚(:‘ –ŒÅÓí«ÇÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚150.909' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.83833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨Öºª‚(:ö –ŒÅÓཱ× disk.nvme0n1.in‚ 32.0703125 disk.sda3.in‚0 disk.rbd5.out‚ 1.2265625 disk.rbd0.in‚0 disk.dm-0.in‚ 32.0703125( disk./.usagePercent‚83.4023239503773 disk.sda3.out‚ 130.60546875 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 32.0703125% disk.nvme0c0n1.out‚10350.05078125 disk.dm-1.out‚ 130.60546875 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 network.recv‚364000# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202 disk.rbd2.out‚ 1.2265625 disk.rbd4.out‚ 1.2265625 cpu‚13.939058715035179 disk.dm-0.out‚10350.05078125# disk.nvme0n1.out‚10350.05078125 disk.dm-1.in‚0 disk.rbd2.in‚0 disk.rbd1.out‚ 1.2265625 disk.drbd1743.in‚0 disk.drbd1743.out‚ 3.02734375 disk.sda1.in‚0 disk.rbd5.in‚0$ memory_percent‚27.67217755317688* proc.memory.availableMB‚47400.76171875 proc.cpu.threads‚29 network.sent‚ 5.635595e+06$ disk./.usageGB‚1242.754898071289 disk.sda.in‚0 disk.sda.out‚ 130.60546875 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd3.in‚0&f2«- _wandbruntime‚2691  _runtime‚2691þ5ÍǬÁ & lr-AdamW/pg1‚0.0001571845653738819 & lr-AdamW/pg2‚0.0001571845653738819 trainer/global_step‚6099 % _timestamp‚1.7858084136386507e+09 _runtime‚2691.930325246 _step‚484’¦}üÌ­Æ trainer/global_step‚6099 % _timestamp‚1.7858084136386507e+09 _runtime‚2691.930325246 _step‚484 & lr-AdamW/pg1‚0.0001571845653738819 & lr-AdamW/pg2‚0.0001571845653738819亘¯¹2®- _wandbruntime‚2691  _runtime‚2691ïü2Ð'¯ÊO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5367212303360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5367124873728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110089009332224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.785808413639619e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64412893 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102119595829472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104721779989504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104721797028864 trainer/global_step‚6099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104969261718528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _step‚485 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107473688096352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64422549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102119612868832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104721797028864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110089009332224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41485994 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5512362265600 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5367212303360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110481599080448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5367124873728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24018095 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 2691.93494204 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104939770 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104721779989504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40526877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64412893 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40526877 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64422549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110088904863232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41485744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107473585253868 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24018097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104949428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40526879 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110088904863232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104969236814848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5354075227520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121¯ ¢Õ'°ÏO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5367212303360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5367124873728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110089009332224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.785808413639619e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64412893 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102119595829472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104721779989504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚104721797028864 trainer/global_step‚6099 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104969261718528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _step‚485 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107473688096352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64422549 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102119612868832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104721797028864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110089009332224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41485994 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5512362265600 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5367212303360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110481599080448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5367124873728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24018095 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 2691.93494204 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104939770 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104721779989504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40526877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64412893 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40526877 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64422549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110088904863232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41485744 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107473585253868 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24018097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104949428 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40526879 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110088904863232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104969236814848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5354075227520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121åRô=¬2±- _wandbruntime‚2692  _runtime‚26924±>¼¶²ÖM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41492555 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354952700854 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104966608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64423416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40533520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5513403172864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚2692.268847474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚110107055721984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104738946752512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40533520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110106951245824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17470523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚6099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64433086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110499712243200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102136353041152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858084139768298e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104966608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110499872008192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104738963791872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64433086 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107491202899410 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚486 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64423416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104738946752512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5354866897618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110107055721984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104986468835328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107491305742006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40533520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚110106951245824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17470523 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚6099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64433086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110499712243200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102136353041152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858084139768298e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104966608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110499872008192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104738963791872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64433086 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107491202899410 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 _step‚486 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64423416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104738946752512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5354866897618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚110107055721984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104986468835328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107491305742006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41492555 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354952700854 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104966608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64423416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40533520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5513403172864 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 _runtime‚2692.268847474 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2æÎm–s2´- _wandbruntime‚2692  _runtime‚2692²M µþ .loss/FRt1/detection_loss‚2.519989375397563 /loss/FBt2/detection_loss‚1.1867184121058219 % _timestamp‚1.7858084139781494e+09 0loss/FBt1/assignment_loss‚0.4368147476401617 2loss/FBt2/assignment_loss‚0.006233672507925231 trainer/global_step‚6099 _runtime‚2692.269650709 /loss/FRt2/assignment_loss‚3.282792005687952 .loss/FRt2/detection_loss‚2.474450351049503 /loss/FBt1/detection_loss‚0.9229902267456055 _step‚487 0loss/FRt1/assignment_loss‚2.7586695186793806 2loss/SRqqt1/assignment_loss‚1.2112397830041048 1loss/SRqqt1/detection_loss‚1.0460331543686125 0loss/SRqqt2/detection_loss‚2.449278502695022 &loss/total_loss‚19.570273619094557 epoch‚0 2loss/SRqqt2/assignment_loss‚1.2750638692129044QÙL˜ ¶ƒ 0loss/FBt1/assignment_loss‚0.4368147476401617 2loss/FBt2/assignment_loss‚0.006233672507925231 trainer/global_step‚6099 _runtime‚2692.269650709 /loss/FRt2/assignment_loss‚3.282792005687952 .loss/FRt2/detection_loss‚2.474450351049503 /loss/FBt1/detection_loss‚0.9229902267456055 _step‚487 0loss/FRt1/assignment_loss‚2.7586695186793806 2loss/SRqqt1/assignment_loss‚1.2112397830041048 1loss/SRqqt1/detection_loss‚1.0460331543686125 0loss/SRqqt2/detection_loss‚2.449278502695022 &loss/total_loss‚19.570273619094557 epoch‚0 2loss/SRqqt2/assignment_loss‚1.2750638692129044 .loss/FRt1/detection_loss‚2.519989375397563 /loss/FBt2/detection_loss‚1.1867184121058219 % _timestamp‚1.7858084139781494e+09çµ4uœ·‚(:‘ ¥ŒÅÓ‰ŽÍÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.13277777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 nÅÈq¸‚(:æ ¥ŒÅÓê´¤× disk.drbd1743.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚83.40232011026428! disk.nvme0c0n1.out‚ 10350.3125 disk.rbd1.in‚0 disk.rbd0.out‚0% memory_percent‚27.672719955444336 proc.cpu.threads‚29 network.recv‚366323 disk.nvme0c0n1.in‚ 32.0703125 disk.sda3.out‚ 131.40234375 disk.nvme0n1.in‚ 32.0703125 disk.sda1.in‚0 disk.rbd5.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.sda.in‚0 disk.rbd2.out‚ 1.2265625 disk.sda2.in‚0 disk.dm-0.out‚ 10350.3125 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 32.0703125 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd5.out‚ 1.2265625' proc.memory.availableMB‚ 47400.40625) proc.memory.percent‚17.19306707382202 cpu‚13.939609764677526 disk.sda3.in‚0 disk.drbd1743.out‚3.046875# disk./.usageGB‚1242.75484085083 disk.rbd4.out‚ 1.2265625 disk.dm-1.out‚ 131.40234375 disk.nvme0n1.out‚ 10350.3125 network.sent‚ 5.680601e+06 disk.sda.out‚ 131.40234375 disk.dm-1.in‚0 disk.rbd1.out‚ 1.2265625Åö2¹- _wandbruntime‚2712  _runtime‚2712`pǺÁ & lr-AdamW/pg1‚0.0001584731746981472 & lr-AdamW/pg2‚0.0001584731746981472 trainer/global_step‚6149 % _timestamp‚1.7858084337770333e+09 _runtime‚2712.068850686 _step‚488&S»Ì»Æ & lr-AdamW/pg1‚0.0001584731746981472 & lr-AdamW/pg2‚0.0001584731746981472 trainer/global_step‚6149 % _timestamp‚1.7858084337770333e+09 _runtime‚2712.068850686 _step‚488èY°)2¼- _wandbruntime‚2712  _runtime‚2712Á0.VÒ'½ÌO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5397963691028 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5411121009152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚6149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5397877887904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105580118139904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40859029 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64940895 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚489 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64940895 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41826395 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24214945 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40859027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17611701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17611450 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105580135179264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105829592654848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5411208438784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5411208438784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111387301075968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102956621484832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111387139206656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110991239149056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102956604445472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5411121009152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41826648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5557683517440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105799922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108354585175860 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108354482333376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40859029 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105580118139904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64950551 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24214947 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110991239149056 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64950551 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5557546551808ÔÊ؉¾ÑO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102956604445472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5411121009152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41826648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5557683517440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105799922 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108354585175860 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1083e÷N#54482333376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40859029 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105580118139904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64950551 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24214947 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110991239149056 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64950551 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5557546551808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5397963691028 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5411121009152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚6149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5397877887904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105580118139904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40859029 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64940895 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 _step‚489 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64940895 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41826395 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24214945 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40859027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17611701 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17611450 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105580135179264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105829592654848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5411208438784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5411208438784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111387301075968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102956621484832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚111387139206656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110991239149056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291é-ú)c2¿- _wandbruntime‚2712  _runtime‚2712b’7‰Û&ÀÕM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40865670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105597284902912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111009285531648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105826760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17614566 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚490 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111009390007808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105597284902912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108372099978918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5558585624064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40865672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858084341148174e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105846824675328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64961088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64951418 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5398755361126 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108372202821514 trainer/global_step‚6149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111405410299392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105817088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5412000628736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24218882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17614314 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105826760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111405246340096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64961088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102973344617792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5412088065536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41833196 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105597301942272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41833450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24218884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105597301942272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5412000628736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105817088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5558446568448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64951418 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105597284902912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108372099978918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5558585624064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40865672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858084341148174e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105846824675328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64961088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64951418 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5398755361126 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108372202821514 trainer/global_step‚6149 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111405410299392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105817088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5412000628736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24218882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17614314 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105826760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111405246340096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64961088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102973344617792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5412088065536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41833196 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105597301942272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41833450 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24218884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105597301942272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5412000628736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105817088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5558446568448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64951418 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105597284902912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111009285531648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105826760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17614566 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚490 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ê^¸`2Â- _wandbruntime‚2712  _runtime‚2712[Úð‡Ãÿ _runtime‚2712.407634606 /loss/FRt2/detection_loss‚2.5678522381289253 0loss/SRqqt1/detection_loss‚1.184255581841405 /loss/FBt2/detection_loss‚0.9323686733964438 epoch‚0 _step‚491 0loss/FBt1/assignment_loss‚0.3814740583018437 2loss/FBt2/assignment_loss‚0.025815038622524388 .loss/FRt1/detection_loss‚2.522980638421499 % _timestamp‚1.7858084341161227e+09 0loss/FRt1/assignment_loss‚2.8314531995699954 0loss/FRt2/assignment_loss‚3.4948206249324754 2loss/SRqqt2/assignment_loss‚1.5593549359229304 .loss/FBt1/detection_loss‚2.290393312772115 &loss/total_loss‚21.497150754172388 2loss/SRqqt1/assignment_loss‚1.3215938222727606 1loss/SRqqt2/detection_loss‚2.3847886299894703 trainer/global_step‚6149dç¡# Ä„ 0loss/FBt1/assignment_loss‚0.3814740583018437 2loss/FBt2/assignment_loss‚0.025815038622524388 .loss/FRt1/detection_loss‚2.522980638421499 % _timestamp‚1.7858084341161227e+09 0loss/FRt1/assignment_loss‚2.8314531995699954 0loss/FRt2/assignment_loss‚3.4948206249324754 2loss/SRqqt2/assignment_loss‚1.5593549359229304 .loss/FBt1/detection_loss‚2.290393312772115 &loss/total_loss‚21.497150754172388 2loss/SRqqt1/assignment_loss‚1.3215938222727606 1loss/SRqqt2/detection_loss‚2.3847886299894703 trainer/global_step‚6149 _runtime‚2712.407634606 /loss/FRt2/detection_loss‚2.5678522381289253 0loss/SRqqt1/detection_loss‚1.184255581841405 /loss/FBt2/detection_loss‚0.9323686733964438 epoch‚0 _step‚491ëweb›Å‚(: ´ŒÅÓ‚˜÷Ò gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚86.3338888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kŽ8%rÆ‚(:ç ´ŒÅÓˆ¨çÕ$ memory_percent‚27.45656967163086 disk.sda1.in‚0 disk.rbd4.out‚ 1.2265625' disk.nvme0c0n1.out‚10351.8916015625 disk.rbd3.in‚0 disk.nvme0n1.in‚32.25% disk./.usageGB‚1242.7548751831055 disk.rbd2.out‚ 1.2265625 disk.dm-1.out‚ 132.02734375 disk.dm-0.in‚32.25) proc.memory.percent‚17.19306707382202( disk./.usagePercent‚83.4023224143321 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 132.02734375 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 network.recv‚367984 disk.rbd5.in‚0 disk.sda2.out‚0" disk.dm-0.out‚10351.8916015625 cpu‚13.9402579264075 network.sent‚ 5.725352e+06 disk.sda1.out‚0 disk.rbd5.out‚ 1.2265625 disk.sda.out‚ 132.02734375 disk.rbd1.out‚ 1.2265625 disk.drbd1743.out‚ 3.0546875% disk.nvme0n1.out‚10351.8916015625# proc.memory.rssMB‚ 11267.6484375 disk.rbd0.out‚0 disk.drbd1743.in‚0& proc.memory.availableMB‚ 47542.0625 proc.cpu.threads‚29 disk.nvme0c0n1.in‚32.25 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0ð¦›Ç‚(: ÃŒÅÓ±êÞÒ gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.94' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.18888888888888 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i`ØÈ‚(:ü ÃŒÅÓ¤Ö¬Ö disk.sda3.out‚ 132.80859375 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 32.40234375% memory_percent‚27.45625Î+T&ø9727478027 network.sent‚ 5.770106e+06) disk./.usagePercent‚83.40233009455812% disk./.usageGB‚1242.7549896240234 disk.rbd5.out‚ 1.2265625 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.40234375 disk.rbd3.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚10352.19091796875 disk.rbd1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 47542.265625 disk.dm-1.out‚ 132.80859375 disk.drbd1743.out‚ 3.07421875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚10352.19091796875 disk.sda.out‚ 132.80859375) proc.memory.percent‚17.19306707382202 disk.rbd5.in‚0 disk.rbd4.out‚ 1.2265625 disk.sda.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚1.3125 disk.sda3.in‚0 disk.dm-0.in‚ 32.40234375# disk.dm-0.out‚10352.19091796875 network.recv‚370213 disk.dm-1.in‚0 disk.rbd1.out‚1.3125# proc.memory.rssMB‚ 11267.6484375 cpu‚13.94087870234084 proc.cpu.threads‚29Ën4]2É- _wandbruntime‚2732  _runtime‚2732Fל ÇÊÁ & lr-AdamW/pg1‚0.0001597617840224125 & lr-AdamW/pg2‚0.0001597617840224125 trainer/global_step‚6199 % _timestamp‚1.7858084546774557e+09 _runtime‚2732.969120915 _step‚492º'øîÌËÆ _runtime‚2732.969120915 _step‚492 & lr-AdamW/pg1‚0.0001597617840224125 & lr-AdamW/pg2‚0.0001597617840224125 trainer/global_step‚6199 % _timestamp‚1.7858084546774557e+09ìW÷Ÿ‚2Ì- _wandbruntime‚2732  _runtime‚2732~¤JÒ'ÍÌO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106438473329664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42166039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112292807195648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41191179 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24411795 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5455204574208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109235482255368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103793613061472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41191177 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5602705220096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24411797 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106660074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106689948494848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111893677903872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106669732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5455117144576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5441852154536 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65478553 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65478553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚493 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41191177 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17753979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858084546784086e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106689973398528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5602833797120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103793630100832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚6199 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65468897 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111893573434880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106438473329664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41191179 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17754242 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109235379412884 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111893677903872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42165774 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5441766351412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5455117144576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111893573434880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _runtime‚2732.973929389 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5455204574208 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111893677903872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42165774 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5441766351412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5455117144576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111893573434880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _runtime‚2732.973929389 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5455204574208 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103793613061472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41191177 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5602705220096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24411797 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106660074 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106689948494848 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111893677903872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106669732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5455117144576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5441852154536 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65478553 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65478553 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚493 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41191177 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17753979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858084546784086e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106689973398528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5602833797120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103793630100832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 trainer/global_step‚6199 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65468897 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111893573434880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106438473329664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41191179 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17754242 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109235379412884 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5442643824634 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41197820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65479420 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858084550191886e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106677240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106677240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41197820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106455640092672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24415734 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112310759209984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103810353233792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106707155611648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚6199 _runtime‚2733.311302078 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5455996764160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42172422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17756690 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5456084200960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109252997058426 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5456084200960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42172664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106686912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17756930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106686912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65479420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5603603598336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111911724293632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41197822 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65489090 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106455640092672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24415732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ò:½ÑÛM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41197820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65479420 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 % _timestamp‚1.7858084550191886e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106677240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106677240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41197820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚106455640092672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24415734 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112310759209984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103810353233792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106707155611648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚6199 _runtime‚2733.311302078 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5455996764160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42172422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17756690 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5456084200960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109252997058426 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5456084200960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42172664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106686912 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17756930 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106686912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65479420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5603603598336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111911724293632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41197822 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65489090 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106455640092672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24415732 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65489090 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5442729627870 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112310918974976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5603738459648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103810370273152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 _step‚494 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41197822 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5442643824634 E›Õ‚(: ÒŒÅÓçßâÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚151.44' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.13333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BƒQ{Ö‚(:ð ÒŒÅÓ¼‚»Õ disk.sda2.in‚0% memory_percent‚27.456295490264893) proc.memory.availableMB‚ 47542.2421875 cpu‚13.941199467242388 disk.rbd0.in‚0% disk./.usageGB‚1242.7549324035645 disk.rbd3.in‚0 disk.nvme0n1.in‚ 32.40234375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.40234375 disk.sda.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.19306707382202 network.sent‚ 5.772581e+06 network.recv‚371159 disk.drbd1743.in‚0 disk.rbd5.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.rbd3.out‚0 disk.rbd4.out‚ 1.2265625 disk.sda3.in‚0 disk.sda1.in‚0) disk./.usagePercent‚83.40232625444511 disk.drbd1743.out‚3.09375 disk.dm-0.in‚ 32.40234375! disk.dm-0.out‚10352.841796875$ disk.nvme0n1.out‚10352.841796875 disk.rbd5.out‚ 1.2265625 disk.rbd4.in‚0& disk.nvme0c0n1.out‚10352.841796875 disk.dm-1.out‚ 133.2578125 disk.rbd1.out‚1.375 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.out‚ 133.2578125 disk.rbd2.out‚1.375 disk.sda3.out‚ 133.2578125 disk.dm-1.in‚0 proc.cpu.threads‚29“9Q2×- _wandbruntime‚2752  _runtime‚2752~KÛÇØÁ trainer/global_step‚6249 % _timestamp‚1.7858084746494853e+09 _runtime‚2752.941145908 _step‚496 & lr-AdamW/pg1‚0.0001610503933466778 & lr-AdamW/pg2‚0.0001610503933466778sëÁ²ÌÙÆ _runtime‚2752.941145908 _step‚496 & lr-AdamW/pg1‚0.0001610503933466778 & lr-AdamW/pg2‚0.0001610503933466778 trainer/global_step‚6249 % _timestamp‚1.7858084746494853e+09ð‚gu/2Ú- _wandbruntime‚2752  _runtime‚2752ÛècåÒ'ÛÌO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚2752.945906752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112796012189696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5499113280000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107296811480064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107296794440704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5647839816192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5647976781824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65996899 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42505467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104630638716832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41523327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110116276492392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚497 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 % _timestamp‚1.7858084746504445e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5499200709632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41523329 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41523329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112795907720704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107296811480064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5499113280000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5485740618044 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113198306020352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41523327 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107296794440704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5499200709632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107520226 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17896577 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚112795907720704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66006555 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42505222 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112796012189696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5485654814920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24608645 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107529884 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24608647 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107550304334848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104630621677472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66006555 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65996899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17896820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110116379334876 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107550329238528 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110116379334876 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107550329238528 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107296794440704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5647839816192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5647976781824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65996899 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42505467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104630638716832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41523327 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110116276492392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚497 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 % _timestamp‚1.7858084746504445e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5499200709632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41523329 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41523329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112795907720704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107296811480064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5499113280000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5485740618044 QBDeviceüÈ9‰StatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113198306020352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392ñ©­'2Ý- _wandbruntime‚2753  _runtime‚2753‰µÜ&ÞÖM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41529972 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107313961203712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42512016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110133996980530 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5500080336384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5486618091378 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107537392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17899700 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647378889152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66007422 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66017092 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5499992899584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5499992899584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113216415342080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42512284 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5500080336384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107547064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66007422 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112814058579456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41529970 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110133894137934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112813954103296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5648878986752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107313978243072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚498 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107537392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5486532288142 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647361849792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107547064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66007422 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112814058579456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41529970 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110133894137934 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112813954103296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5648878986752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107313978243072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚498 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107537392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5486532288142 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104647361849792 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41529972 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107313961203712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42512016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110133996980530 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5500080336384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5486618091378 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107537392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17899700 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104647378889152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66007422 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66017092 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5499992899584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5499992899584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113216415342080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42512284 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0òí;‚+2à- _wandbruntime‚2753  _runtime‚2753É‹á¯á€ 2loss/SRqqt1/assignment_loss‚1.1446780979384592 1loss/SRqqt1/detection_loss‚0.9427196129184702 /loss/FBt1/detection_loss‚1.2489754768709342 epoch‚0 1loss/FBt1/assignment_loss‚0.12443393576415929 2loss/FBt2/assignment_loss‚0.009544540994467252 /loss/FRt1/detection_loss‚2.3667751039529956 /loss/FBt2/detection_loss‚1.0167334549395102 %loss/total_loss‚18.80265217875821 _runtime‚2753.285195737 0loss/FRt1/assignment_loss‚2.5931493854799936 1loss/SRqqt2/detection_loss‚2.1423013280286933 % _timestamp‚1.7858084749937222e+09 /loss/FRt2/assignment_loss‚3.224176212326511 1loss/SRqqt2/assignment_loss‚1.517269978017518 /loss/FRt2/detection_loss‚2.4718950515264995 trainer/global_step‚6249 _step‚4997¨ â… 2loss/FBt2/assignment_loss‚0.009544540994467252 /loss/FRt1/detection_loss‚2.3667751039529956 /loss/FBt2/detection_loss‚1.0167334549395102 %loss/total_loss‚18.80265217875821 _runtime‚2753.285195737 0loss/FRt1/assignment_loss‚2.5931493854799936 1loss/SRqqt2/detection_loss‚2.1423013280286933 % _timestamp‚1.7858084749937222e+09 /loss/FRt2/assignment_loss‚3.224176212326511 1loss/SRqqt2/assignment_loss‚1.517269978017518 /loss/FRt2/detection_loss‚2.4718950515264995 trainer/global_step‚6249 _step‚499 2loss/SRqqt1/assignment_loss‚1.1446780979384592 1loss/SRqqt1/detection_loss‚0.9427196129184702 /loss/FBt1/detection_loss‚1.2489754768709342 epoch‚0 1loss/FBt1/assignment_loss‚0.12443393576415929óûŠj%œã‚(:‘ áŒÅÓœÔôÒ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚151.615' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.23055555555557 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0înðtä‚(:é áŒÅÓíöÕ network.sent‚ 5.817537e+06 network.recv‚373478 disk.rbd5.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.rbd2.in‚0 disk.dm-1.out‚ 134.078125 disk.rbd1.in‚0% memory_percent‚27.456378936767578 cpu‚13.94150340243402% disk./.usageGB‚1243.8793411254883 disk.dm-0.in‚ 32.40234375 disk.rbd4.in‚0& disk.nvme0c0n1.out‚10353.095703125 disk.dm-1.in‚0 disk.rbd1.out‚1.375& proc.memory.availableMB‚ 47542.1875 proc.cpu.threads‚29 disk.sda3.out‚ 134.078125) disk./.usagePercent‚83.47778626721517 disk.drbd1743.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.19306707382202! disk.dm-0.out‚10353.095703125 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.40234375 disk.sda3.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 32.40234375$ disk.nvme0n1.out‚10353.095703125 disk.rbd4.out‚ 1.2265625 disk.rbd5.out‚ 1.2265625 disk.sda.in‚0 disk.rbd2.out‚1.375 disk.sda1.out‚0 disk.drbd1743.out‚3.09375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda.out‚ 134.078125 disk.sda2.out‚0ŽÁÛ"2å- _wandbruntime‚2773  _runtime‚2773Q¯ÉÆæÀ trainer/global_step‚6299 $ _timestamp‚1.785808495009334e+09 _runtime‚2773.302967931 _step‚500 & lr-AdamW/pg1‚0.0001623390026709431 & lr-AdamW/pg2‚0.0001623390026709431æ*¡ËçÅ $ _timestamp‚1.785808495009334e+09 _runtime‚2773.302967931 _step‚500 & lr-AdamW/pg1‚0.0001623390026709431 & lr-AdamW/pg2‚0.0001623390026709431 trainer/global_step‚6299ô 5—\2è- _wandbruntime‚2773  _runtime‚2773=ÝU!éÌO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5543196845056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108410685078528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚108155149630464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114103805721600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105467647332832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108410660174848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 trainer/global_step‚6299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66534557 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113698346475520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5543109415424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108155132591104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41855479 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42846255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108155132591104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110997173571900 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41855479 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113698242006528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚501 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108380378 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108155149630464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18040758 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66534557 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18040506 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5529543278428 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41855477 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113698346475520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5692989968896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110997276414384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108390036 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113698242006528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114103650143744 % _timestamp‚1.7858084950100253e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚2773.303057789 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5693120643072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_í Ìg}bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5543109415424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5529629081552 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24805495 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113698242006528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114103650143744 % _timestamp‚1.7858084950100253e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚2773.303057789 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5693120643072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5543109415424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5529629081552 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24805495 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113698346475520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5543109415424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108155132591104 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41855479 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42846255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108155132591104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110997173571900 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41855479 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113698242006528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚501 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108380378 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108155149630464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18040758 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66534557 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18040506 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5529543278428 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41855477 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113698346475520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5692989968896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110997276414384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392õxB2ë- _wandbruntime‚2773  _runtime‚2773¸Ä_Ü&ìÖM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111014791217442 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41862120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5693893377024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105484370465792 _runtime‚2773.646850821 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42852996 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18043562 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108407216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24809434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5530420751650 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108407216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚6299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108172299354112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18043295 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105484387505152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24809432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108427892195328 _step‚502 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108172299354112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108172316393472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113716392865280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5544076471808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5543989035008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5543989035008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7858084953549025e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5544076471808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42852727 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41862120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680v{Zá&íÛM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7858084953549025e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5544076471808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42852727 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41862120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111014791217442 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41862120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5693893377024 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105484370465792 _runtime‚2773.646850821 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42852996 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18043562 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108407216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24809434 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5530420751650 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108407216 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚6299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108172299354112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18043295 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105484387505152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24809432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108427892195328 _step‚502 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108172299354112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚108172316393472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113716392865280 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5544076471808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5543989035008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5543989035008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0öäÛ¥É2î- _wandbruntime‚2773  _runtime‚2773ž›àï€ 2loss/FBt2/assignment_loss‚0.028118183315252784 .loss/FRt2/detection_loss‚2.698797476134802 epoch‚0 0loss/FRt2/assignment_loss‚3.3540792276984766 /loss/FRt1/detection_loss‚2.6853261702344753 1loss/SRqqt2/detection_loss‚2.3122311986982824 /loss/FBt1/detection_loss‚1.6667795777320862 /loss/FBt2/detection_loss‚1.4759727887498835 &loss/total_loss‚21.101120509986274 0loss/FRt1/assignment_loss‚2.7979863211512566 1loss/SRqqt1/assignment_loss‚1.283723943236777 2loss/SRqqt2/assignment_loss‚1.2590671434998513 0loss/FBt1/assignment_loss‚0.3939384996891008 1loss/SRqqt1/detection_loss‚1.1450999798460275 trainer/global_step‚6299 _runtime‚2773.647614923 _step‚503 % _timestamp‚1.7858084953561614e+09QçYïMð… 0loss/FRt1/assignment_loss‚2.7979863211512566 1loss/SRqqt1/assigØžVã¾nment_loss‚1.283723943236777 2loss/SRqqt2/assignment_loss‚1.2590671434998513 0loss/FBt1/assignment_loss‚0.3939384996891008 1loss/SRqqt1/detection_loss‚1.1450999798460275 trainer/global_step‚6299 _runtime‚2773.647614923 _step‚503 % _timestamp‚1.7858084953561614e+09 2loss/FBt2/assignment_loss‚0.028118183315252784 .loss/FRt2/detection_loss‚2.698797476134802 epoch‚0 0loss/FRt2/assignment_loss‚3.3540792276984766 /loss/FRt1/detection_loss‚2.6853261702344753 1loss/SRqqt2/detection_loss‚2.3122311986982824 /loss/FBt1/detection_loss‚1.6667795777320862 /loss/FBt2/detection_loss‚1.4759727887498835 &loss/total_loss‚21.101120509986274÷·cœñ‚(:‘ ðŒÅÓüÓ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚155.758' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.53222222222223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0szd†ò‚(:û ðŒÅÓÙõØ) disk./.usagePercent‚83.47753128371107 disk.drbd1743.out‚ 3.12109375$ disk.nvme0n1.out‚11500.111328125 disk.sda1.out‚0% memory_percent‚27.455997467041016& proc.memory.availableMB‚ 47542.4375 proc.cpu.threads‚29 disk.sda1.in‚0% disk./.usageGB‚1243.8755416870117 disk.rbd2.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 1.2265625 network.sent‚ 5.862162e+06& disk.nvme0c0n1.out‚11500.111328125 disk.sda.out‚ 134.6484375 disk.sda2.out‚0 disk.nvme0n1.in‚ 32.40234375# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202! disk.nvme0c0n1.in‚ 32.40234375 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd4.in‚0 cpu‚13.941829954289316 disk.rbd1.out‚ 1.4072265625 disk.rbd5.in‚0 disk.rbd2.out‚ 1.4072265625 disk.rbd1.in‚0 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.40234375! disk.dm-0.out‚11500.111328125 disk.rbd0.in‚0 network.recv‚375092 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 134.6484375 disk.dm-1.out‚ 134.6484375 disk.rbd5.out‚1.3125‡´Åœó‚(:‘ ÿŒÅÓÉÞÒÒ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚154.691' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.93944444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-È°2Šô‚(:ÿ ÿŒÅÓÓÛÀÕ disk.sda1.in‚0 disk.rbd4.in‚0& disk.nvme0c0n1.out‚11500.416015625 disk.sda.in‚0 disk.rbd2.out‚ 1.4072265625) proc.memory.percent‚17.19306707382202 disk.rbd1.out‚ 1.4072265625 disk.drbd1743.in‚0 disk.drbd1743.out‚3.140625! disk.nvme0c0n1.in‚ 32.40234375% disk./.usageGB‚1242.7507934570312 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd5.out‚1.375 disk.dm-1.out‚ 135.47265625 network.sent‚ 5.906871e+06 network.recv‚376882 disk.rbd3.in‚0$ disk.nvme0n1.out‚11500.416015625 disk.dm-1.in‚0 disk.dm-0.in‚ 32.40234375 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 135.47265625# proc.memory.rssMB‚ 11267.6484375 cpu‚13.942677667579618) disk./.usagePercent‚83.40204848627046 disk.nvme0n1.in‚ 32.40234375 disk.rbd0.out‚0 disk.rbd4.out‚ 1.2265625 disk.sda3.in‚0! disk.dm-0.out‚11500.416015625 disk.rbd2.in‚0* proc.memory.availableMB‚47542.61328125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 135.47265625% memory_percent‚27.455729246139526 proc.cpu.threads‚29ýˆ2õ- _wandbruntime‚2794  _runtime‚2794žØÙÅö¿ _step‚504 & lr-AdamW/pg1‚0.0001636276119952084 & lr-AdamW/pg2‚0.0001636276119952084 trainer/global_step‚6349 # _timestamp‚1.78580851576529e+09 _runtime‚2794.057087041û“—‚Ê÷Ä & lr-AdamW/pg1‚0.0001636276119952084 & lr-AdamW/pg2‚0.0001636276119952084 trainer/global_step‚6349 # _timestamp‚1.78580851576529e+09 _runtime‚2794.057087041 _step‚504øûDö2ø- _wandbruntime‚2794  _runtime‚2794'œÑ'ùËO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25002345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67052903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5573517545060 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42187629 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43184351 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109271016014848 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114600576292352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67062559 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5587192980480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111878070651408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5587192980480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67052903 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42187629 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109013470741504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18182242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚6349 _step‚505 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5573431741936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887™Ä¹áÖ'úÐO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42187629 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109013470741504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18182242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚6349 _step‚505 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5573431741936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25002345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67052903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5573517545060 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42187629 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43184351 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109271016014848 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114600576292352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67062559 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5587192980480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111878070651408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5587192980480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67052903 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6349 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5588072607232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115027328308224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111895688296950 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109030654543872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7858085161181612e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109288248035328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚506 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109267368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109030637504512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42194272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5574309215158 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109257696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5587985170432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42194270 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5587985170432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5739105176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109257696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25006284 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111895791139546 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106321396121152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42194272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5574395018394 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67073096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18185088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_trÇÇ4N ain_batch_end/allocated_bytes.all.allocated‚114618727151104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67063426 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43191119 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5739237940736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5588072607232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109267368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43191372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚2794.410099008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840`:§á&ýÛM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109257696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25006284 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111895791139546 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106321396121152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42194272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5574395018394 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67073096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18185088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114618727151104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67063426 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43191119 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5739237940736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5588072607232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109267368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43191372 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚2794.410099008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6349 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5588072607232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115027328308224 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111895688296950 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109030654543872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7858085161181612e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109288248035328 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚506 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109267368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109030637504512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42194272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5574309215158 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109257696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5587985170432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42194270 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5587985170432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5739105176576ú—vc2þ- _wandbruntime‚2794  _runtime‚2794j2eÿþ .loss/FRt2/detection_loss‚2.697915553230129 1loss/SRqqt2/detection_loss‚2.0544698401575996 /loss/FBt2/detection_loss‚1.0802128390542098 trainer/global_step‚6349 $ _timestamp‚1.785808516119986e+09 _runtime‚2794.411496358 0loss/FRt1/assignment_loss‚2.7833099590407477 0loss/FRt2/assignment_loss‚3.1795476038162023 /loss/FRt1/detection_loss‚2.2485964820616777 1loss/SRqqt1/detection_loss‚1.0793643771088681 _step‚507 1loss/FBt1/assignment_loss‚0.16626119456395827 &loss/total_loss‚19.026287866092723 epoch‚0 1loss/SRqqt1/assignment_loss‚1.068203412659932 1loss/SRqqt2/assignment_loss‚1.251756446081258 1loss/FBt2/assignment_loss‚0.09372102931366441 /loss/FBt1/detection_loss‚1.32292912900447851ÿDÜ €ƒ epoch‚0 1loss/SRqqt1/assignment_loss‚1.068203412659932 1loss/SRqqt2/assignment_loss‚1.251756446081258 1loss/FBt2/assignment_loss‚0.09372102931366441 /loss/FBt1/detection_loss‚1.3229291290044785 .loss/FRt2/detection_loss‚2.697915553230129 1loss/SRqqt2/detection_loss‚2.0544698401575996 /loss/FBt2/detection_loss‚1.0802128390542098 trainer/global_step‚6349 $ _timestamp‚1.785808516119986e+09 _runtime‚2794.411496358 0loss/FRt1/assignment_loss‚2.7833099590407477 0loss/FRt2/assignment_loss‚3.1795476038162023 /loss/FRt1/detection_loss‚2.2485964820616777 1loss/SRqqt1/detection_loss‚1.0793643771088681 _step‚507 1loss/FBt1/assignment_loss‚0.16626119456395827 &loss/total_loss‚19.026287866092723û Ôᛂ(: ŽÅÓ’ÌÑÒ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚150.016' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.34222222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š’I-‰‚‚(:þ ŽÅÓ͉ãÖ disk.rbd3.out‚0 disk.rbd5.out‚1.375) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29 network.recv‚377731 disk.drbd1743.in‚0 disk.rbd2.out‚ 1.47021484375 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd4.in‚0 disk.dm-1.in‚0% disk./.usageGB‚1242.7507019042969 disk.sda2.in‚0 disk.dm-0.in‚ 32.40234375 disk.nvme0n1.in‚ 32.40234375 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 32.40234375$ memory_percent‚27.45608687400818 disk.rbd3.in‚0$ disk.nvme0n1.out‚11501.822265625 disk.rbd0.in‚0 disk.rbd4.out‚1.3125 disk.rbd2.in‚0 disk.dm-1.out‚ 136.00390625* proc.memory.availableMB‚47542.37890625# proc.memory.rssMB‚ 11267.6484375! disk.dm-0.out‚11501.822265625 disk.sda.in‚0 disk.sda.out‚ 136.00390625 network.sent‚ 5.909348e+06 disk.rbd1.out‚ 1.47021484375 disk.rbd5.in‚0 disk.sda3.out‚ 136.00390625 cpu‚13.942745886586629 disk.rbd0.out‚0& disk.nvme0c0n1.out‚11501.822265625 disk.sda3.in‚0) disk./.usagePercent‚83.40204234208963 disk.drbd1743.out‚ 3.1484375;È»g2ƒ- _wandbruntime‚2814  _runtime‚2814™~+Ç„Á & lr-AdamW/pg2‚0.0001649162213194737 trainer/global_step‚6399 % _timestamp‚1.7858085362149081e+09 _runtime‚2814.506598222 _step‚508 & lr-AdamW/pg1‚0.0001649162213194737Q=:îÌ…Æ % _timestamp‚1.7858085362149081e+09 _runtime‚2814.506598222 _step‚508 & lr-AdamW/pg1‚0.0001649162213194737 & lr-AdamW/pg2‚0.0001649162213194737 trainer/global_step‚6399ü§Q¸2†- _wandbruntime‚2814  _runtime‚2814˜CyÒ'‡ÌO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25199195 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112758967730916 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42519779 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67580905 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115503015047168 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107141647525472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109871825931264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67580905 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112759070573400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110100682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43523704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115502910578176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115503015047168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42519777 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5631101686272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115914878564864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107141664564832 _step‚509 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110110340 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115914725084160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110131371854848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5783481806336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5617320205444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18324509 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5631189115904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110131396758528 trainer/global_step‚6399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67590561„’6ˆÑO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5631101686272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115914878564864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107141664564832 _step‚509 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110110340 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115914725084160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110131371854848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5783481806336 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5617320205444 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18324509 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5631189115904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110131396758528 trainer/global_step‚6399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67590561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25199195 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112758967730916 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42519779 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67580905 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115503015047168 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107141647525472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_poo Ó{H l.allocated‚109871825931264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67580905 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112759070573400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110100682 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43523704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115502910578176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115503015047168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42519777 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ýBý2‰- _wandbruntime‚2814  _runtime‚2814åµKÐÜ&ŠÖM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42526420 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115932992834560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚6399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42526420 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115932833069568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109888992694272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25203134 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112776688219054 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5631981305856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43530473 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67601098 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109888975654912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115521061436928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110117848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5618283481902 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18327341 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110148578971648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43530729 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110127520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858085365632844e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110127520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107158387697792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25203132 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚510 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18327595 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115521061436928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67591428 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596Ârá&‹ÛM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110127520 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858085365632844e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110127520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107158387697792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25203132 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _step‚510 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18327595 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115521061436928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67591428 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42526420 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115932992834560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚6399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42526420 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115932833069568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109888992694272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25203134 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112776688219054 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5631981305856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43530473 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67601098 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109888975654912 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115521061436928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110117848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5618283481902 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18327341 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110148578971648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43530729 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0þÞd6Ü2Œ- _wandbruntime‚2814  _runtime‚2814 Yÿ 0loss/FRt2/assignment_loss‚3.3243746908231713 .loss/FRt1/detection_loss‚2.402497108934856 &loss/total_loss‚20.799359759399728 % _timestamp‚1.7858085365647202e+09 _step‚511 0loss/FRt1/assignment_loss‚2.8945343785169646 /loss/FRt2/detection_loss‚2.5022153963987854 1loss/SRqqt1/detection_loss‚1.0393463253483541 epoch‚0 _runtime‚2814.856338464 2loss/SRqqt2/assignment_loss‚1.0001505025662483 0loss/FBt1/assignment_loss‚0.8610535778207122 2loss/FBt2/assignment_loss‚0.015834068788649347 /loss/FBt2/detection_loss‚1.3151602679281496 2loss/SRqqt1/assignment_loss‚1.0816345026865144 1loss/SRqqt2/detection_loss‚1.7826654176227748 -loss/FBt1/detection_loss‚2.57989352196455 trainer/global_step‚6399± òŸ Ž„ % _timestamp‚1.7858085365647202e+09 _step‚511 0loss/FRt1/assignment_loss‚2.8945343785169646 /loss/FRt2/detection_loss‚2.5022153963987854 1loss/SRqqt1/detection_loss‚1.0393463253483541 epoch‚0 _runtime‚2814.856338464 2loss/SRqqt2/assignment_loss‚1.0001505025662483 0loss/FBt1/assignment_loss‚0.8610535778207122 2loss/FBt2/assignment_loss‚0.015834068788649347 /loss/FBt2/detection_loss‚1.3151602679281496 2loss/SRqqt1/assignment_loss‚1.0816345026865144 1loss/SRqqt2/detection_loss‚1.7826654176227748 -loss/FBt1/detection_loss‚2.57989352196455 trainer/global_step‚6399 0loss/FRt2/assignment_loss‚3.3243746908231713 .loss/FRt1/detection_loss‚2.402497108934856 &loss/total_loss‚20.799359759399728ÿ÷ÁOXœ‚(:‘ ÅÓ¢šÞÒ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚150.083' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.37944444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 §ï…‰‚(:þ ÅÓﺱÖ) proc.memory.percent‚17.19306707382202 cpu‚13.943258030412869 disk.rbd0.out‚0 disk.rbd2.out‚ 1.47021484375 disk.sda2.in‚0 network.sent‚ 5.954248e+06% disk./.usageGB‚1242.5377960205078 disk.rbd5.in‚0 disk.drbd1743.out‚ 3.15234375 disk.sda1.out‚0 disk.sda.out‚ 136.796875 disk.dm-1.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚29 network.recv‚379837& disk.nvme0c0n1.out‚11502.439453125 disk.rbd1.out‚ 1.47021484375 disk.nvme0n1.in‚ 32.4140625# proc.memory.rssMB‚ 11267.6484375 disk.sda3.in‚0 disk.sda3.out‚ 136.796875 disk.dm-1.out‚ 136.796875 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.4140625 disk.rbd3.out‚0 disk.rbd4.out‚1.3125 disk.nvme0c0n1.in‚ 32.4140625 disk.rbd3.in‚0) disk./.usagePercent‚83.38775404958861 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0! disk.dm-0.out‚11502.439453125$ disk.nvme0n1.out‚11502.439453125% memory_percent‚27.456766366958618* proc.memory.availableMB‚47541.93359375 disk.rbd5.out‚ 1.4072265625 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0Þ/7ì2‘- _wandbruntime‚2834  _runtime‚2834{¤-sÉ’Ã _step‚512 ' lr-AdamW/pg1‚0.00016620483064373902 ' lr-AdamW/pg2‚0.00016620483064373902 trainer/global_step‚6449 % _timestamp‚1.7858085561603115e+09 _runtime‚2834.452030818ã¡ Î“È _runtime‚2834.452030818 _step‚512 ' lr-AdamW/pg1‚0.00016620483064373902 ' lr-AdamW/pg2‚0.00016620483064373902 trainer/global_step‚6449 % _timestamp‚1.7858085561603115e+09€B¶œ32”- _wandbruntime‚2834  _runtime‚2834ù^n"•ËO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚110730147042304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116405244864000 <Ã×%ËZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107978673180832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25396045 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚513 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68108907 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43864077 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116820242704896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42851927 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110991727694848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858085561613374e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18468262 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68108907 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110960834 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5828515010048 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5675185251328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25396047 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712WÛì'Ö'–ÐO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚110730147042304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116405244864000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107978673180832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25396045 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚513 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68108907 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43864077 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116820242704896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42851927 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110991727694848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858085561613374e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18468262 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68108907 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110960834 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5828515010048 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5675185251328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25396047 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚858031246Áy2—- _wandbruntime‚2834  _runtime‚2834kÏ4Ü&˜ÖM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110987672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110747313805312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42858570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110978000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116838365640704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113657585298562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18471144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18470895 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5662086142174 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43871128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110987672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107995396313792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43870877 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113657482455966 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68119430 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110978000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68129100 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68119430 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107995413353152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5662171945410 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68129100 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5676064878080 trainer/global_step‚6449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110747313805312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111008959715328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42858572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5676064878080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116423395722752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5675977441280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5675977441280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110747330844672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110747330844672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42858570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚514 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116423291246592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111008934811648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5829430829056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2834.790882287 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5829563593216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25399982 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116838523308544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42858572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251Ï:€G.™ÛM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚514 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116423291246592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111008934811648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5829430829056 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2834.790882287 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5829563593216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25399982 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116838523308544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42858572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110987672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚110747313805312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42858570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 110978000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116838365640704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113657585298562 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18471144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18470895 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5662086142174 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43871128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110987672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚107995396313792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43870877 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113657482455966 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68119430 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110978000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68129100 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68119430 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107995413353152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5662171945410 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68129100 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5676064878080 trainer/global_step‚6449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.sma¡Ãœ³ll_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110747313805312 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111008959715328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42858572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5676064878080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116423395722752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5675977441280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5675977441280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110747330844672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚110747330844672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42858570‚´5I2š- _wandbruntime‚2834  _runtime‚2834eá>›‚ /loss/FBt1/detection_loss‚3.3540551979094744 epoch‚0 2loss/FBt1/assignment_loss‚0.026450454045402694 /loss/FBt2/detection_loss‚1.3440609566494823 trainer/global_step‚6449 $ _timestamp‚1.785808556500209e+09 _runtime‚2834.791647492 _step‚515 0loss/FRt2/assignment_loss‚2.8061983820832808 2loss/SRqqt2/assignment_loss‚1.5711680233478547 2loss/FBt2/assignment_loss‚0.009111716635435982 &loss/total_loss‚21.250217879653896 0loss/FRt1/assignment_loss‚3.0541067314999446 2loss/SRqqt1/assignment_loss‚1.1288788974044783 /loss/FRt1/detection_loss‚2.4941405905854133 .loss/FRt2/detection_loss‚2.608139466356348 1loss/SRqqt1/detection_loss‚1.0407929254034836 1loss/SRqqt2/detection_loss‚1.8131145377332964}éS¸ œ‡ 0loss/FRt2/assignment_loss‚2.8061983820832808 2loss/SRqqt2/assignment_loss‚1.5711680233478547 2loss/FBt2/assignment_loss‚0.009111716635435982 &loss/total_loss‚21.250217879653896 0loss/FRt1/assignment_loss‚3.0541067314999446 2loss/SRqqt1/assignment_loss‚1.1288788974044783 /loss/FRt1/detection_loss‚2.4941405905854133 .loss/FRt2/detection_loss‚2.608139466356348 1loss/SRqqt1/detection_loss‚1.0407929254034836 1loss/SRqqt2/detection_loss‚1.8131145377332964 /loss/FBt1/detection_loss‚3.3540551979094744 epoch‚0 2loss/FBt1/assignment_loss‚0.026450454045402694 /loss/FBt2/detection_loss‚1.3440609566494823 trainer/global_step‚6449 $ _timestamp‚1.785808556500209e+09 _runtime‚2834.791647492 _step‚515ƒ‹èì>‘‚(:† ¬ÅÓ°ËâÒ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚155.691' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚86.495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰é¨)„ž‚(:ù ¬ÅÓÌÝÖ) proc.memory.availableMB‚ 47542.2421875 network.recv‚381647% disk./.usageGB‚1242.5374374389648 disk.dm-1.out‚ 137.59375 disk.sda2.out‚0 disk.rbd3.out‚0 network.sent‚ 5.999225e+06 disk.rbd5.in‚0( disk.nvme0c0n1.out‚11503.04150390625& disk.nvme0n1.out‚11503.04150390625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 137.59375 disk.rbd1.out‚ 1.533203125% memory_percent‚27.456295490264893( disk./.usagePercent‚83.3877299848804 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.drbd1743.out‚ 3.1796875) proc.memory.percent‚17.19306707382202 disk.rbd5.out‚ 1.4072265625 disk.sda.out‚ 137.59375 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.4140625# disk.dm-0.out‚11503.04150390625 cpu‚13.943544737557165 disk.nvme0n1.in‚ 32.4140625 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 32.4140625 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚29 disk.sda1.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.rbd4.out‚1.375 disk.rbd2.out‚ 1.533203125 disk.rbd1.in‚0 disk.rbd3.in‚00$œŸ‚(:‘ »ÅÓ›»ÙÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚157.056' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚87.25333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¯‘ ‚(:†  »ÅÓÛð¥Ö disk.sda2.in‚0 disk.drbd1743.in‚0# disk.dm-0.out‚11565.78759765625 disk.sda1.in‚0 disk.rbd4.out‚1.375 disk.nvme0c0n1.in‚ 32.4140625 disk.sda3.out‚ 138.23046875) proc.memory.percent‚17.19306707382202 disk.dm-0.in‚ 32.4140625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 network.sent‚ 6.043932e+06) disk./.usagePercent‚83.39453773723092 disk.sda2.out‚0 disk.rbd5.out‚ 1.47021484375 disk.dm-1.out‚ 138.23046875% memory_percent‚27.456670999526978 disk.drbd1743.out‚ 3.19921875 disk.nvme0n1.in‚ 32.4140625& disk.nvme0n1.out‚11565.78759765625 disk.rbd5.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 1.533203125( disk.nvme0c0n1.out‚11565.78759765625 disk.rbd2.in‚0 disk.rbd3.out‚0 network.recv‚383693 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd4.in‚0 disk.sda.out‚ 138.23046875 disk.rbd2.out‚ 1.533203125% disk./.usageGB‚1242.6388778686523# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29* proc.memory.availableMB‚47541.99609375 cpu‚13.944090576680342Xïá>2¡- _wandbruntime‚2854  _runtime‚2854é\ƒÉ¢Ã ' lr-AdamW/pg2‚0.00016749343996800433 trainer/global_step‚6499 % _timestamp‚1.7858085761972759e+09 _runtime‚2854.490915355 _step‚516 ' lr-AdamW/pg1‚0.00016749343996800433è‘*ÖÎ£È trainer/global_step‚6499 % _timestamp‚1.7858085761972759e+09 _runtime‚2854.490915355 _step‚516 ' lr-AdamW/pg1‚0.00016749343996800433 ' lr-AdamW/pg2‚0.00016749343996800433„ÄvJá2¤- _wandbruntime‚2854  _runtime‚2854 ¡uÂÑ'¥ËO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5719181386752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5719093957120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117725965246464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111852108438528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108815681796832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68636909 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44204249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68636909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44204514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43184077 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114520761889932 trainer/global_step‚6499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111588502232064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111588502232064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43184079 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68646565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117307683618816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5873717745152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108815664757472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5719093957120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111830644 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5873856807936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117307579149824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114520864732416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚517 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117725801280000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111588485192704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68646565 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43184077 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25592897 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5705182935584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25592895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18611617 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117307579149824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43184079 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251[n/Ö'¦ÐO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5873717745152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108815664757472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5719093957120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111830644 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5873856807936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117307579149824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114520864732416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚517 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117725801280000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111588485192704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68646565 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43184077 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25592897 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5705182935584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25592895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18611617 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117307579149824 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43184079 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5719181386752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5719093957120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117725965246464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111852108438528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108815681796832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68636909 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44204249 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68636909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44204514 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43184077 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114520761889932 trainer/global_step‚6499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111588502232064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111588502232064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43184079 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68646565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117307683618816…°Ó«2§- _wandbruntime‚2854  _runtime‚2854¹}þj€¨ÕM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43190720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25596834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117743914262528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚114538482378070 _runtime‚2854.824710576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108832421969152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111847824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5719973576704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111869315555328 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108832404929792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5720061013504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5874623610880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68657102 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5705974605682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.785808576532737e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18614619 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HKoÓ[@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111838152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117325625532416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111847824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18614354 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44211453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44211186 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚518 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43190720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111869290651648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111838152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68647432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117325730008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚6499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111605668995072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5719973576704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68657102 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25596832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43190722 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136Lìsà&©ÚM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43190722 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43190720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25596834 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117743914262528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚114538482378070 _runtime‚2854.824710576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108832421969152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111847824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5719973576704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111869315555328 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108832404929792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5720061013504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5874623610880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68657102 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5705974605682 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.785808576532737e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18614619 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111838152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117325625532416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111847824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18614354 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44211453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44211186 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚518 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43190720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111869290651648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111838152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68647432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117325730008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚6499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111605668995072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5719973576704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68657102 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25596832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0†2õŸÕ2ª- _wandbruntime‚2854  _runtime‚2854ÕõLº«þ /loss/FBt1/detection_loss‚1.6094878441654146 /loss/FBt2/detection_loss‚1.1302398028550669 epoch‚0 _runtime‚2854.826064365 0loss/FRt2/assignment_loss‚3.2469858886339726 0loss/FBt1/assignment_loss‚0.2311540800873928 .loss/FRt1/detection_loss‚2.580283274820873 trainer/global_step‚6499 2loss/SRqqt2/assignment_loss‚1.1495412800733635 1loss/SRqqt1/detection_loss‚0.9972325352940096 0loss/SRqqt2/detection_loss‚2.030673970899931 % _timestamp‚1.7858085765345683e+09 _step‚519 0loss/FRt1/assignment_loss‚2.5570865784372603 2loss/SRqqt1/assignment_loss‚1.1816886403786513 2loss/FBt2/assignment_loss‚0.005014739885723962 -loss/FRt2/detection_loss‚2.65627727829493 &loss/total_loss‚19.375665913826587”Å ¬ƒ 2loss/SRqqt2/assignment_loss‚1.1495412800733635 1loss/SRqqt1/detection_loss‚0.9972325352940096 0loss/SRqqt2/detection_loss‚2.030673970899931 % _timestamp‚1.7858085765345683e+09 _step‚519 0loss/FRt1/assignment_loss‚2.5570865784372603 2loss/SRqqt1/assignment_loss‚1.1816886403786513 2loss/FBt2/assignment_loss‚0.005014739885723962 -loss/FRt2/detection_loss‚2.65627727829493 &loss/total_loss‚19.375665913826587 /loss/FBt1/detection_loss‚1.6094878441654146 /loss/FBt2/detection_loss‚1.1302398028550669 epoch‚0 _runtime‚2854.826064365 0loss/FRt2/assignment_loss‚3.2469858886339726 0loss/FBt1/assignment_loss‚0.2311540800873928 .loss/FRt1/detection_loss‚2.580283274820873 trainer/global_step‚6499‡P¥·}œ­‚(:‘ ÊÅÓ··ˆÓ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚143.506' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚79.72555555555556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñéy½˜®‚(:  ÊÅÓ‰ÇšÖ disk.nvme0c0n1.in‚ 32.4140625 disk.rbd1.in‚0 network.sent‚ 6.046421e+06& disk.nvme0n1.out‚11566.43896484375 disk.sda2.in‚0 cpu‚13.944543144041385 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0% memory_percent‚27.456849813461304 proc.cpu.threads‚29 disk.rbd0.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚11566.43896484375 disk.sda3.out‚ 138.76171875 disk.dm-1.out‚ 138.76171875 disk.rbd1.out‚ 1.533203125 disk.drbd1743.in‚0 disk.drbd1743.out‚ 3.20703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd4.out‚ 1.4072265625 disk.dm-0.in‚ 32.4140625 disk.rbd3.in‚0) disk./.usagePercent‚83.39453594517818% disk./.usageGB‚1242.6388511657715 disk.sda.out‚ 138.76171875 disk.dm-1.in‚0# disk.dm-0.out‚11566.43896484375 network.recv‚384699 disk.rbd4.in‚0 disk.sda3.in‚0* proc.memory.availableMB‚47541.87890625) proc.memory.percent‚17.19306707382202 disk.nvme0n1.in‚ 32.4140625 disk.rbd5.in‚0 disk.rbd5.out‚ 1.47021484375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375a<æ£2¯- _wandbruntime‚2873  _runtime‚2873U¿ûÈ°Â _step‚520 ' lr-AdamW/pg1‚0.00016878204929226962 ' lr-AdamW/pg2‚0.00016878204929226962 trainer/global_step‚6549 $ _timestamp‚1.785808595596646e+09 _runtime‚2873.888541047iÚáð걂 ' lr-AdamW/pg1‚0.00016878204929226962 ' lr-AdamW/pg2‚0.00016878204929226962 trainer/global_step‚6549 $ _timestamp‚1.785808595596646e+09 _runtime‚2873.888541047 _step‚520ˆžE2²- _wandbruntime‚2873  _runtime‚2873ô±øUÒ'³ÌO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118209913435648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚118210017904640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112446840382464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112446823343104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115401761811924 _runtime‚2873.892891928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚118209913435648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43516227 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5763090092544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚118210017904640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112712464278528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43516229 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112712439374848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109652690412832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5919071964160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112690796 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25789747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 trainer/global_step‚6549 _step‚521 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25789745 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115401658969440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5918939192832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5749071399092 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858085956008222e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112681138 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748985595968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43516229 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109652673373472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69174567 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69164911 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69164911 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18756484 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5763090092544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112446840382464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112446823343104 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5763177522176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5763177522176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18756245 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538Ù¿¾å,´ÑO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69164911 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69164911 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18756484 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5763090092544 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112446840382464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112446823343104 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5763177522176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5763177522176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18756245 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118209913435648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚118210017904640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112446840382464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112446823343104 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115401761811924 _runtime‚2873.892891928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚118209913435648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43516227 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5763090092544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚118210017904640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112712464278528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43516229 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112712439374848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109652690412832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5919071964160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112690796 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25789747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 trainer/global_step‚6549 _step‚521 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25789745 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115401658969440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5918939192832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5749071399092 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7858085956008222e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112681138 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748985595968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43516229 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109652673373472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69174567 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632‰Uæ_ 2µ- _wandbruntime‚2874  _runtime‚2874)M;€Û&¶ÕM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44552783 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18759362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112698304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112463990106112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44553046 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115419276614982 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚522 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112707976 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109669430585152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112464007145472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 $ _timestamp‚1.785808595938046e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5764057148928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25793684 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118228064294400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118228064294400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112698304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43522870 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69185104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5919970198016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109669413545792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43522872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69185104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25793682 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5763969712128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43522872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118649490216960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5919843725312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121Åçzºà&·ÚM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112464007145472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112729646491648 _runtime‚2874.229972566 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44552783 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18759362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112698304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112463990106112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44553046 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115419276614982 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚522 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112707976 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109669430585152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112464007145472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 $ _timestamp‚1.785808595938046e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5764057148928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25793684 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118228064294400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚118228064294400 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112698304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43522870 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69185104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5919970198016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109669413545792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43522872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69185104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25793682 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5763969712128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43522872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118649490216960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5919843725312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113305161493504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119537056709632 trainer/global_step‚6599 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69692913 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.Cì›$all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7858086158459513e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113305178532864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807086227968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69702569 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113541290 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5792874059476 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119112247721472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25986595 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116282556048948 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113305178532864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119112352190464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807173657600 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113305161493504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119112352190464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113572795214848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43848377 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69692913 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25986597 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43848379 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69702569 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5807086227968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44887452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚525 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5964095431168 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119536890646016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5792959862600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113550948 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5964236591104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807173657600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44887190 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119112247721472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110489699028832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116282658891432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110489681989472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _runtime‚2894.139141027 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807173657600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44887190 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119112247721472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚110489699028832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116282658891432 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110489681989472 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _runtime‚2894.139141027 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113305161493504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119537056709632 trainer/global_step‚6599 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69692913 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7858086158459513e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113305178532864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807086227968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69702569 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113541290 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5792874059476 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119112247721472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25986595 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116282556048948 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113305178532864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119112352190464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807173657600r0ü…2Ã- _wandbruntime‚2894  _runtime‚2894÷;ú˜Ü&ÄÖM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43855020 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119130398580224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116300276537086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43855022 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113322345295872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110506422161792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚526 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5807965847552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18903627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119555157331456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69713106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7858086161818578e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119130398580224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44894161 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25990532 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113322328256512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5965130096128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5808053284352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5964995234816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚116300173694490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44893926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119130294104064 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5793837335934 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5808053284352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚2894.473850186 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69713106 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5793751532698 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113322345295872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113568128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113322328256512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113590002331648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43855020 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚6599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113568128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69703436 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113558456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69703436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113558456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43855022 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119554997566464RùÔÅÛM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113568128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113322328256512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113590002331648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43855020 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚6599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113568128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69703436 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113558456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69703436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113558456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43855022 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119554997566464 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43855020 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119130398580224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116300276537086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43855022 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113322345295872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110506422161792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚526 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5807965847552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18903627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119555157331456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69713106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233jµõ  SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7858086161818578e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119130398580224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44894161 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25990532 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113322328256512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5965130096128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5808053284352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5964995234816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚116300173694490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44893926 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119130294104064 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5793837335934 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5808053284352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _runtime‚2894.473850186 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69713106 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5793751532698 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113322345295872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 PèJ2Í- _wandbruntime‚2914  _runtime‚2914€Õ'ÉÎà ' lr-AdamW/pg1‚0.00017135926794080024 ' lr-AdamW/pg2‚0.00017135926794080024 trainer/global_step‚6649 % _timestamp‚1.7858086357923632e+09 _runtime‚2914.084128737 _step‚528žqÌÎÏÈ ' lr-AdamW/pg2‚0.00017135926794080024 trainer/global_step‚6649 % _timestamp‚1.7858086357923632e+09 _runtime‚2914.084128737 _step‚528 ' lr-AdamW/pg1‚0.00017135926794080024ò¬2Ð- _wandbruntime‚2914  _runtime‚2914~YiÆÒ'ÑÌO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120014686476288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114163516683264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5851082363392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45227032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19043585 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120442472989696 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5836762522984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114411100 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114163499643904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5836848326108 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70220915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70220915 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19043334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚6649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120014582007296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44180527 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120014582007296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120442628567552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44180529 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111326690605472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5851082363392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6009452609024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70230571 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5851169793024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114163499643904 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114401442 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26183447 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5851169793024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120014686476288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44180527 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5836848326108 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70220915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70220915 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19043334 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 trainer/global_step‚6649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120014582007296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44180527 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120014582007296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120442628567552 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44180529 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111326690605472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5851082363392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6009452609024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70230571 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5851169793024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114163499643904 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114401442 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26183447 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5851169793024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120014686476288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44180527 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceSœ£'ʺtatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5851961982976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45233549 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6010358492160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6010223630848 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45233820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5837639996206 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120460741567488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70241108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111343447817152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114428280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5851961982976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44187170 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114180683446272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5852049419776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19046436 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120460581802496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712õ3%Sà&ÕÚM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70241108 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111343447817152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114428280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5851961982976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44187170 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114180683446272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5852049419776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19046436 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120460581802496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44187170 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70231438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111343430777792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5851961982976 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45233549 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6010358492160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6010223630848 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45233820 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5837639996206 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120460741567488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19186796 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115293531798528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19187064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚6699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5895078498816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26380297 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054671928320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112163716260832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45567091 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26380295 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44512679 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115021854833664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70758573 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70748917 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118044350207964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70758573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44512677 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120916916293120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120916916293120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112163699221472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70748917 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291hºxàØàÑO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120916916293120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112163699221472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70748917 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5895165928448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ^JDevi>̆ÿ#ceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121348203726848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19186796 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115293531798528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19187064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚6699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5895078498816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26380297 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054671928320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112163716260832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45567091 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26380295 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44512679 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115021854833664 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70758573 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70748917 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118044350207964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70758573 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44512677 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120916916293120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121•x/À£2á- _wandbruntime‚2936  _runtime‚2936t~P.Ü&âÖM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45574203 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19189714 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5881528459714 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44519322 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19189969 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121366161255936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5895958118400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70769110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115310738915328 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055571619840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115278760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115310714011648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115039004557312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5895958118400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112180456433152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115288432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120934962675712 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70769110 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112180439393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚534 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115039021596672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44519320 _runtime‚2936.685992381 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115039021596672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26384234 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121366310535168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70759440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120935067151872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115039004557312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26384232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚118062070696102 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120935067151872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚6699 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45573946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44519322 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19189969 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121366161255936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5895958118400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70769110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115310738915328 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055571619840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115278760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115310714011648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115039004557312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5895958118400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112180456433152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115288432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120934962675712 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70769110 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112180439393792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚534 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115039021596672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44519320 _runtime‚2936.685992381 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115039021596672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26384234 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121366310535168 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70759440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120935067151872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115039004557312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26384232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚118062070696102 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120935067151872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚6699 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45573946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121819250578944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115880192984064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858086814807317e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71276919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116153887638528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚118925350129956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5924539450000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44844827 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6099716637184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113000724876832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5939074634240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26577145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548¶×ò×'ðÑO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121819250578944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115880192984064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858086814807317e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71276919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116153887638528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚118925350129956 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5924539450000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44844827 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6099716637184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113000724876832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5939074634240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26577145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658™…¸k¦2ñ- _wandbruntime‚2960  _runtime‚2960V5ýëÚ&òÔM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115897359747072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6100748680704 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122271683671040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 2960.18771866 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚118942967775610 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚118942864933014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 $ _timestamp‚1.785808681895636e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116148584 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44851472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44851470 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71287442 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115897342707712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5939954253824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116171069851648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113017448009792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5925502726458 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116138912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5940041690624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26581082 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5925416923222 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44851470 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121837296961536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121837401437696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6100613819392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113017465049152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5939954253824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115897342707712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5940041690624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44851472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71297112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 trainer/global_step‚6749 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45913673 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6100748680704 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@Õ²Ä ¤$DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122271683671040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 2960.18771866 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚118942967775610 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚118942864933014 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 $ _timestamp‚1.785808681895636e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116148584 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44851472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44851470 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71287442 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115897342707712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5939954253824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116171069851648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113017448009792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5925502726458 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116138912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5940041690624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26581082 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 _step‚538 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5925416923222 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44851470 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121837296961536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121837401437696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6100613819392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113017465049152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5939954253824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115897342707712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5940041690624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44851472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71297112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 trainer/global_step‚6749 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45913673 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116991556 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113837733492832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113837716453472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116738531134464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122721584864768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122721584864768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119806144366980 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26773997 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71814577 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26773995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45176979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19473275 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71804921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 % _timestamp‚1.7858087026200655e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123159256342528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6144890578432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116991556 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113837733492832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113837716453472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116738531134464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122721584864768 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122721584864768 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119806144366980 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26773997 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71814577 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26773995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45176979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19473275 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71804921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 % _timestamp‚1.7858087026200655e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123159256342528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6144890578432 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122739631247360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45183620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113854473665152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122739735723520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116999064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71815444 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116755680858112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46254359 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5983950389248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122739735723520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116755680858112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46254101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚542 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6145933304320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119823864855118 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19476169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71825114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116755697897472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116755697897472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71825114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26777934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5984037826048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71815444 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 2981.24889215 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19476425 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6145804734464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113854456625792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119823762012522 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45183620 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123177230426112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969391189966 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5984037826048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117008736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26777932 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117031425691648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45183622 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117031450595328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚6799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117008736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116999064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360š_—¡à&ÚM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71825114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116755697897472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116755697897472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71825114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26777934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5984037826048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71815444 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 2981.24889215 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19476425 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6145804734464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113854456625792 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119823762012522 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45183620 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123177230426112 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969391189966 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5984037826048 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117008736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26777932 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117031425691648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45183622 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117031450595328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 trainer/global_step‚6799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117008736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116999064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969305386730 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122739631247360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45183620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113854473665152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122739735723520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116999064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71815444 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116755680858112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46254359 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5983950389248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122739735723520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116755680858112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46254101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚542 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6145933304320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119823864855118 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19476169žŠÉш2‚- _wandbruntime‚2981  _runtime‚2981ô+Ô} ƒƒ 0loss/FRt2/assignment_loss‚2.7915948025974227 2loss/SRqqt1/assignment_loss‚1.0792693810814873 1loss/SRqqt1/detection_loss‚1.1950558863368863 1loss/SRqqt2/detection_loss‚1.9390155946569783 epoch‚0 _step‚543 2loss/FBt2/assignment_loss‚0.037688834124304314 &loss/total_loss‚18.609260562070567 2loss/SRqqt2/assignment_loss‚1.3325717307095017 1loss/FBt1/assignment_loss‚0.18204081182678541 /loss/FRt1/detection_loss‚2.7709192287709032 /loss/FRt2/detection_loss‚2.4763595667886147 /loss/FBt1/detection_loss‚1.2179021040598552 /loss/FBt2/detection_loss‚0.6829233015222209 trainer/global_step‚6799 _runtime‚2981.250073596 0loss/FRt1/assignment_loss‚2.9039193195956092 % _timestamp‚1.7858087029585876e+09åšî„ˆ 1loss/FBt1/assignment_loss‚0.18204081182678541 /loss/FRt1/detection_loss‚2.7709192287709032 /loss/FRt2/detection_loss‚2.4763595667886147 /loss/FBt1/detection_loss‚1.2179021040598552 /loss/FBt2/detection_loss‚0.6829233015222209 trainer/global_step‚6799 _runtime‚2981.250073596 0loss/FRt1/assignment_loss‚2.9039193195956092 % _timestamp‚1.7858087029585876e+09 0loss/FRt2/assignment_loss‚2.7915948025974227 2loss/SRqqt1/assignment_loss‚1.0792693810814873 1loss/SRqqt1/detection_loss‚1.1950558863368863 1loss/SRqqt2/detection_loss‚1.9390155946569783 epoch‚0 _step‚543 2loss/FBt2/assignment_loss‚0.037688834124304314 &loss/total_loss‚18.609260562070567 2loss/SRqqt2/assignment_loss‚1.3325717307095017Ÿá&}â‘…‚(:† ÂŽÅÓøþ‚Ó gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚122.049' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚67.805 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qín†‚(:‚  ÂŽÅÓýöÕ disk.rbd3.out‚0& disk.nvme0n1.out‚11695.43408203125 disk.sda1.out‚0 network.sent‚ 6.320212e+06 disk.nvme0n1.in‚ 32.4140625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125# disk.dm-0.out‚11695.43408203125 disk.sda3.in‚0# proc.memory.rssMB‚ 11267.6484375% disk./.usageGB‚1242.7217407226562 disk.rbd5.out‚ 1.533203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 144.2890625$ memory_percent‚27.45678424835205( proc.memory.availableMB‚ 47541.921875 proc.cpu.threads‚29 disk.dm-0.in‚ 32.4140625 disk.rbd4.out‚ 1.533203125 disk.dm-1.in‚0 disk.drbd1743.out‚3.328125) proc.memory.percent‚17.19306707382202 cpu‚13.946582392707244 disk.sda1.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0) disk./.usagePercent‚83.40009873288969 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 32.4140625( disk.nvme0c0n1.out‚11695.43408203125 disk.sda2.out‚0 disk.drbd1743.in‚0 network.recv‚399151 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.rbd2.out‚ 1.533203125 disk.sda3.out‚ 144.2890625 disk.dm-1.out‚ 144.2890625Gýcœ‡‚(:‘ ÑŽÅÓ‡¬‰Ó gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚154.043' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.57944444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p =׈‚(:‚  ÑŽÅÓ•ß³Ö disk.nvme0c0n1.in‚ 32.4140625 disk.sda3.out‚ 144.8515625% memory_percent‚27.456718683242798) disk./.usagePercent‚83.40009745285202& disk.nvme0n1.out‚11695.66845703125 disk.rbd2.out‚ 1.533203125 disk.dm-1.in‚0 disk.rbd1.out‚ 1.533203125 cpu‚13.946997776468663 disk.drbd1743.out‚ 3.3359375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 144.8515625 network.recv‚401218 disk.drbd1743.in‚0 disk.rbd2.in‚0 network.sent‚ 6.3653e+06 disk.rbd3.out‚0 disk.rbd4.in‚0( disk.nvme0c0n1.out‚11695.66845703125) proc.memory.percent‚17.19306707382202# disk.dm-0.out‚11695.66845703125 disk.nvme0n1.in‚ 32.4140625 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚29 disk.dm-0.in‚ 32.4140625 disk.rbd3.in‚0 disk.rbd5.out‚ 1.533203125* proc.memory.availableMB‚47541.96484375# disk./.usageGB‚1242.72172164917 disk.rbd4.out‚ 1.533203125 disk.sda.out‚ 144.8515625 disk.rbd1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 11267.6484375`¤ñ 2‰- _wandbruntime‚3001  _runtime‚3001¢êÉŠÃ ' lr-AdamW/pg2‚0.00017651370523786146 trainer/global_step‚6849 % _timestamp‚1.7858087227836747e+09 _runtime‚3001.075366393 _step‚544 ' lr-AdamW/pg1‚0.00017651370523786146Ú§Î‹È ' lr-AdamW/pg2‚0.00017651370523786146 trainer/global_step‚6849 % _timestamp‚1.7858087227836747e+09 _runtime‚3001.075366393 _step‚544 ' lr-AdamW/pg1‚0.00017651370523786146 ü=ZÓ2Œ- _wandbruntime‚3001  _runtime‚3001XîcËO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6190085666304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45509129 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124064819853312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120687041446488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26970845 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6027066905088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26970847 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6012402180140 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19617099 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123623919150592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6027066905088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124064660081152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 $ _timestamp‚1.785808722784713e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117842050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123624023619584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6027154334720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117874599318528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72332923 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _step‚545 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45509129 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117596869284864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117596852245504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123623919150592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6027154334720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46587944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117874574414848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6190220534784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚3001.079929514 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120687144288972 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45509127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114674725069472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117851708 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚6849 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024AjÖ'ŽÐO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6027154334720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117874599318528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72332923 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _step‚545 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45509129 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117596869284864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117596852245504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123623919150592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6027154334720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46587944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117874574414848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6190220534784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚3001.079929514 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120687144288972 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45509127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114674725069472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117851708 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚6849 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6190085666304 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45509129 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124064819853312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120687041446488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26970845 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6027066905088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26970847 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6012402180140 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19617099 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123623919150592 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6027066905088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124064660081152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 $ _timestamp‚1.785808722784713e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117842050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚123624023619584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72353116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117859216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46594995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117859216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45515772 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45515770 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117614036047872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26974782 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6013279653474 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6013193850238 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124082927471104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123642070009344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120704659092030 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72343446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19620211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6191121035776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123642070009344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72353116 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117868888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _runtime‚3001.413881149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19619952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124082769803264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123641965533184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45515770 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46594734 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117891781531648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114691465241792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6028033961472 trainer/global_step‚6849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858087231219177e+09[y$Ê<#‘ÛM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6013279653474 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6013193850238 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124082927471104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123642070009344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120704659092030 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72343446 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19620211 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6191121035776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123642070009344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72353116 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117868888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _runtime‚3001.413881149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19619952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124082769803264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123641965533184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45515770 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46594734 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117891781531648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114691465241792 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6028033961472 trainer/global_step‚6849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858087231219177e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72353116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117859216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46594995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117859216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45515772 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45515770 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117614036047872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26974782 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538¢wb02’- _wandbruntime‚3001  _runtime‚3001¿®6Û“ÿ &loss/total_loss‚19.676924710884094 epoch‚0 2loss/SRqqt1/assignment_loss‚1.2060623548377085 .loss/FBt1/detection_loss‚2.068299639970064 /loss/FBt2/detection_loss‚1.4308179018917144 _runtime‚3001.415140401 0loss/FRt1/assignment_loss‚3.0092161774635313 1loss/SRqqt1/detection_loss‚0.9521366192685331 $ _timestamp‚1.785808723123664e+09 _step‚547 /loss/FRt2/assignment_loss‚2.901126147111257 2loss/SRqqt2/assignment_loss‚1.3319459511174097 1loss/FBt1/assignment_loss‚0.26117482034487516 2loss/FBt2/assignment_loss‚0.024123271057505224 .loss/FRt2/detection_loss‚2.678478176991145 trainer/global_step‚6849 /loss/FRt1/detection_loss‚2.2621927650351275 1loss/SRqqt2/detection_loss‚1.5513508857952223&¯9 ”„ 0loss/FRt1/assignment_loss‚3.0092161774635313 1loss/SRqqt1/detection_loss‚0.9521366192685331 $ _timestamp‚1.785808723123664e+09 _step‚547 /loss/FRt2/assignment_loss‚2.901126147111257 2loss/SRqqt2/assignment_loss‚1.3319459511174097 1loss/FBt1/assignment_loss‚0.26117482034487516 2loss/FBt2/assignment_loss‚0.024123271057505224 .loss/FRt2/detection_loss‚2.678478176991145 trainer/global_step‚6849 /loss/FRt1/detection_loss‚2.2621927650351275 1loss/SRqqt2/detection_loss‚1.5513508857952223 &loss/total_loss‚19.676924710884094 epoch‚0 2loss/SRqqt1/assignment_loss‚1.2060623548377085 .loss/FBt1/detection_loss‚2.068299639970064 /loss/FBt2/detection_loss‚1.4308179018917144 _runtime‚3001.415140401£gaœ•‚(:‘ àŽÅÓÞáùÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.682' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.04555555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07.„•–‚(:Š  àŽÅÓÄÔòÕ network.recv‚402082 disk.rbd4.out‚ 1.533203125 disk.rbd2.out‚ 1.533203125 disk.rbd1.in‚0# disk.dm-0.out‚11696.96630859375# proc.memory.rssMB‚ 11267.6484375 disk.nvme0c0n1.in‚ 32.4140625 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 1.533203125) proc.memory.percent‚17.19306707382202% disk./.usageGB‚1242.7217254638672 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 32.4140625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11696.96630859375% memory_percent‚27.457505464553833* proc.memory.availableMB‚47541.44921875 network.sent‚ 6.367744e+06) disk./.usagePercent‚83.40009770885955& disk.nvme0n1.out‚11696.96630859375 disk.sda1.in‚0 cpu‚13.947218135649884 disk.sda.out‚ 145.66015625 disk.drbd1743.out‚ 3.36328125 disk.rbd3.out‚0 disk.sda3.out‚ 145.66015625 disk.dm-1.in‚0 disk.dm-1.out‚ 145.66015625 disk.dm-0.in‚ 32.4140625 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 1.533203125 proc.cpu.threads‚29d„É@2—- _wandbruntime‚3022  _runtime‚3022Ie,œÉ˜Ã % _timestamp‚1.7858087439880288e+09 _runtime‚3022.279984747 _step‚548 ' lr-AdamW/pg1‚0.00017780231456212677 ' lr-AdamW/pg2‚0.00017780231456212677 trainer/global_step‚6899aK;]Î™È _runtime‚3022.279984747 _step‚548 ' lr-AdamW/pg1‚0.00017780231456212677 ' lr-AdamW/pg2‚0.00017780231456212677 trainer/global_step‚6899 % _timestamp‚1.7858087439880288e+09¤æp…>2š- _wandbruntime‚3022  _runtime‚3022¤øZqÒ'›ÌO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19761132 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115511750724832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46928827 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45841277 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6235367767040 trainer/global_step‚6899 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6071150470144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46929087 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118734930254848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72860925 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121568041368480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858087439893496e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124526357905408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124526253436416 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72860925 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118455190395904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115511733685472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6071063040512 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72870581 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6056204840524 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45841277 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124526357905408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118711860 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118455207435264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45841279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118702202 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6071150470144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124526253436416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118455190395904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124970167347712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19761390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 60562906436488CŽ×'œÑO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118455190395904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115511733685472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6071063040512 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72870581 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6056204840524 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45841277 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124526357905408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118711860 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118455207435264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45841279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 _step‚549 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118702202 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6071150470144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124526253436416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118455190395904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124970167347712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19761390 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6056290643648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19761132 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115511750724832 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46928827 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45841277 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6235367767040 trainer/global_step‚6899 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6071150470144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46929087 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118734930254848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72860925 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121568041368480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858087439893496e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124526357905408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124526253436416 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72860925 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¥±ï$2- _wandbruntime‚3022  _runtime‚3022Õ ±žÖM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6057082313746 _runtime‚3022.686044469 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚118472374198272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 trainer/global_step‚6899 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚550 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115528490897152 % _timestamp‚1.7858087443939476e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45847920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27171634 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72881118 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118752137371648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72871448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72871448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124988434590208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27171632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121585659014134 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72881118 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115528473857792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124988276922368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124544404295168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118719368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46935658 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118752162275328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118729040 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45847922 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6072030096896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121585556171538 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6236272314880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6057168116982 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118719368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118729040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45847922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6236272314880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6057168116982 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118719368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118729040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45847922 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚550 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115528490897152 % _timestamp‚1.7858087443939476e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45847920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27171634 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72881118 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118752137371648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72871448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72871448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124988434590208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27171632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121585659014134 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72881118 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115528473857792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124988276922368 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124544404295168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118719368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46935658 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118752162275328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118729040 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45847922 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6072030096896¦Y'(2 - _wandbruntime‚3022  _runtime‚3022¤}7Сÿ 0loss/FRt1/assignment_loss‚2.3632165463857873 /loss/FRt2/assignment_loss‚2.790781252820727 1loss/FBt1/assignment_loss‚0.16010916067492864 .loss/FRt1/detection_loss‚2.297638863325119 /loss/FRt2/detection_loss‚2.5841346299144585 1loss/SRqqt1/detection_loss‚1.0370564370557214 trainer/global_step‚6899 epoch‚0 _step‚551 1loss/SRqqt2/detection_loss‚1.8279820942297214 &loss/total_loss‚18.787994786040894 2loss/SRqqt1/assignment_loss‚1.0549764150697174 2loss/SRqqt2/assignment_loss‚1.2359568441241253 % _timestamp‚1.7858087443951705e+09 _runtime‚3022.686682105 1loss/FBt2/assignment_loss‚0.12868949689820522 .loss/FBt1/detection_loss‚2.232213702466753 /loss/FBt2/detection_loss‚1.0752393430756295d ¼ ¢„ 0loss/FRt1/assignment_loss‚2.3632165463857873 /loss/FRt2/assignment_loss‚2.790781252820727 1loss/FBt1/assignment_loss‚0.16010916067492864 .loss/FRt1/detection_loss‚2.297638863325119 /loss/FRt2/detection_loss‚2.5841346299144585 1loss/SRqqt1/detection_loss‚1.0370564370557214 trainer/global_step‚6899 epoch‚0 _step‚551 1loss/SRqqt2/detection_loss‚1.8279820942297214 &loss/total_loss‚18.787994786040894 2loss/SRqqt1/assignment_loss‚1.0549764150697174 2loss/SRqqt2/assignment_loss‚1.2359568441241253 % _timestamp‚1.7858087443951705e+09 _runtime‚3022.686682105 1loss/FBt2/assignment_loss‚0.12868949689820522 .loss/FBt1/detection_loss‚2.232213702466753 /loss/FBt2/detection_loss‚1.0752393430756295§*A®œ£‚(:‘ ïŽÅÓÈöôÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚152.146' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.52555555555554 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†µSx’¤‚(:‡  ïŽÅÓ¤–ò× network.sent‚ 6.41267e+06% disk./.usageGB‚1242.7217826843262 disk.rbd2.out‚ 1.533203125 disk.drbd1743.out‚ 3.3828125& disk.nvme0n1.out‚11697.20458984375 disk.rbd5.out‚ 1.533203125 disk.rbd4.out‚ 1.533203125 disk.nvme0c0n1.in‚ 32.4140625 network.recv‚404322 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 146.35546875 disk.rbd3.in‚0( disk.nvme0c0n1.out‚11697.20458984375# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125% memory_percent‚27.457106113433838) proc.memory.availableMB‚ 47541.7109375 cpu‚13.946932938368409) disk./.usagePercent‚83.40010154897256 disk.sda3.out‚ 146.35546875 disk.nvme0n1.in‚ 32.4140625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚29 disk.sda.out‚ 146.35546875 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.4140625 disk.rbd5.in‚0 disk.rbd2.in‚0# disk.dm-0.out‚11697.20458984375 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd4.in‚0 disk.sda1.in‚0U¾E2¥- _wandbruntime‚3043  _runtime‚3043Üґɦà ' lr-AdamW/pg2‚0.00017909092388639208 trainer/global_step‚6949 % _timestamp‚1.7858087648396733e+09 _runtime‚3043.131331505 _step‚552 ' lr-AdamW/pg1‚0.00017909092388639208z›¢zÎ§È ' lr-AdamW/pg1‚0.00017909092388639208 ' lr-AdamW/pg2‚0.00017909092388639208 trainer/global_step‚6949 % _timestamp‚1.7858087648396733e+09 _runtime‚3043.131331505 _step‚552¨×ãò;2¨- _wandbruntime‚3043  _runtime‚3043ê´0Ò'©ÌO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119313528546304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6100179107156 _step‚553 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6115146605568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19904032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125875704103424 trainer/global_step‚6949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27364547 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858087648408148e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6115146605568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46173427 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125428692191232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116348759340832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116348742301472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 _runtime‚3043.136016614 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6115059175936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119595310998528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122448835605504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73388927 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73398583 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46173429 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119313545585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19903760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73398583 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27364545 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119572012 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6115059175936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125428587722240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122448938447988 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125428587722240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73388927 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6280548682752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119595286094848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0º¾]ªÑO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6115146605568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46173427 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125428692191232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116348759340832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116348742301472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 _runtime‚3043.136016614 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6115059175936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119595310998528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122448835605504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73388927 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73398583 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46173429 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119313545585664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19903760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73398583 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27364545 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119572012 O|P/RzIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6115059175936 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125428587722240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122448938447988 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125428587722240 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73388927 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6280548682752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119595286094848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119313528546304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6100179107156 _step‚553 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6115146605568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19904032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125875704103424 trainer/global_step‚6949 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27364547 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 % _timestamp‚1.7858087648408148e+09©£”kq2«- _wandbruntime‚3043  _runtime‚3043èeÇÛ&¬ÕM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125893815808000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122466556093642 trainer/global_step‚6949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119579520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73409120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73399450 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119330712348672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119330695309312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6281322596352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119589192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119612493211648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6116026232320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19906890 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122466453251046 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6100970777254 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47275125 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46180072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119579520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119330712348672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6115938795520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27368484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125446738580992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47275374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚3043.569269945 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125893969281536 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125446634104832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6116026232320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚116365499513152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116365482473792 $ _timestamp‚1.785808765276838e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6281451166208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125446738580992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46180072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119589192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73409120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73399450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27368482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _step‚554 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119589192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73409120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73399450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27368482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _step‚554 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119589192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119612493211648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6116026232320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19906890 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122466453251046 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6100970777254 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47275125 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46180072 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119579520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119330712348672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6115938795520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27368484 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125446738580992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47275374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚3043.569269945 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125893969281536 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125446634104832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6116026232320 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚116365499513152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚116365482473792 $ _timestamp‚1.785808765276838e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6281451166208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125446738580992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46180072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672ª? À®2®- _wandbruntime‚3043  _runtime‚3043šµäw¯þ trainer/global_step‚6949 2loss/FBt2/assignment_loss‚0.019550748187069096 .loss/FRt1/detection_loss‚2.305129870102647 .loss/FRt2/detection_loss‚2.590849695603053 /loss/FBt1/detection_loss‚1.3319011773311829 /loss/FBt2/detection_loss‚1.1080174647048926 &loss/total_loss‚18.984747565286526 epoch‚0 1loss/SRqqt1/assignment_loss‚1.235249235092298 2loss/SRqqt2/assignment_loss‚1.2827938963231202 % _timestamp‚1.7858087652792597e+09 _step‚555 1loss/SRqqt1/detection_loss‚1.0360520687474875 _runtime‚3043.570892209 0loss/FRt1/assignment_loss‚2.7772145385152838 /loss/FRt2/assignment_loss‚3.027640325493283 0loss/FBt1/assignment_loss‚0.3466851347022239 1loss/SRqqt2/detection_loss‚1.9236634104839854:LIq °ƒ 0loss/FRt1/assignment_loss‚2.7772145385152838 /loss/FRt2/assignment_loss‚3.027640325493283 0loss/FBt1/assignment_loss‚0.3466851347022239 1loss/SRqqt2/detection_loss‚1.9236634104839854 trainer/global_step‚6949 2loss/FBt2/assignment_loss‚0.019550748187069096 .loss/FRt1/detection_loss‚2.305129870102647 .loss/FRt2/detection_loss‚2.590849695603053 /loss/FBt1/detection_loss‚1.3319011773311829 /loss/FBt2/detection_loss‚1.1080174647048926 &loss/total_loss‚18.984747565286526 epoch‚0 1loss/SRqqt1/assignment_loss‚1.235249235092298 2loss/SRqqt2/assignment_loss‚1.2827938963231202 % _timestamp‚1.7858087652792597e+09 _step‚555 1loss/SRqqt1/detection_loss‚1.0360520687474875 _runtime‚3043.570892209«Fз–±‚(:’ þŽÅÓËžþÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚106.757' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚59.309444444444445 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´1û·‰²‚(:þ þŽÅÓö»Ö) disk./.usagePercent‚83.40077715285543 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 1.533203125% disk.nvme0n1.out‚11708.0888671875 disk.sda1.in‚0 disk.rbd5.in‚0 disk.nvme0c0n1.in‚ 32.4140625 disk.sda2.in‚0) proc.memory.percent‚17.19306707382202 disk.sda3.out‚ 146.921875 disk.dm-1.in‚0 disk.dm-1.out‚ 146.921875# proc.memory.rssMB‚ 11267.6484375 disk.dm-0.in‚ 32.4140625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd2.in‚0 network.sent‚ 6.457545e+06 disk.drbd1743.in‚0' disk.nvme0c0n1.out‚11708.0888671875 disk.sda.in‚0 disk.sda2.out‚0% memory_percent‚27.457469701766968 disk.drbd1743.out‚3.40625 disk.rbd0.out‚0 disk.sda.out‚ 146.921875 disk.rbd1.in‚0 proc.cpu.threads‚29% disk./.usageGB‚1242.7318496704102 disk.rbd5.out‚ 1.533203125 disk.rbd2.out‚ 1.533203125 disk.sda3.in‚0 disk.rbd1.out‚ 1.533203125* proc.memory.availableMB‚47541.47265625 network.recv‚406234" disk.dm-0.out‚11708.0888671875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.4140625 cpu‚13.946924168136517c^œ³‚(:‘ ÅÓ¤ºÜÒ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚123.112' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.39555555555556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ÒÿŽ´‚(:ƒ  ÅÓÁµêÕ disk.sda.in‚0 disk.rbd2.out‚ 1.533203125 disk.sda3.out‚ 147.81640625 disk.dm-1.out‚ 147.81640625 disk.dm-0.in‚ 32.4140625 disk.rbd0.out‚0 proc.cpu.threads‚29% disk./.usageGB‚1242.7319412231445 disk.rbd1.out‚ 1.533203125 disk.rbd4.in‚0 network.sent‚ 6.502327e+06 disk.sda2.out‚0 disk.drbd1743.out‚ 3.4296875% disk.nvme0n1.out‚11708.3779296875% memory_percent‚27.456748485565186) proc.memory.percent‚17.19306707382202 disk.sda.out‚ 147.81640625 disk.sda3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.4140625 disk.rbd5.out‚ 1.533203125 disk.rbd4.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375 cpu‚13.9464665696445 network.recv‚408363 disk.nvme0c0n1.in‚ 32.4140625' disk.nvme0c0n1.out‚11708.3779296875 disk.drbd1743.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 47541.9453125 disk.dm-1.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚11708.3779296875) disk./.usagePercent‚83.40078329703624 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd5.in‚0 disk.rbd0.in‚07³f2µ- _wandbruntime‚3065  _runtime‚3065P,<™Æ¶À _runtime‚3065.722486373 _step‚556 & lr-AdamW/pg1‚0.0001803795332106574 & lr-AdamW/pg2‚0.0001803795332106574 trainer/global_step‚6999 $ _timestamp‚1.785808787430809e+09^³­úË·Å & lr-AdamW/pg1‚0.0001803795332106574 & lr-AdamW/pg2‚0.0001803795332106574 trainer/global_step‚6999 $ _timestamp‚1.785808787430809e+09 _runtime‚3065.722486373 _step‚556¬µdÿ2¸- _wandbruntime‚3065  _runtime‚3065‹ß2·¥¹ÌO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚126331026477056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚557 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6159142740992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46505577 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120455641934848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123329835527496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46505579 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73926585 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 % _timestamp‚1.7858087874319193e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126330922008064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126330922008064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6159055311360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6143981767540 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73926585 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120171866696704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130N~ßÖ×'ºÑO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6159055311360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46505579 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126781432942080 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6144067570664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120455666838528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 trainer/global_step‚6999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120422506 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120171883736064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120171883736064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117185767956832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117185750917472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120432164 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73916929 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6159142740992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46505577 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126331026477056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47608185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6143981767540 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73926585 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120171866696704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚126331026477056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚557 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6159142740992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46505577 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120455641934848 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123329835527496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46505579 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73926585 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 % _timestamp‚1.7858087874319193e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚126330922008064 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126330922008064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6159055311360 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47614967 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46512222 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6159934930944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73937122 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126799563677696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120439672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117202491089792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6159934930944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120189050499072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6326559055360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3066.100175483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20049892 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73927452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126799408107008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123347453173150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126349072866816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73937122 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27565334 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117202508129152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120439672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6160022367744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120449344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6144945043998 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120189050499072 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6160022367744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126348968390656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46512220 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251@”—aͽÙM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6159934930944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73937122 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126799563677696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120439672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117202491089792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6159934930944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120189050499072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6326559055360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚3066.100175483 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20049892 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73927452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126799408107008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123347453173150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚126349072866816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73937122 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27565334 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117202508129152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120439672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6160022367744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120449344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6144945043998 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120189050499072 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6160022367744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚126348968390656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46512220 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120189033459712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27565332 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120189033459712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47615226 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 _step‚558 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73927452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6144859240762 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47614967 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46512222 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640®-–m»2¾- _wandbruntime‚3066  _runtime‚3066íÿY¿ý 0loss/FBt1/assignment_loss‚0.5159833824708286 /loss/FRt2/detection_loss‚2.6423381230410405 epoch‚0 # _timestamp‚1.78580878780956e+09 2loss/SRqqt2/assignment_loss‚1.2225356000034433 .loss/FRt1/detection_loss‚2.490242764443518 1loss/SRqqt1/detection_loss‚1.0281702761600415 1loss/SRqqt2/detection_loss‚2.1356132742213574 /loss/FBt2/detection_loss‚1.1587978028581434 %loss/total_loss‚19.96385008810465 _runtime‚3066.101175639 2loss/SRqqt1/assignment_loss‚0.9201115280638138 2loss/FBt2/assignment_loss‚0.007401218618063358 /loss/FBt1/detection_loss‚2.2110443362034857 trainer/global_step‚6999 _step‚559 /loss/FRt1/assignment_loss‚2.688440593610327 0loss/FRt2/assignment_loss‚2.9431711884105907zí§hÀ‚ %loss/total_loss‚19.96385008810465 _runtime‚3066.101175639 2loss/SRqqt1/assignment_loss‚0.9201115280638138 2loss/FBt2/assignment_loss‚0.007401218618063358 /loss/FBt1/detection_loss‚2.2110443362034857 trainer/global_step‚6999 _step‚559 /loss/FRt1/assignment_loss‚2.688440593610327 0loss/FRt2/assignment_loss‚2.9431711884105907 0loss/FBt1/assignment_loss‚0.5159833824708286 /loss/FRt2/detection_loss‚2.6423381230410405 epoch‚0 # _timestamp‚1.78580878780956e+09 2loss/SRqqt2/assignment_loss‚1.2225356000034433 .loss/FRt1/detection_loss‚2.490242764443518 1loss/SRqqt1/detection_loss‚1.0281702761600415 1loss/SRqqt2/detection_loss‚2.1356132742213574 /loss/FBt2/detection_loss‚1.1587978028581434¯d˜¸œÁ‚(:‘ œÅÓð›‹Ó gpu.0.gpu‚14.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚154.043' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚85.57944444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÑWÄ~‚(:ó œÅÓÕØ·Ø) proc.memory.percent‚17.19306707382202) disk./.usagePercent‚83.38410261412776 disk.rbd5.in‚0 disk.drbd1743.out‚3.4375$ memory_percent‚27.45717763900757 network.sent‚ 6.504776e+06 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚148.4375 disk.rbd3.out‚0 cpu‚13.9468926746043 disk.sda2.out‚0% disk./.usageGB‚1242.4833869934082 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 32.4140625 disk.rbd1.in‚0 disk.dm-0.in‚ 32.4140625 network.recv‚409338 disk.nvme0n1.in‚ 32.4140625% disk.nvme0n1.out‚11708.9736328125 disk.rbd2.in‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1743.in‚0) proc.memory.availableMB‚ 47541.6640625 disk.sda3.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚29 disk.rbd5.out‚ 1.533203125 disk.rbd4.out‚ 1.533203125 disk.sda3.out‚148.4375 disk.sda2.in‚0" disk.dm-0.out‚11708.9736328125# proc.memory.rssMB‚ 11267.6484375 disk.sda1.in‚0 disk.sda.out‚148.4375 disk.rbd2.out‚ 1.533203125 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚11708.9736328125FÃ2Ã- _wandbruntime‚3085  _runtime‚3085jMÄÆÄÀ & lr-AdamW/pg1‚0.0001816681425349227 & lr-AdamW/pg2‚0.0001816681425349227 trainer/global_step‚7049 % _timestamp‚1.7858088074531379e+09 _runtime‚ 3085.74486687 _step‚560‡CËÅÅ _step‚560 & lr-AdamW/pg1‚0.0001816681425349227 & lr-AdamW/pg2‚0.0001816681425349227 trainer/global_step‚7049 % _timestamp‚1.7858088074531379e+09 _runtime‚ 3085.74486687°Œß»2Æ- _wandbruntime‚3085  _runtime‚3085XÄMzÒ'ÇÌO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27758247 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127233256293888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74454587 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6187870231048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121030204847104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46837727 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20190932 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20190683 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚7049 % _timestamp‚1.7858088074541507e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6370825688064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118022759533472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27758245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127686987429376 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47949179 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6203051446784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6370964750848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127233256293888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121030221886464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118022776572832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46837729 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124210629764520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203051446784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6187956034172 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121030204847104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127233256293888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121030221886464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118022776572832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46837729 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124210629764520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6203051446784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6187956034172 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121030204847104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚561 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74454587 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6187870231048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121030204847104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46837727 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20190932 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20190683 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚7049 % _timestamp‚1.7858088074541507e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6370825688064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118022759533472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27758245 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127686987429376 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0±o–‘2É- _wandbruntime‚3086  _runtime‚3086™Ð#eÊÕM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46844372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121047388649472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127251407152640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127704932718080 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124228247410062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6371864784896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6203931066368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46844370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20193670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6188833507506 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46844370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6371727826432 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74465124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27762184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127251407152640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6204018503168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74455454 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118039516745152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47955602 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚562 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121309496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121333229795328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121309496 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121047371610112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6203931066368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127705094580224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124228350252658m3ãà&ËÚM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46844372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121047388649472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7049 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127251407152640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127704932718080 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124228247410062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6371864784896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6203931066368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46844370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20193670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6188833507506 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46844370 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6371727826432 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74465124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27762184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127251407152640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6204018503168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74455454 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118039516745152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47955602 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚562 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121309496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121333229795328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121309496 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121047371610112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6203931066368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127705094580224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124228350252658 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0²…ö=N2Ì- _wandbruntime‚3086  _runtime‚3086]–óÍý -loss/FBt1/detection_loss‚2.05738821678928 _step‚563 /loss/FBt2/detection_loss‚1.2541878654488496 epoch‚0 trainer/global_step‚7049 /loss/FRt2/assignment_loss‚3.090311311026837 2loss/FBt2/assignment_loss‚0.005851802153581584 0loss/SRqqt2/detection_loss‚1.958227925002575 % _timestamp‚1.7858088077959251e+09 _runtime‚3086.087432558 2loss/SRqqt1/assignment_loss‚1.3341300908314924 1loss/FBt1/assignment_loss‚0.08183577656632934 .loss/FRt1/detection_loss‚2.495584539762915 /loss/FRt2/detection_loss‚2.5229804471452186 1loss/SRqqt1/detection_loss‚0.9189198211235448 &loss/total_loss‚19.044926802097102 0loss/FRt1/assignment_loss‚2.4836706845727687 1loss/SRqqt2/assignment_loss‚0.841838321673709-»Eî΂ /loss/FBt2/detection_loss‚1.2541878654488496 epoch‚0 trainer/global_step‚7049 /loss/FRt2/assignment_loss‚3.090311311026837 2loss/FBt2/assignment_loss‚0.005851802153581584 0loss/SRqqt2/detection_loss‚1.958227925002575 % _timestamp‚1.7858088077959251e+09 _runtime‚3086.087432558 2loss/SRqqt1/assignment_loss‚1.3341300908314924 1loss/FBt1/assignment_loss‚0.08183577656632934 .loss/FRt1/detection_loss‚2.495584539762915 /loss/FRt2/detection_loss‚2.5229804471452186 1loss/SRqqt1/detection_loss‚0.9189198211235448 &loss/total_loss‚19.044926802097102 0loss/FRt1/assignment_loss‚2.4836706845727687 1loss/SRqqt2/assignment_loss‚0.841838321673709 -loss/FBt1/detection_loss‚2.05738821678928 _step‚563³acÓýœÏ‚(:‘ «ÅÓéï€Ó gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚147.947' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.19277777777778 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Q)ȃЂ(:ø «ÅÓúåÎÕ network.recv‚411430) disk./.usagePercent‚83.36717385991858 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.4140625 disk.rbd5.out‚ 1.533203125 disk.sda.out‚149.1875 disk.rbd1.out‚ 1.533203125 disk.drbd1743.in‚0% disk.nvme0n1.out‚11713.0009765625 disk.sda1.out‚0# proc.memory.rssMB‚ 11267.6484375 disk.rbd4.in‚0% memory_percent‚27.457118034362793 disk.nvme0c0n1.in‚ 32.4140625' disk.nvme0c0n1.out‚11713.0009765625 disk.dm-1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 47541.703125) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29% disk./.usageGB‚1242.2311363220215 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.dm-1.out‚149.1875 disk.sda2.out‚0 disk.drbd1743.out‚ 3.45703125 disk.dm-0.in‚ 32.4140625" disk.dm-0.out‚11713.0009765625 disk.rbd5.in‚0 cpu‚13.94716816071494 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd4.out‚ 1.533203125 disk.rbd2.out‚ 1.533203125 disk.sda3.in‚0 disk.sda3.out‚149.1875 network.sent‚ 6.549716e+06·B±œÑ‚(:‘ ºÅÓ±áÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚150.016' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.34222222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥×#$ŽÒ‚(:ƒ  ºÅÓÛËÃÕ disk.rbd0.out‚0( disk.nvme0c0n1.out‚11713.96240234375 disk.rbd2.out‚ 1.533203125 disk.rbd1.out‚ 1.533203125) disk./.usagePercent‚83.36716771573775 disk.dm-1.in‚0 network.recv‚413218 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 32.4140625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 32.4140625$ disk./.usageGB‚1242.231044769287 disk.rbd4.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 149.921875 disk.sda2.in‚0 disk.drbd1743.out‚ 3.48046875 disk.rbd3.out‚0 proc.cpu.threads‚29 disk.dm-0.in‚ 32.4140625 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.sda2.out‚0% memory_percent‚27.457493543624878) proc.memory.percent‚17.19306707382202 disk.rbd4.out‚ 1.533203125 disk.sda.out‚ 149.921875 disk.sda3.in‚0 disk.drbd1743.in‚0 cpu‚13.947260286292009 disk.sda3.out‚ 149.921875 disk.rbd3.in‚0& disk.nvme0n1.out‚11713.96240234375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.availableMB‚47541.45703125# proc.memory.rssMB‚ 11267.6484375 network.sent‚ 6.594481e+06# disk.dm-0.out‚11713.96240234375 disk.rbd5.out‚ 1.533203125” Ζ2Ó- _wandbruntime‚3106  _runtime‚3106³Â«LÅÔ¿ trainer/global_step‚7099 % _timestamp‚1.7858088278863144e+09 _runtime‚3106.177961475 _step‚564 % lr-AdamW/pg1‚0.000182956751859188 % lr-AdamW/pg2‚0.000182956751859188árÝÊÕÄ _step‚564 % lr-AdamW/pg1‚0.000182956751859188 % lr-AdamW/pg2‚0.000182956751859188 trainer/global_step‚7099 % _timestamp‚1.7858088278863144e+09 _runtime‚3106.177961475´’eI2Ö- _wandbruntime‚3106  _runtime‚3106ï–ÚÒ'×ÌO trainer/global_step‚7099 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74972933 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚565 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6231844497680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122142810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122176378518528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128135695048704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48288769 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128135590579712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47169877 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121888560036864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121888542997504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125091629686512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122152468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128592408266752 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125091526844028 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118859785188832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118859768149472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3106.182931858 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20333672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48288512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6416179034624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0yÁƒ=öØÑO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6247047582208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128135590579712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47169879 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 % _timestamp‚1.7858088278872788e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121888542997504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27955097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122176353614848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74972933 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _step‚565 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeªQ`/áviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6231844497680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122142810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122176378518528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128135695048704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48288769 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128135590579712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47169877 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121888560036864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚121888542997504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125091629686512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122152468 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128592408266752 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125091526844028 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚118859785188832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118859768149472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3106.182931858 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20333672 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48288512 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 149286400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6416179034624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 124382720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚7099 Jèò2Ù- _wandbruntime‚3106  _runtime‚3106i§ £Ù&ÚÓM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27959034 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6247927201792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122169648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74993126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20336277 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128610516104704 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48295309 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 3106.53928377 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47176520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122193585635328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118876525361152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128153741438464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121905726799872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74983456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47176522 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128153741438464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78580882824716e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122159976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47176522 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121905709760512 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47176520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121905726799872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6232636167778 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74993126 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚566 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚118876508321792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122169648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121905709760512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888ù;,uÞ&ÛØM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121905726799872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74983456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47176522 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128153741438464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78580882824716e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122159976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47176522 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121905709760512 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47176520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121905726799872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6232636167778 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74993126 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚566 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚118876508321792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122169648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121905709760512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27959034 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6247927201792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122169648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74993126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20336277 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128610516104704 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48295309 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 3106.53928377 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47176520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122193585635328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118876525361152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128153741438464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816¶q§C-2Ü- _wandbruntime‚3106  _runtime‚3106(jÝý 1loss/SRqqt1/assignment_loss‚1.127306003217833 2loss/SRqqt2/assignment_loss‚1.0855062817473238 1loss/FBt1/assignment_loss‚0.20695584948684467 -loss/FRt2/detection_loss‚2.73003388941288 /loss/FBt2/detection_loss‚0.9561016578443868 _runtime‚3106.540281842 _step‚567 /loss/FRt2/assignment_loss‚3.242461260996367 2loss/FBt2/assignment_loss‚0.011807074535169994 /loss/FBt1/detection_loss‚0.5592257517869335 &loss/total_loss‚17.811683935961685 $ _timestamp‚1.785808828248724e+09 0loss/FRt1/assignment_loss‚2.4950300322638617 epoch‚0 .loss/FRt1/detection_loss‚2.267488720350795 1loss/SRqqt1/detection_loss‚1.0047606701809395 1loss/SRqqt2/detection_loss‚2.1250067441383513 trainer/global_step‚7099­ ½Þ‚ .loss/FRt1/detection_loss‚2.267488720350795 1loss/SRqqt1/detection_loss‚1.0047606701809395 1loss/SRqqt2/detection_loss‚2.1250067441383513 trainer/global_step‚7099 1loss/SRqqt1/assignment_loss‚1.127306003217833 2loss/SRqqt2/assignment_loss‚1.0855062817473238 1loss/FBt1/assignment_loss‚0.20695584948684467 -loss/FRt2/detection_loss‚2.73003388941288 /loss/FBt2/detection_loss‚0.9561016578443868 _runtime‚3106.540281842 _step‚567 /loss/FRt2/assignment_loss‚3.242461260996367 2loss/FBt2/assignment_loss‚0.011807074535169994 /loss/FBt1/detection_loss‚0.5592257517869335 &loss/total_loss‚17.811683935961685 $ _timestamp‚1.785808828248724e+09 0loss/FRt1/assignment_loss‚2.4950300322638617 epoch‚0·¿ ršß‚(: ÉÅÓ€˜ùÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚146.59' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚81.4388888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݳÔà‚(:…  ÉÅÓ¾ûÖ disk.rbd2.out‚ 1.533203125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 6.596925e+06! disk.nvme0c0n1.in‚ 32.41796875 disk.sda2.in‚0! disk.dm-0.out‚11714.294921875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 32.41796875 disk.rbd5.in‚0 disk.rbd5.out‚ 1.533203125 disk.dm-1.out‚ 151.15234375 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.41796875 disk.rbd3.out‚0$ disk.nvme0n1.out‚11714.294921875 cpu‚13.947539676721195 disk.rbd4.in‚0 disk.drbd1743.out‚ 3.5078125 disk.sda1.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.19306707382202) disk./.usagePercent‚83.36716387562474 disk.sda.in‚0 disk.sda.out‚ 151.15234375 disk.rbd2.in‚0 disk.rbd4.out‚ 1.533203125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚27.45710015296936* proc.memory.availableMB‚47541.71484375& disk.nvme0c0n1.out‚11714.294921875 disk.sda3.out‚ 151.15234375 proc.cpu.threads‚29 network.recv‚414324% disk./.usageGB‚1242.2309875488281 disk.rbd1.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375ÊRL÷2á- _wandbruntime‚3126  _runtime‚3126,¬ÙÆâÀ & lr-AdamW/pg2‚0.0001842453611834533 trainer/global_step‚7149 $ _timestamp‚1.785808848420081e+09 _runtime‚3126.711724811 _step‚568 & lr-AdamW/pg1‚0.0001842453611834533aËãÅ & lr-AdamW/pg2‚0.0001842453611834533 trainer/global_step‚7149 $ _timestamp‚1.785808848420081e+09 _runtime‚3126.711724811 _step‚568 & lr-AdamW/pg1‚0.0001842453611834533¸VËç(2ä- _wandbruntime‚3126  _runtime‚3126|Ž|úxåËO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75500935 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129037924865536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123012620 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125972423923536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 trainer/global_step‚7149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123002962 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6291131147264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47502029 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6291131147264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6275732961188 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48629516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48629255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75510591 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6291043717632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47502027 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119696793804832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6275647158064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122746898187264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122746881147904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125972526766020 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119696776765472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122746881147904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20477310 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6461394259968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129038029334528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6291043717632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122746898187264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129497970943488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123036709454848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47502027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20477569 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47502029 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28151947 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129037924865536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123012620 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125972423923536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 trainer/global_step‚7149 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123002962 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6291131147264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47502029 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6291131147264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6275732961188 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48629516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48629255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75510591 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6291043717632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47502027 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119696793804832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6275647158064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122746898187264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122746881147904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125972526766020 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119696776765472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122746881147904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20477310 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6461394259968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129038029334528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6291043717632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122746898187264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129497970943488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123036709454848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47502027 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20477569 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47502029 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28151947 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7858088487554317e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123029800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119713516937792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6291923337216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123020128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75511458 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48636047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122764047910912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47508670 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123029800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3127.047402152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75511458 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6292010774016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122764064950272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129516236719616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20480410 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20480165 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123020128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129055971248128øuý  éÛM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122764047910912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47508670 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123029800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3127.047402152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75511458 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6292010774016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122764064950272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129516236719616 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20480410 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20480165 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123020128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129055971248128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6291923337216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123053916571648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserveq€.¼Ad_bytes.all.current‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47508670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125990041569078 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129056075724288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6276524631286 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47508672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28155884 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7858088487554317e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123029800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119713516937792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6291923337216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚570 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123020128 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75511458 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48636047 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360ºpCR2ê- _wandbruntime‚3127  _runtime‚3127Êa¼Ÿëÿ /loss/FRt1/assignment_loss‚2.760351132123898 /loss/FRt2/assignment_loss‚2.964517711156822 2loss/FBt2/assignment_loss‚0.022776589294035362 2loss/SRqqt1/assignment_loss‚1.2014930716495176 1loss/FBt1/assignment_loss‚0.16087113859032343 .loss/FRt1/detection_loss‚2.599015985734952 /loss/FRt2/detection_loss‚2.5990019160580924 1loss/SRqqt1/detection_loss‚1.0956803967710584 /loss/FBt1/detection_loss‚2.2072437391099005 &loss/total_loss‚19.475296136644452 epoch‚0 trainer/global_step‚7149 % _timestamp‚1.7858088487566519e+09 _runtime‚3127.048153271 1loss/SRqqt2/assignment_loss‚0.997802703268826 1loss/SRqqt2/detection_loss‚1.9860518788918853 /loss/FBt2/detection_loss‚0.8804898739951413 _step‚571 Œ“8 ì„ &loss/total_loss‚19.475296136644452 epoch‚0 trainer/global_step‚7149 % _timestamp‚1.7858088487566519e+09 _runtime‚3127.048153271 1loss/SRqqt2/assignment_loss‚0.997802703268826 1loss/SRqqt2/detection_loss‚1.9860518788918853 /loss/FBt2/detection_loss‚0.8804898739951413 _step‚571 /loss/FRt1/assignment_loss‚2.760351132123898 /loss/FRt2/assignment_loss‚2.964517711156822 2loss/FBt2/assignment_loss‚0.022776589294035362 2loss/SRqqt1/assignment_loss‚1.2014930716495176 1loss/FBt1/assignment_loss‚0.16087113859032343 .loss/FRt1/detection_loss‚2.599015985734952 /loss/FRt2/detection_loss‚2.5990019160580924 1loss/SRqqt1/detection_loss‚1.0956803967710584 /loss/FBt1/detection_loss‚2.2072437391099005»ˆ-‚‡œí‚(:‘ ØÅÓÈÕâÒ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚131.079' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚72.82166666666667 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ý­Ñ“î‚(:ˆ  ØÅÓÖÁ•Ö) proc.memory.percent‚17.19306707382202 disk.dm-1.out‚ 152.50390625 disk.drbd1743.out‚3.515625# disk.dm-0.out‚11715.00244140625& disk.nvme0n1.out‚11715.00244140625 disk.sda1.out‚0) disk./.usagePercent‚83.36716387562474 disk.drbd1743.in‚0 disk.rbd5.out‚ 1.533203125 proc.cpu.threads‚29 network.recv‚416490 disk.rbd1.in‚0( disk.nvme0c0n1.out‚11715.00244140625 disk.sda.out‚ 152.50390625# proc.memory.rssMB‚ 11267.6484375 network.sent‚ 6.641939e+06% disk./.usageGB‚1242.2309875488281 disk.sda2.out‚0 disk.rbd5.in‚0 disk.sda.in‚0 disk.rbd2.out‚ 1.533203125 disk.sda3.in‚0 disk.sda3.out‚ 152.50390625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 32.41796875 disk.sda1.in‚0 disk.rbd4.in‚0! disk.nvme0c0n1.in‚ 32.41796875 disk.dm-0.in‚ 32.41796875 disk.rbd3.in‚0 cpu‚13.947937167844406 disk.sda2.in‚0 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.533203125 disk.rbd4.out‚ 1.533203125$ memory_percent‚27.45816707611084( proc.memory.availableMB‚ 47541.015625¬xžq2ï- _wandbruntime‚3147  _runtime‚3147ë~»ãÇðÁ & lr-AdamW/pg1‚0.0001855339705077186 & lr-AdamW/pg2‚0.0001855339705077186 trainer/global_step‚7199 % _timestamp‚1.7858088689344988e+09 _runtime‚3147.226169294 _step‚572†NEÌñÆ & lr-AdamW/pg1‚0.0001855339705077186 & lr-AdamW/pg2‚0.0001855339705077186 trainer/global_step‚7199 % _timestamp‚1.7858088689344988e+09 _runtime‚3147.226169294 _step‚572¼SPm—2ò- _wandbruntime‚3147  _runtime‚3147”íÍÒ'óÌO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48969465 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123897065294848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6319621424696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 % _timestamp‚1.7858088689354968e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129940259151360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123605236337664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123605219298304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129940363620352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47834177 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28348795 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120533802420832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6319535621572 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47834177 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123872772 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6335039853056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120533785381472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20620668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚126853321003044 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚573 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _runtime‚3147.230972644 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20620430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126853423845528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6335127282688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48969225 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130403659000320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76028937 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129940259151360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47834179 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚7199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123897090198528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123863114 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76038593 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6335127282688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28348797 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129940363620352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76038593 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47834179 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123605219298304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6506568801792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335039853056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123605219298304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6506568801792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335039853056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129940259151360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123605236337664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123605219298304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129940363620352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47834177 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28348795 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120533802420832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6319535621572 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47834177 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123872772 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6335039853056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120533785381472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20620668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚126853321003044 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 _step‚573 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 _runtime‚3147.230972644 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20620430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚126853423845528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6335127282688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48969225 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130403659000320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76028937 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129940259151360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47834179 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚7199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123897090198528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856½¨‘2õ- _wandbruntime‚3147  _runtime‚3147 ;öÖM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47840820 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76039460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20623219 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6507478567936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120550542593152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pooÞ»¹Ál.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76049130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20623479 % _timestamp‚1.7858088693062818e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76049130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48976213 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123622386061312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123914297315328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123622386061312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28352734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120550525553792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123880280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47840820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47840822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6336006909440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6320413094794 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6335919472640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48975951 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130421775883264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123880280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123889952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6335919472640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126871041491182 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6320498898030 _runtime‚3147.598334542 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129958410010112 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123622403100672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123622403100672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123889952 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251!ÂÝøá&÷ÛM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47840822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6336006909440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6320413094794 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6335919472640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48975951 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130421775883264 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123880280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123889952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6335919472640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126871041491182 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6320498898030 _runtime‚3147.598334542 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129958410010112 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123622403100672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123622403100672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123889952 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47840820 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76039460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20623219 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6507478567936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120550542593152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76049130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20623479 % _timestamp‚1.7858088693062818e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76049130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48976213 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123622386061312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123914297315328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123622386061312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28352734 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120550525553792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123880280 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47840820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130¾*eKó2ø- _wandbruntime‚3147  _runtime‚3147B-Lùþ /loss/FRt1/detection_loss‚2.2616367167896696 1loss/SRqqt2/detection_loss‚1.7997528923855675 trainer/global_step‚7199 2loss/SRqqt1/assignment_loss‚1.0236311691813171 epoch‚0 $ _timestamp‚1.785808869307779e+09 0loss/FRt1/assignment_loss‚2.4021227836608885 2loss/SRqqt2/assignment_loss‚1.4523902679643323 3loss/FBt2/assignment_loss‚0.0035150086902763383 /loss/FBt1/detection_loss‚0.8885818488068051 _step‚575 0loss/FRt2/assignment_loss‚2.6834251398735858 0loss/FBt1/assignment_loss‚0.2208190674003837 -loss/FRt2/detection_loss‚2.42286017220071 1loss/SRqqt1/detection_loss‚1.0226800897890436 .loss/FBt2/detection_loss‚0.819635718965271 %loss/total_loss‚17.00105087570785 _runtime‚3147.599333696Ž"#Ê úƒ 1loss/SRqqt1/detection_loss‚1.0226800897890436 .loss/FBt2/detection_loss‚0.819635718965271 %loss/total_loss‚17.00105087570785 _runtime‚3147.599333696 /loss/FRt1/detection_loss‚2.2616367167896696 1loss/SRqqt2/detection_loss‚1.7997528923855675 trainer/global_step‚7199 2loss/SRqqt1/assignment_loss‚1.0236311691813171 epoch‚0 $ _timestamp‚1.785808869307779e+09 0loss/FRt1/assignment_loss‚2.4021227836608885 2loss/SRqqt2/assignment_loss‚1.4523902679643323 3loss/FBt2/assignment_loss‚0.0035150086902763383 /loss/FBt1/detection_loss‚0.8885818488068051 _step‚575 0loss/FRt2/assignment_loss‚2.6834251398735858 0loss/FBt1/assignment_loss‚0.2208190674003837 -loss/FRt2/detection_loss‚2.42286017220071¿~ “<œû‚(:‘ çÅÓ¬óÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚149.478' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.04333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WxŽü‚(:ƒ  çÅÓ­®ŽÖ& disk.nvme0n1.out‚11777.46337890625 disk.rbd4.out‚ 1.533203125 disk.rbd2.out‚ 1.533203125 disk.dm-1.in‚0# proc.memory.rssMB‚ 11267.6484375 cpu‚13.948255041207322 proc.cpu.threads‚29 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 153.359375 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.41796875$ memory_percent‚27.45823860168457' proc.memory.availableMB‚ 47540.96875 disk.sda3.out‚ 153.359375 disk.sda2.out‚0 disk.drbd1743.out‚ 3.53515625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd5.out‚ 1.533203125 disk.rbd2.in‚0 disk.sda3.in‚0) proc.memory.percent‚17.19306707382202 disk.rbd0.in‚0( disk.nvme0c0n1.out‚11777.46337890625 disk.sda.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd4.in‚0) disk./.usagePercent‚83.37167780046848% disk./.usageGB‚1242.2982482910156 disk.sda1.out‚0 network.sent‚ 6.686664e+06 disk.nvme0n1.in‚ 32.41796875 disk.dm-1.out‚ 153.359375 network.recv‚418607! disk.nvme0c0n1.in‚ 32.41796875 disk.rbd1.out‚ 1.533203125# disk.dm-0.out‚11777.46337890625kHÿ›ý‚(: öÅÓÉŽ„Ó gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚148.485' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.49166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K̆â‹þ‚(:€  öÅÓ•×Õ disk.sda1.in‚0& disk.nvme0c0n1.out‚11777.986328125 disk.rbd2.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 cpu‚13.948305188828279 disk.sda1.out‚0 disk.rbd5.out‚ 1.533203125 disk.sda2.in‚0 disk.nvme0n1.in‚ 32.41796875 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 153.91796875 disk.drbd1743.out‚3.5625 disk.rbd3.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 32.41796875 disk.sda.in‚0 disk.sda3.out‚ 153.91796875 disk.dm-1.in‚0 disk.sda2.out‚0 network.recv‚420474$ disk./.usageGB‚1242.298252105713 disk.sda.out‚ 153.91796875 network.sent‚ 6.731639e+06$ disk.nvme0n1.out‚11777.986328125 disk.rbd4.out‚ 1.533203125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125 disk.dm-0.in‚ 32.41796875% memory_percent‚27.457910776138306) proc.memory.percent‚17.19306707382202 disk.rbd2.out‚ 1.533203125! disk.dm-0.out‚11777.986328125 proc.cpu.threads‚29' disk./.usagePercent‚83.371678056476 disk.rbd4.in‚0 disk.drbd1743.in‚0* proc.memory.availableMB‚47541.18359375žªóÓ2ÿ- _wandbruntime‚3168  _runtime‚3168`ÐþÇ€Á & lr-AdamW/pg1‚0.0001868225798319839 & lr-AdamW/pg2‚0.0001868225798319839 trainer/global_step‚7249 % _timestamp‚1.7858088899022343e+09 _runtime‚3168.193896693 _step‚576IÅÝÄÌÆ & lr-AdamW/pg1‚0.0001868225798319839 & lr-AdamW/pg2‚0.0001868225798319839 trainer/global_step‚7249 % _timestamp‚1.7858088899022343e+09 _runtime‚3168.193896693 _step‚576À1«Óí2‚- _wandbruntime‚3168  _runtime‚3168õóŸÂv$ƒÌO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20763375 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 % _timestamp‚1.7858088899031873e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚124463557448704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48166327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6379123418112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20763128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48166329 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130842593437184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49308773 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6363509888204 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124732924 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76566595 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3168.198962134 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6379035988480 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6379035988480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76556939 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124757446038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124723266 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28545647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131309051900928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124463574488064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124757446038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124723266 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28545647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131309051900928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚124463574488064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130842593437184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49308773 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÁÊj¹÷2…- _wandbruntime‚3168  _runtime‚3168ÓãUÜ&†ÖM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48172970 trainer/global_step‚7249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124750104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚3168.530856299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124480724211712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121387534169792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552664800256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6364301558302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6380003044864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130860639819776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49315655 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858088902388752e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49315906 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127751835728094 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28549584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20766073 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76577132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28549582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6380003044864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76567462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6379915608064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124750104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127751938570690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 2537553920¬þ3á&‡ÛM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48172970 trainer/global_step‚7249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124750104 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚3168.530856299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124480724211712 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121387534169792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6552664800256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6364301558302 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6380003044864 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130860639819776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49315655 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858088902388752e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49315906 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127751835728094 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28549584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20766073 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76577132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28549582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6380003044864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76567462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6379915608064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124750104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127751938570690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121ÂHžõ‰2ˆ- _wandbruntime‚3168  _runtime‚3168ª8Ûñ‰þ 2loss/SRqqt1/assignment_loss‚1.0192743992907989 2loss/FBt2/assignment_loss‚0.044807096148377926 /loss/FRt1/detection_loss‚2.2917763263333675 1loss/SRqqt2/detection_loss‚1.9155374890879582 /loss/FBt2/detection_loss‚0.8961268628947436 /loss/FRt2/assignment_loss‚2.861628573545268 /loss/FRt2/detection_loss‚2.7624046156104183 trainer/global_step‚7249 _runtime‚3168.531851005 _step‚579 0loss/FBt1/assignment_loss‚0.0988756871373031 0loss/SRqqt1/detection_loss‚1.182700122900357 % _timestamp‚1.7858088902400577e+09 /loss/FRt1/assignment_loss‚2.773074966530467 2loss/SRqqt2/assignment_loss‚1.0891653274333006 /loss/FBt1/detection_loss‚2.1015788465738297 %loss/total_loss‚19.03695031348619 epoch‚0‰Š¡š Šƒ 0loss/SRqqt1/detection_loss‚1.182700122900357 % _timestamp‚1.7858088902400577e+09 /loss/FRt1/assignment_loss‚2.773074966530467 2loss/SRqqt2/assignment_loss‚1.0891653274333006 /loss/FBt1/detection_loss‚2.1015788465738297 %loss/total_loss‚19.03695031348619 epoch‚0 2loss/SRqqt1/assignment_loss‚1.0192743992907989 2loss/FBt2/assignment_loss‚0.044807096148377926 /loss/FRt1/detection_loss‚2.2917763263333675 1loss/SRqqt2/detection_loss‚1.9155374890879582 /loss/FBt2/detection_loss‚0.8961268628947436 /loss/FRt2/assignment_loss‚2.861628573545268 /loss/FRt2/detection_loss‚2.7624046156104183 trainer/global_step‚7249 _runtime‚3168.531851005 _step‚579 0loss/FBt1/assignment_loss‚0.0988756871373031ÃLIÆå`‹‚(:‘ …ÅÓµ®‡Ó gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1GŒê<602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚117.688' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.38222222222223 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ÚŒˆŒ‚(:ý …ÅÓ™–…Ø disk.sda3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 32.41796875 disk.sda1.in‚0 proc.cpu.threads‚29 disk.sda.in‚0! disk.dm-0.out‚11778.189453125 disk.sda1.out‚0 disk.rbd4.out‚ 1.533203125) disk./.usagePercent‚83.37167831248354 disk.rbd2.in‚0 disk.drbd1743.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 154.78125 disk.dm-1.in‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1743.out‚ 3.56640625 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 cpu‚13.948635074170284% disk./.usageGB‚1242.2982559204102 disk.dm-1.out‚ 154.78125! disk.nvme0c0n1.in‚ 32.41796875) proc.memory.percent‚17.19306707382202 network.sent‚ 6.734122e+06 disk.sda2.out‚0 disk.nvme0n1.in‚ 32.41796875$ disk.nvme0n1.out‚11778.189453125& disk.nvme0c0n1.out‚11778.189453125# proc.memory.rssMB‚ 11267.6484375 network.recv‚421581 disk.rbd2.out‚ 1.533203125 disk.rbd1.in‚0 disk.rbd0.out‚0% memory_percent‚27.457964420318604) proc.memory.availableMB‚ 47541.1484375 disk.sda3.out‚ 154.78125 disk.rbd5.in‚0 disk.rbd5.out‚ 1.533203125ê&2- _wandbruntime‚3188  _runtime‚3188Ò¯/.ÇŽÁ & lr-AdamW/pg1‚0.0001881111891562492 & lr-AdamW/pg2‚0.0001881111891562492 trainer/global_step‚7299 % _timestamp‚1.7858089102797904e+09 _runtime‚3188.571429477 _step‚580\nÌÆ & lr-AdamW/pg1‚0.0001881111891562492 & lr-AdamW/pg2‚0.0001881111891562492 trainer/global_step‚7299 % _timestamp‚1.7858089102797904e+09 _runtime‚3188.571429477 _step‚580Äáý2- _wandbruntime‚3188  _runtime‚3188à–2ªÒ'‘ÌO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77094597 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77084941 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6423032123904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48498479 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77094597 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131745032192000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6423032123904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48498479 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131745032192000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125583418 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48498477 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131744927723008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 % _timestamp‚1.7858089102807527e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125321895599104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20907043 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28742495 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚581 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125617776974848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¦Î×'’ÑO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131745032192000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6423032123904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48498479 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131745032192000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125583418 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48498477 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131744927723008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 % _timestamp‚1.7858089102807527e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125321895599104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20907043 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28742495 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚581 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125617776974848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77094597 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77084941 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6423032123904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48498479 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77094597 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Åayx·2“- _wandbruntime‚3188  _runtime‚3188VG£Û&”ÕM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48505122 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6423999180288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125610256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6408275825046 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131763078581760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125610256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 trainer/global_step‚7299 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125339079401472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49655991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6408190021810 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3188.920210634 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77095464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 $ _timestamp‚1.785808910628282e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77095464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49656239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20909559 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132232697667584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚128632835650198 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132232855335424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128632732807602 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131762974105600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125339079401472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131763078581760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125339062362112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6597846340096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125600584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _step‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125600584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125339062362112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132232855335424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚128632732807602 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131762974105600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125339079401472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131763078581760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125339062362112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6597846340096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125600584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 _step‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125600584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125339062362112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125610256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6408275825046 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131763078581760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125610256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 trainer/global_step‚7299 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125339079401472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49655991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6408190021810 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3188.920210634 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77095464 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 $ _timestamp‚1.785808910628282e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77095464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49656239 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ÆýàÓh2–- _wandbruntime‚3188  _runtime‚3188å<Ò— _runtime‚ 3188.92147203 2loss/SRqqt2/assignment_loss‚1.3959148333718379 1loss/SRqqt1/detection_loss‚0.9757112169452011 /loss/FBt1/detection_loss‚1.8316766089863248 trainer/global_step‚7299 % _timestamp‚1.7858089106296325e+09 &loss/total_loss‚19.254142328335682 epoch‚0 0loss/FRt1/assignment_loss‚2.5931472488139806 0loss/FRt2/assignment_loss‚2.8247935814516887 1loss/SRqqt1/assignment_loss‚1.015011268829306 1loss/FBt1/assignment_loss‚0.19037773419100923 /loss/FRt2/detection_loss‚2.6267097407863256 /loss/FBt2/detection_loss‚1.3532709528627838 3loss/FBt2/assignment_loss‚0.0074115794745431415 _step‚583 .loss/FRt1/detection_loss‚2.561071035109068 1loss/SRqqt2/detection_loss‚1.8790465275136132¾ Dê ˜† .loss/FRt1/detection_loss‚2.561071035109068 1loss/SRqqt2/detection_loss‚1.8790465275136132 _runtime‚ 3188.92147203 2loss/SRqqt2/assignment_loss‚1.3959148333718379 1loss/SRqqt1/detection_loss‚0.9757112169452011 /loss/FBt1/detection_loss‚1.8316766089863248 trainer/global_step‚7299 % _timestamp‚1.7858089106296325e+09 &loss/total_loss‚19.254142328335682 epoch‚0 0loss/FRt1/assignment_loss‚2.5931472488139806 0loss/FRt2/assignment_loss‚2.8247935814516887 1loss/SRqqt1/assignment_loss‚1.015011268829306 1loss/FBt1/assignment_loss‚0.19037773419100923 /loss/FRt2/detection_loss‚2.6267097407863256 /loss/FBt2/detection_loss‚1.3532709528627838 3loss/FBt2/assignment_loss‚0.0074115794745431415 _step‚583Ç0õŒhœ™‚(:‘ ”ÅÓÓÙíÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚136.155' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚75.64166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰“ yŠš‚(:ÿ ”ÅÓО× disk.sda.in‚0# proc.memory.rssMB‚ 11267.6484375 cpu‚13.94891415788992" disk.dm-0.out‚11841.0224609375 disk.rbd0.in‚0 disk.rbd2.out‚ 1.533203125% memory_percent‚27.458685636520386 disk.sda1.out‚0( disk./.usagePercent‚83.3758732519396 disk.rbd5.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 1.533203125 network.recv‚423452% disk.nvme0n1.out‚11841.0224609375 disk.sda.out‚ 155.359375 disk.dm-1.out‚ 155.359375% disk./.usageGB‚1242.3607635498047 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.41796875 disk.rbd2.in‚0 disk.sda3.in‚0* proc.memory.availableMB‚47540.67578125 disk.rbd4.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.41796875 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 network.sent‚ 6.779141e+06 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 32.41796875' disk.nvme0c0n1.out‚11841.0224609375 disk.sda3.out‚ 155.359375 disk.rbd1.out‚ 1.533203125 disk.drbd1743.out‚3.59375 disk.rbd5.out‚ 1.533203125ÈêÒ2›- _wandbruntime‚3209  _runtime‚3209.ìɜà ' lr-AdamW/pg2‚0.00018939979848051452 trainer/global_step‚7349 % _timestamp‚1.7858089309001124e+09 _runtime‚3209.191877754 _step‚584 ' lr-AdamW/pg1‚0.00018939979848051452‚)UïÎÈ ' lr-AdamW/pg1‚0.00018939979848051452 ' lr-AdamW/pg2‚0.00018939979848051452 trainer/global_step‚7349 % _timestamp‚1.7858089309001124e+09 _runtime‚3209.191877754 _step‚584ÈTA 2ž- _wandbruntime‚3209  _runtime‚3209ƒóâñÑ'ŸËO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77612943 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6467028259328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7349 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48830629 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6641975510528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6467028259328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133120108325376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6642116670464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132647262008832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123044811229472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126180233749504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126180250788864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133120274388992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126478157718528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126443570 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126180250788864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126453228 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126478132814848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129496012241568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 $ _timestamp‚1.785808930901222e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123044828268832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28939345 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77622599 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132647262008832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132647262008832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77612943 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6467028259328 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚7349 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48830629 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6641975510528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6467028259328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚133120108325376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6642116670464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132647262008832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123044811229472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126180233749504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126180250788864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133120274388992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126478157718528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126443570 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126180250788864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 N@DeviceStatsMoni•¶mÿtor.on_train_batch_start/allocation.all.allocated‚ 126453228 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126478132814848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129496012241568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 $ _timestamp‚1.785808930901222e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123044828268832ÉwØ}Ä2¡- _wandbruntime‚3209  _runtime‚3209ÖÄ}Ü&¢ÖM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6467907878912 _runtime‚3209.593266044 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126197400512512 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77623466 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126470408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6467995315712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48837272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133138222520320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858089313011806e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126495339931648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7349 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48837270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49994966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6467995315712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21051684 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚132665412867584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6467907878912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126470408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77633136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126197417551872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28943282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77633136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123061568441152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6452164288554 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126197417551872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123061551401792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126460736 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48837272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132665412867584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49995206 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6452078485318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126460736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21051922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77623466 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚132665308391424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6642882588672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6467995315712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48837272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133138222520320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858089313011806e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126495339931648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7349 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48837270 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49994966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6467995315712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21051684 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚132665412867584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6467907878912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126470408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77633136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126197417551872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28943282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77633136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123061568441152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6452164288554 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126197417551872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123061551401792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126460736 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48837272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132665412867584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49995206 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6452078485318 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126460736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21051922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77623466 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚132665308391424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6642882588672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392ÊëAÖ2¤- _wandbruntime‚3209  _runtime‚3209j-—*¥ÿ 1loss/SRqqt1/detection_loss‚1.1203448868566943 /loss/FBt1/assignment_loss‚0.038748472910027 /loss/FRt1/detection_loss‚2.3554688809686395 /loss/FRt2/detection_loss‚2.5979789033895586 epoch‚0 % _timestamp‚1.7858089313026538e+09 _runtime‚ 3209.5942072 _step‚587 0loss/FRt1/assignment_loss‚2.5567600482917694 0loss/FRt2/assignment_loss‚2.9169068874382393 2loss/SRqqt1/assignment_loss‚0.9973855256959195 &loss/total_loss‚18.928154711569963 2loss/SRqqt2/assignment_loss‚0.9872184857016518 2loss/FBt2/assignment_loss‚0.011452278180766282 1loss/SRqqt2/detection_loss‚1.9351570935298998 /loss/FBt1/detection_loss‚2.5615404231680765 /loss/FBt2/detection_loss‚0.8491928254387209 trainer/global_step‚7349³Ûê% ¦„ _step‚587 0loss/FRt1/assignment_loss‚2.5567600482917694 0loss/FRt2/assignment_loss‚2.9169068874382393 2loss/SRqqt1/assignment_loss‚0.9973855256959195 &loss/total_loss‚18.928154711569963 2loss/SRqqt2/assignment_loss‚0.9872184857016518 2loss/FBt2/assignment_loss‚0.011452278180766282 1loss/SRqqt2/detection_loss‚1.9351570935298998 /loss/FBt1/detection_loss‚2.5615404231680765 /loss/FBt2/detection_loss‚0.8491928254387209 trainer/global_step‚7349 1loss/SRqqt1/detection_loss‚1.1203448868566943 /loss/FBt1/assignment_loss‚0.038748472910027 /loss/FRt1/detection_loss‚2.3554688809686395 /loss/FRt2/detection_loss‚2.5979789033895586 epoch‚0 % _timestamp‚1.7858089313026538e+09 _runtime‚ 3209.5942072Ë5¯fœ§‚(:‘ £ÅÓŽ½îÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.86055555555556 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dæ:É’¨‚(:‡  £ÅÓÖò”Ö network.sent‚ 6.82392e+06! disk.nvme0c0n1.in‚ 32.41796875% disk.nvme0n1.out‚11841.2685546875 cpu‚13.949117360695375 network.recv‚425308 disk.sda2.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.41796875% memory_percent‚27.459919452667236 disk.sda1.out‚0 disk.sda3.out‚ 156.17578125 disk.dm-1.in‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1743.in‚0) proc.memory.percent‚17.19306707382202 disk.rbd2.out‚ 1.533203125 disk.sda3.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29( disk./.usagePercent‚83.3758545633896% disk./.usageGB‚1242.3604850769043 disk.sda1.in‚0 disk.rbd4.out‚ 1.533203125' disk.nvme0c0n1.out‚11841.2685546875 disk.sda.in‚0 disk.sda.out‚ 156.17578125 disk.rbd1.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 1.533203125 disk.rbd0.in‚0 disk.dm-1.out‚ 156.17578125 disk.drbd1743.out‚ 3.61328125 disk.dm-0.in‚ 32.41796875" disk.dm-0.out‚11841.2685546875 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd4.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 47539.8671875-Djœ©‚(:‘ ²ÅÓ÷ÖÑÒ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚118.226' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚65.68111111111111 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä碾ˆª‚(:ý ²ÅÓ¾ÔÚÕ network.sent‚ 6.868792e+06 disk.sda3.out‚ 156.76953125 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1743.in‚0' proc.memory.availableMB‚ 47540.03125" disk.nvme0n1.out‚ 11842.2265625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125 cpu‚13.949689493414985! disk.nvme0c0n1.in‚ 32.41796875 disk.sda.out‚ 156.76953125 disk.rbd4.in‚0 disk.rbd2.out‚ 1.533203125 disk.sda3.in‚0 disk.dm-0.in‚ 32.41796875# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29 network.recv‚427250) disk./.usagePercent‚83.37585481939713% disk./.usageGB‚1242.3604888916016 disk.rbd5.in‚0 disk.sda.in‚0 disk.rbd2.in‚0 disk.drbd1743.out‚ 3.62109375) proc.memory.percent‚17.19306707382202 disk.nvme0n1.in‚ 32.41796875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 156.76953125 disk.sda2.out‚0 disk.dm-0.out‚ 11842.2265625$ memory_percent‚27.45966911315918 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd5.out‚ 1.533203125 disk.rbd4.out‚ 1.533203125$ disk.nvme0c0n1.out‚ 11842.2265625ñŽ©v2«- _wandbruntime‚3229  _runtime‚3229vÖs<ǬÁ & lr-AdamW/pg1‚0.0001906884078047798 & lr-AdamW/pg2‚0.0001906884078047798 trainer/global_step‚7399 % _timestamp‚1.7858089510668206e+09 _runtime‚3229.358499519 _step‚588ûuRÌ­Æ & lr-AdamW/pg1‚0.0001906884078047798 & lr-AdamW/pg2‚0.0001906884078047798 trainer/global_step‚7399 % _timestamp‚1.7858089510668206e+09 _runtime‚3229.358499519 _step‚588Ìm©2®- _wandbruntime‚3229  _runtime‚3229MŽ™¯ÌO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49162777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134025664016896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49162779 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133549700763648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49162779 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127038571899904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78150601 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6511111824384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127313380 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚3229.362959546 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78150601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133549596294656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6687175362048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6511024394752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133549596294656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6511024394752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856aÔþ×'°ÑO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49162779 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133549700763648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49162779 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127038571899904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78150601 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6511111824384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127313380 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 _runtime‚3229.362959546 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78150601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133549596294656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6687175362048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6511024394752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133549596294656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6511024394752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49162777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134025664016896Íõg0Š2±- _wandbruntime‚3229  _runtime‚3229`òþÜ&²ÖM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78151468 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133567747153408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6495966948826 _runtime‚3229.721547996 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6511904014336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6688210348544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127330560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49169422 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6688071292928 trainer/global_step‚7399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123898577057152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127055738662912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858089514295065e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127055755702272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127330560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133567747153408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127320888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134043931023872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127355720675328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78151468 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127355695771648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29140132 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49169420 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127055738662912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21195421 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123898560017792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29140134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50335553 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130394629809214 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130394526966618 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49169422 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127320888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50335815 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134043767064576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6496052752062 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78161138 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21195681 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0׈>"³ÛM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134043767064576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6496052752062 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78161138 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21195681 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78151468 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133567747153408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6495966948826 _runtime‚3229.721547996 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6511904014336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6688210348544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127330560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49169422 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6688071292928 trainer/global_step‚7399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123898577057152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127055738662912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858089514295065e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127055755702272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127330560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133567747153408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127320888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134043931023872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127355720675328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78151468 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127355695771648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29140132 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49169420 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127055738662912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21195421 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123898560017792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29140134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 C9Device•Vå£StatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50335553 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130394629809214 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130394526966618 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49169422 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127320888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50335815Îiþ›U2´- _wandbruntime‚3229  _runtime‚3229Ô Á¨µü /loss/FRt1/assignment_loss‚2.701442199803534 0loss/FBt1/assignment_loss‚0.5026201520592062 4loss/FBt2/assignment_loss‚0.00023439304828561566 /loss/FRt2/detection_loss‚2.4996554820161117 &loss/total_loss‚20.284277305488615 trainer/global_step‚7399 $ _timestamp‚1.785808951430888e+09 _runtime‚3229.722368796 _step‚591 epoch‚0 2loss/SRqqt1/assignment_loss‚1.0090255574212366 2loss/SRqqt2/assignment_loss‚0.7021889723837376 .loss/FBt1/detection_loss‚3.408884061234338 /loss/FRt2/assignment_loss‚2.794472630400407 -loss/FRt1/detection_loss‚2.56886891896526 0loss/SRqqt1/detection_loss‚0.989453842830179 1loss/SRqqt2/detection_loss‚2.1032274630852044 .loss/FBt2/detection_loss‚1.0042036322411156h¶ 2loss/SRqqt1/assignment_loss‚1.0090255574212366 2loss/SRqqt2/assignment_loss‚0.7021889723837376 .loss/FBt1/detection_loss‚3.408884061234338 /loss/FRt2/assignment_loss‚2.794472630400407 -loss/FRt1/detection_loss‚2.56886891896526 0loss/SRqqt1/detection_loss‚0.989453842830179 1loss/SRqqt2/detection_loss‚2.1032274630852044 .loss/FBt2/detection_loss‚1.004203632241115 /loss/FRt1/assignment_loss‚2.701442199803534 0loss/FBt1/assignment_loss‚0.5026201520592062 4loss/FBt2/assignment_loss‚0.00023439304828561566 /loss/FRt2/detection_loss‚2.4996554820161117 &loss/total_loss‚20.284277305488615 trainer/global_step‚7399 $ _timestamp‚1.785808951430888e+09 _runtime‚3229.722368796 _step‚591 epoch‚0ψr“›·‚(: ÁÅÓŽ¯ËÒ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚133.265' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚74.0361111111111 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_›ãR¢¸‚(:—  ÁÅÓ³µ¥Ö% disk.nvme0c0n1.out‚11904.73828125 disk.dm-1.out‚ 157.359375 disk.rbd2.out‚ 1.533203125 disk.dm-1.in‚ 0.00390625 disk.sda2.in‚0( proc.memory.availableMB‚ 47540.484375% disk./.usageGB‚1242.4178924560547 disk.drbd1743.out‚ 3.62890625 disk.rbd1.out‚ 1.533203125 disk.rbd5.in‚0 disk.sda.in‚ 0.00390625 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.dm-0.in‚ 32.41796875 disk.dm-0.out‚11904.73828125 disk.nvme0n1.in‚ 32.41796875 disk.rbd1.in‚0 cpu‚13.949760760601439) disk./.usagePercent‚83.37970722077237# disk.nvme0n1.out‚11904.73828125 disk.sda1.in‚0 disk.rbd4.in‚0 disk.sda3.out‚ 157.359375% memory_percent‚27.458977699279785# proc.memory.rssMB‚ 11267.6484375 disk.rbd3.in‚0 disk.rbd5.out‚ 1.533203125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 32.41796875 disk.sda.out‚ 157.359375 disk.rbd2.in‚0 disk.sda3.in‚ 0.00390625) proc.memory.percent‚17.19306707382202 network.sent‚ 6.871225e+06 network.recv‚428454 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd4.out‚ 1.533203125$OŒ2¹- _wandbruntime‚3250  _runtime‚32501O¹ ɺà ' lr-AdamW/pg1‚0.00019197701712904514 ' lr-AdamW/pg2‚0.00019197701712904514 trainer/global_step‚7449 % _timestamp‚1.7858089717099009e+09 _runtime‚3250.001552279 _step‚592dŽxŸÎ»È ' lr-AdamW/pg2‚0.00019197701712904514 trainer/global_step‚7449 % _timestamp‚1.7858089717099009e+09 _runtime‚3250.001552279 _step‚592 ' lr-AdamW/pg1‚0.00019197701712904514иֺS2¼- _wandbruntime‚3250  _runtime‚3250—Ì’…Ò'½ÌO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858089717108254e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131257806400584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6555020530176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚593 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50670982 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50670748 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127896910050304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127896927089664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49494927 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6555107959808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131257909243068 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127896910050304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21337935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127896927089664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29333047 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49494929 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚134451930580480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78668947 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚134452035049472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚134452035049472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49494929 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78678603 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚124718845500832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128173532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78668947 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29333045 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6732516608512 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78678603 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134931238817792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49494927 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6555020530176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6555107959808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128198844494848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200€jV×'¾ÑO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29333047 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49494929 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚134451930580480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 122285568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78668947 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚134452035049472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚134452035049472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49494929 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78678603 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚124718845500832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128173532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78668947 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29333045 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6732516608512 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78678603 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134931238817792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49494927 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6555020530176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6555107959808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128198844494848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858089717108254e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131257806400584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6555020530176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 147189248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚593 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50670982 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50670748 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127896910050304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127896927089664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49494927 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6555107959808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131257909243068 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127896910050304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21337935 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127896927089664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632ÑÌ¡#2¿- _wandbruntime‚3250  _runtime‚32504´ûƒ­ÀÖM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128190712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128181040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49501570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49501572 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134949505994752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚134470081439232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 trainer/global_step‚7449 _runtime‚3250.337530428 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128181040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6539855412334 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚594 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128216051611648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124735585673152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6555900149760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21340771 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6733429479424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78679470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50677487 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29336982 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50677755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134949346229760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128216076515328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/acti-êi/ ve.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127914076813312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6555987586560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49501572 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78679470 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21340505 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128190712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29336984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 % _timestamp‚1.7858089720455644e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49501570 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131275424046126 Ò³á&ÁÛM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49501572 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78679470 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21340505 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128190712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29336984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 % _timestamp‚1.7858089720455644e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49501570 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131275424046126 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128190712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128181040 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49501570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49501572 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134949505994752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚134470081439232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 trainer/global_step‚7449 _runtime‚3250.337530428 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128181040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6539855412334 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚594 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128216051611648 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124735585673152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6555900149760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21340771 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6733429479424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78679470 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50677487 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29336982 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50677755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134949346229760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128216076515328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127914076813312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6555987586560Ò¿L²2Â- _wandbruntime‚3250  _runtime‚3250Zk~Ãÿ 0loss/FRt1/assignment_loss‚2.2788580565512935 0loss/FRt2/assignment_loss‚3.1388163534884757 1loss/SRqqt1/assignment_loss‚1.138595750288072 /loss/FRt2/detection_loss‚2.4268372721494513 /loss/FBt1/detection_loss‚1.8737298721447586 1loss/SRqqt1/detection_loss‚0.9257313202100771 0loss/SRqqt2/detection_loss‚1.912717049404119 %loss/total_loss‚17.82666374398888 epoch‚0 trainer/global_step‚7449 _runtime‚3250.338537898 2loss/SRqqt2/assignment_loss‚1.0545544628627026 2loss/FBt1/assignment_loss‚0.036105908328489277 /loss/FBt2/detection_loss‚0.6265457103621553 % _timestamp‚1.7858089720467496e+09 0loss/FBt2/assignment_loss‚0.0402532102781324 /loss/FRt1/detection_loss‚2.3739187779211544 _step‚595èi– Ä„ /loss/FBt1/detection_loss‚1.8737298721447586 1loss/SRqqt1/detection_loss‚0.9257313202100771 0loss/SRqqt2/detection_loss‚1.912717049404119 %loss/total_loss‚17.82666374398888 epoch‚0 trainer/global_step‚7449 _runtime‚3250.338537898 2loss/SRqqt2/assignment_loss‚1.0545544628627026 2loss/FBt1/assignment_loss‚0.036105908328489277 /loss/FBt2/detection_loss‚0.6265457103621553 % _timestamp‚1.7858089720467496e+09 0loss/FBt2/assignment_loss‚0.0402532102781324 /loss/FRt1/detection_loss‚2.3739187779211544 _step‚595 0loss/FRt1/assignment_loss‚2.2788580565512935 0loss/FRt2/assignment_loss‚3.1388163534884757 1loss/SRqqt1/assignment_loss‚1.138595750288072 /loss/FRt2/detection_loss‚2.4268372721494513Óá-ћł(: ÐÅÓ–ÅÒ gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚147.121' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚81.7338888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\:†ë®Æ‚(:£  ÐÅÓÏü£Õ cpu‚13.950143253755147 disk.sda3.out‚ 157.91796875 disk.nvme0n1.in‚ 32.41796875 disk.rbd5.in‚0 disk.rbd5.out‚ 1.533203125 disk.dm-1.in‚ 0.00390625% memory_percent‚27.460122108459473) proc.memory.percent‚17.19306707382202 disk.sda1.in‚0 network.sent‚ 6.916219e+06 disk.dm-0.in‚ 32.41796875 disk.rbd4.out‚ 1.533203125 disk.sda.in‚ 0.00390625 disk.rbd2.out‚ 1.533203125 disk.sda2.out‚0( proc.memory.availableMB‚ 47539.734375 disk.rbd0.in‚0 disk.rbd1.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375( disk.nvme0c0n1.out‚11905.03466796875 disk.sda1.out‚0 network.recv‚430463% disk./.usageGB‚1242.4178771972656# disk.dm-0.out‚11905.03466796875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0& disk.nvme0n1.out‚11905.03466796875 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 proc.cpu.threads‚29 disk.drbd1743.out‚3.65625 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.41796875 disk.rbd1.in‚0) disk./.usagePercent‚83.37970619674225 disk.drbd1743.in‚0 disk.sda.out‚ 157.91796875 disk.rbd2.in‚0 disk.dm-1.out‚ 157.91796875ý·@»œÇ‚(:‘ ßÅÓ¾²ÐÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.80666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À \°§È‚(:œ  ßÅÓÀÓÕ disk.sda3.out‚ 159.140625 disk.sda3.in‚ 0.00390625 disk.rbd1.out‚ 1.533203125$ memory_percent‚27.45962142944336 disk.rbd0.out‚0 disk.dm-1.in‚ 0.00390625 network.sent‚ 6.960948e+06 disk.drbd1743.out‚ 3.67578125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd4.in‚0 disk.rbd2.in‚0& proc.memory.availableMB‚ 47540.0625) proc.memory.percent‚17.19306707382202( disk.nvme0c0n1.out‚11905.31201171875# proc.memory.rssMB‚ 11267.6484375& disk.nvme0n1.out‚11905.31201171875 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1242.4170455932617 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 32.41796875 disk.rbd5.in‚0 disk.dm-0.in‚ 32.41796875 disk.rbd5.out‚ 1.533203125 disk.rbd0.in‚0 disk.sda.out‚ 159.140625 disk.rbd1.in‚0 cpu‚13.95047335102595 proc.cpu.threads‚29 network.recv‚432244! disk.nvme0c0n1.in‚ 32.41796875 disk.sda.in‚ 0.00390625 disk.sda2.out‚0) disk./.usagePercent‚83.37965038709979# disk.dm-0.out‚11905.31201171875 disk.rbd4.out‚ 1.533203125 disk.rbd2.out‚ 1.533203125 disk.dm-1.out‚ 159.140625 disk.sda2.in‚0*5>'2É- _wandbruntime‚3270  _runtime‚3270Y*¶ÈÊ % _timestamp‚1.7858089917612147e+09 _runtime‚ 3270.05290755 _step‚596 ' lr-AdamW/pg1‚0.00019326562645331042 ' lr-AdamW/pg2‚0.00019326562645331042 trainer/global_step‚7499|f_ÍËÇ % _timestamp‚1.7858089917612147e+09 _runtime‚ 3270.05290755 _step‚596 ' lr-AdamW/pg1‚0.00019326562645331042 ' lr-AdamW/pg2‚0.00019326562645331042 trainer/global_step‚7499Ô¶¬•ø2Ì- _wandbruntime‚3270  _runtime‚3270Í(LÒ'ÍÌO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129033684 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125555854116832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135354264866304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51013494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6599104095232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129024026 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21483339 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49827077 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49827079 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6599016665600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51013234 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21483597 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚132138806322576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79206605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6777616691712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚132138703480092 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128755248200704 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 trainer/global_step‚7499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135354369335296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135354264866304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79196949 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135836976798720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129059200334848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6777751560192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129059225238528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6582952205744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49827079 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79206605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6582866402620 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128755265240064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚135836817026560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7858089917622461e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6599016665600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128755248200704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6599104095232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125555837077472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79196949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125555854116832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135354264866304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51013494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6599104095232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129024026 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21483339 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49827077 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49827079 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6599016665600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51013234 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21483597 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚132138806322576 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79206605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6777616691712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚132138703480092 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚128755248200704 NEDeviceStatsMonitor.on_train_batch_start/allocatº‰@èion.small_pool.current‚9656 trainer/global_step‚7499 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135354369335296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135354264866304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79196949 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135836976798720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129059200334848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6777751560192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129059225238528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6582952205744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49827079 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79206605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6582866402620 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚128755265240064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚135836817026560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7858089917622461e+09 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6599016665600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚128755248200704 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6599104095232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125555837077472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79196949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6599983721984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49833720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49833722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79207472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51020256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129041192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51020003 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6778517920768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125572577249792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858089920977619e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79217142 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135372311248896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129050864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135854925372416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6583829679078 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79217142 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚3270.389785496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128772432003072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21486422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135855087234560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128772432003072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128772414963712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129076407451648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128772414963712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚132156423968230 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49833722 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _step‚598 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6599983721984 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚132156321125634 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49833720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29533834 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6583743875842 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29533832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125572594289152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129050864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129041192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129076432355328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79207472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51020256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129041192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51020003 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6778517920768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125572577249792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7858089920977619e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79217142 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135372311248896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129050864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135854925372416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6583829679078 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79217142 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _runtime‚3270.389785496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚128772432003072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21486422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135855087234560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚128772432003072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚128772414963712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129076407451648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚128772414963712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚132156423968230 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49833722 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _step‚598 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6599983721984 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚132156321125634 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49833720 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29533834 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6583743875842 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29533832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125572594289152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129050864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129041192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129076432355328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6599983721984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49833720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49833722 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121Ö=óÿT2Ò- _wandbruntime‚3270  _runtime‚3270µÓÓ‚ 2loss/SRqqt2/assignment_loss‚1.0389219358125155 2loss/FBt1/assignment_loss‚0.021078423522979506 1loss/SRqqt2/detection_loss‚1.4913131358722846 trainer/global_step‚7499 epoch‚0 _runtime‚3270.390488204 2loss/SRqqt1/assignment_loss‚1.0134960529779822 2loss/FBt2/assignment_loss‚0.016366542827398974 /loss/FRt2/detection_loss‚2.4688688533173666 .loss/FBt2/detection_loss‚1.163836834167596 %loss/total_loss‚18.29012808307386 % _timestamp‚1.7858089920990472e+09 _step‚599 0loss/FRt2/assignment_loss‚3.1832690040270486 /loss/FRt1/detection_loss‚2.6515866354317734 1loss/SRqqt1/detection_loss‚0.9780767338671561 /loss/FBt1/detection_loss‚1.4471982060414221 0loss/FRt1/assignment_loss‚2.8161157252083364m,… Ô‡ 2loss/SRqqt1/assignment_loss‚1.0134960529779822 2loss/FBt2/assignment_loss‚0.016366542827398974 /loss/FRt2/detection_loss‚2.4688688533173666 .loss/FBt2/detection_loss‚1.163836834167596 %loss/total_loss‚18.29012808307386 % _timestamp‚1.7858089920990472e+09 _step‚599 0loss/FRt2/assignment_loss‚3.1832690040270486 /loss/FRt1/detection_loss‚2.6515866354317734 1loss/SRqqt1/detection_loss‚0.9780767338671561 /loss/FBt1/detection_loss‚1.4471982060414221 0loss/FRt1/assignment_loss‚2.8161157252083364 2loss/SRqqt2/assignment_loss‚1.0389219358125155 2loss/FBt1/assignment_loss‚0.021078423522979506 1loss/SRqqt2/detection_loss‚1.4913131358722846 trainer/global_step‚7499 epoch‚0 _runtime‚3270.390488204×ô óUœÕ‚(:‘ îÅÓ×ÓÎÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚99.716' gpu.0.enforcedPowerLimitWatts‚180.0) gpu.0.powerPercent‚55.397777777777776 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i]Í¢®Ö‚(:£  îÅÓ¬õÕ× disk.drbd1743.in‚0 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.drbd1743.out‚ 3.68359375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 159.78515625$ memory_percent‚27.45969295501709 disk.nvme0n1.in‚ 32.41796875 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11967.79833984375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375! disk.nvme0c0n1.in‚ 32.41796875 disk.rbd2.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 47540.015625) proc.memory.percent‚17.19306707382202 cpu‚13.95052804602853 disk.rbd5.out‚ 1.533203125 disk.sda.in‚ 0.00390625 disk.sda2.out‚0 proc.cpu.threads‚29% disk./.usageGB‚1242.4797668457031 disk.rbd4.in‚0 disk.sda3.out‚ 159.78515625 disk.rbd2.out‚ 1.533203125& disk.nvme0n1.out‚11967.79833984375 disk.sda1.out‚0 disk.rbd4.out‚ 1.533203125 disk.sda.out‚ 159.78515625 network.sent‚ 6.99499e+06 network.recv‚433861) disk./.usagePercent‚83.38385966297776 disk.dm-0.in‚ 32.41796875# disk.dm-0.out‚11967.79833984375O@xÔ2×- _wandbruntime‚3291  _runtime‚3291€åi…ÇØÁ & lr-AdamW/pg2‚0.0001945542357775757 trainer/global_step‚7549 % _timestamp‚1.7858090130007367e+09 _runtime‚3291.292405577 _step‚600 & lr-AdamW/pg1‚0.0001945542357775757£SiõÌÙÆ _step‚600 & lr-AdamW/pg1‚0.0001945542357775757 & lr-AdamW/pg2‚0.0001945542357775757 trainer/global_step‚7549 % _timestamp‚1.7858090130007367e+09 _runtime‚3291.292405577ØÍ´4ª2Ú- _wandbruntime‚3291  _runtime‚3291ãeK»? ÛÌO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21626799 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6626754866128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136256599152128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136256703621120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129613603390464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136742376143360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133019600559600 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚7549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6643012801024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6643100230656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29726747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚3291.296965993 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79734607 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6822819968512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79734607 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51353801 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126392862732832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129613603390464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129613586351104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ž“129613586351104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21627054 % _timestamp‚1.7858090130016968e+09 _step‚601 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50159227 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50159227 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51353544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6626840669252 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136256703621120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6643012801024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129884178 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129893836 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129919581078528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6822950642688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50159229 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136256599152128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126392845693472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79724951 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136742531721216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29726745 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50159229 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658æùâ×'ÜÑO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚129919556174848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136256599152128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126392845693472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79724951 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136742531721216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29726745 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50159229 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21626799 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6626754866128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136256599152128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136256703621120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129613603390464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136742376143360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133019600559600 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 trainer/global_step‚7549 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6643012801024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6643100230656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29726747 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _runtime‚3291.296965993 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79734607 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6822819968512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79734607 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51353801 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126392862732832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129613603390464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129613586351104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129613586351104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21627054 % _timestamp‚1.7858090130016968e+09 _step‚601 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50159227 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50159227 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51353544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6626840669252 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136256703621120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6643012801024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129884178 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129893836 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129919581078528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6822950642688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50159229 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126409602905152 _runtime‚3291.625825007 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50165870 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79745144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133037321047738 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6627632339350 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136274750010880 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136274750010880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6643892420608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21629639 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50165872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6823851758080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6643979857408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51360321 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51360577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136760486479872 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136274645534720 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129911016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79735474 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129901344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129936763291648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79735474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126409585865792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.785809013333825e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29730684 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29730682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129630770153472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50165870 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21629893 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712ó$øfßÚM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133037218205142 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79745144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6823723188224 trainer/global_step‚7549 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129911016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126409602905152 _runtime‚3291.625825007 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50165870 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79745144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133037321047738 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6627632339350 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136274750010880 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136274750010880 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6643892420608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21629639 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50165872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6823851758080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6643979857408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51360321 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51360577 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136760486479872 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136274645534720 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129911016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79735474 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129901344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129936763291648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79735474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126409585865792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.785809013333825e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29730684 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29730682 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129630770153472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50165870 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21629893 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856Úr㣼2à- _wandbruntime‚3291  _runtime‚3291<Ï0ò á… &loss/total_loss‚17.903296085422973 epoch‚0 .loss/FRt1/detection_loss‚2.692198399087073 1loss/SRqqt1/detection_loss‚1.0324212848441676 /loss/FBt1/detection_loss‚1.3085605007181584 trainer/global_step‚7549 2loss/SRqqt1/assignment_loss‚0.9494611921225442 4loss/FBt2/assignment_loss‚3.5992192326170145e-06 _step‚603 0loss/FRt1/assignment_loss‚2.7034378337188505 0loss/FRt2/assignment_loss‚3.3986173559118202 3loss/FBt1/assignment_loss‚0.0001690277421908526 1loss/SRqqt2/detection_loss‚1.6220633427740312 /loss/FBt2/detection_loss‚0.7806821917934764 _runtime‚3291.626512617 2loss/SRqqt2/assignment_loss‚0.8150512286662763 /loss/FRt2/detection_loss‚2.6006301288251525 $ _timestamp‚1.785809013334982e+09ÚüC5⊠2loss/SRqqt1/assignment_loss‚0.9494611921225442 4loss/FBt2/assignment_loss‚3.5992192326170145e-06 _step‚603 0loss/FRt1/assignment_loss‚2.7034378337188505 0loss/FRt2/assignment_loss‚3.3986173559118202 3loss/FBt1/assignment_loss‚0.0001690277421908526 1loss/SRqqt2/detection_loss‚1.6220633427740312 /loss/FBt2/detection_loss‚0.7806821917934764 _runtime‚3291.626512617 2loss/SRqqt2/assignment_loss‚0.8150512286662763 /loss/FRt2/detection_loss‚2.6006301288251525 $ _timestamp‚1.785809013334982e+09 &loss/total_loss‚17.903296085422973 epoch‚0 .loss/FRt1/detection_loss‚2.692198399087073 1loss/SRqqt1/detection_loss‚1.0324212848441676 /loss/FBt1/detection_loss‚1.3085605007181584 trainer/global_step‚7549Û]GÈ›ã‚(: ýÅÓ´‡æÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚155.51' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.39444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ‰®ä‚(:£  ýÅÓéæ˜Ø disk.drbd1743.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 160.5390625# disk.dm-0.out‚11968.12646484375$ memory_percent‚27.46022343635559) proc.memory.percent‚17.19306707382202) disk./.usagePercent‚83.38386017499282 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚11968.12646484375 disk.sda.in‚ 0.00390625& disk.nvme0n1.out‚11968.12646484375 disk.rbd5.in‚0 disk.rbd5.out‚ 1.533203125* proc.memory.availableMB‚47539.66796875# proc.memory.rssMB‚ 11267.6484375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125 disk.drbd1743.out‚ 3.70703125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.41796875 network.sent‚ 7.029491e+06 disk.rbd2.out‚ 1.533203125 disk.rbd0.in‚0 proc.cpu.threads‚29 disk.rbd2.in‚0 disk.sda3.in‚ 0.00390625 disk.dm-1.out‚ 160.5390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 32.41796875 disk.rbd4.in‚0 disk.rbd4.out‚ 1.533203125 disk.sda3.out‚ 160.5390625 cpu‚13.95099908763821 network.recv‚435814% disk./.usageGB‚1242.4797744750977! disk.nvme0c0n1.in‚ 32.41796875g÷ɘ2å- _wandbruntime‚3311  _runtime‚3311¬˜ïNÉæà % _timestamp‚1.7858090331128442e+09 _runtime‚3311.404563858 _step‚604 ' lr-AdamW/pg1‚0.00019584284510184105 ' lr-AdamW/pg2‚0.00019584284510184105 trainer/global_step‚7599#(s3ÎçÈ ' lr-AdamW/pg1‚0.00019584284510184105 ' lr-AdamW/pg2‚0.00019584284510184105 trainer/global_step‚7599 % _timestamp‚1.7858090331128442e+09 _runtime‚3311.404563858 _step‚604Üå…æ2è- _wandbruntime‚3311  _runtime‚3311ö3¿ÀÑ'éËO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50491377 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137647949455872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130471924501504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21771316 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80252953 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29923595 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 % _timestamp‚1.7858090331139314e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6670729132760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21771058 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6687096366080 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130471941540864 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137158933437952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80262609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127229854309472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 3311.40811981 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130753988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670643329636 _step‚605 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50491379 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50491377 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51694653 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6687096366080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137159037906944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130779936918528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6868166018048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51694913 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80262609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6687008936448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133900497639108 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137158933437952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80262609 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127229854309472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚ 3311.40811981 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130753988 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670643329636 _step‚605 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50491379 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50491377 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51694653 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6687096366080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137159037906944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130779936918528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6868166018048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51694913 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80262609 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6687008936448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133900497639108 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 % _timestamp‚1.7858090331139314e+09Ý‘t¬2ë- _wandbruntime‚3311  _runtime‚3311~¤ÌºìÕM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80273146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130489091264512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6671606606094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137666061561856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130761496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6687888556032 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29927532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 $ _timestamp‚1.785809033509801e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130771168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51701644 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80273146 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137176979820544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50498020 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137177084296704 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6868942430208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127246611521152 >6DeviceSâtDý!tatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50498022 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80263476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51701379 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133918218127246 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50498022 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚606 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137666212938240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133918115284650 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130771168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚3311.802143563 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137177084296704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21773847 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6869068902912 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130797119131648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80263476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29927534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21774110 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130761496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6687975992832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130797144035328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÀK+ªà&íÚM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130489108303872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29927532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 126472704 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 $ _timestamp‚1.785809033509801e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130771168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51701644 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80273146 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137176979820544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50498020 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137177084296704 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6868942430208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127246611521152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50498022 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80263476 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51701379 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133918218127246 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50498022 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚606 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137666212938240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133918115284650 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130771168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _runtime‚3311.802143563 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137177084296704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21773847 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6869068902912 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130797119131648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80263476 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29927534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21774110 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130761496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6687975992832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130797144035328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80273146 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130489091264512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6671606606094 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137666061561856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130761496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6687888556032 Eâá‚Ò'ùÌO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128066862925472 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131330279691264 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80790611 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131604482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138061267723776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138061267723776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50823527 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131330262651904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21914514 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6714531793144 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50823527 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50823529 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131614140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128066879964832 _step‚609 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131330262651904 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6913387155968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6714617596268 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80790611 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131604482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138061267723776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138061267723776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50823527 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131330262651904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21914514 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6714531793144 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50823527 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50823529 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131614140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128066879964832 _step‚609 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131330262651904 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6913387155968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6714617596268 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6914290122752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131631320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131631320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138571630233088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134799012364158 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6731972128256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52041770 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131657499875328 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80791478 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131657474971648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131347429414912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138571789998080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚610 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30124384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128083603097792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6731884691456 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131621648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50830170 _runtime‚3333.413486464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80791478 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7858090551215048e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6715495069602 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50830172 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52042021 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131347446454272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80801148 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚7649 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0·½ºxá&ýÛM % _timestamp‚1.7858090551215048e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6715495069602 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50830172 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52042021 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131347446454272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80801148 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 trainer/global_step‚7649 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50830170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131347446454272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30124382 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6914290122752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131631320 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131631320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138571630233088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134799012364158 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6731972128256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52041770 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚131657499875328 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80791478 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131657474971648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131347429414912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138571789998080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _step‚610 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30124384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128083603097792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6731884691456 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131621648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50830170 _runtime‚3333.413486464 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80791478 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0â/E:2þ- _wandbruntime‚3333  _runtime‚3333kðîõÿï 2loss/SRqqt2/assignment_loss‚1.6432833756497573 loss/FBt2/assignment_loss‚0 /loss/FBt1/detection_loss‚1.4964659387866657 /loss/FBt2/detection_loss‚1.2106137262391192 _runtime‚3333.414256138 /loss/FRt2/detection_loss‚2.8674357995181015 1loss/SRqqt2/detection_loss‚1.9041442931407975 epoch‚0 _step‚611 0loss/FRt1/assignment_loss‚2.6343213625856348 /loss/FRt2/assignment_loss‚3.223201909535368 2loss/SRqqt1/assignment_loss‚1.1208272155706904 &loss/total_loss‚19.933128201402177 trainer/global_step‚7649 1loss/FBt1/assignment_loss‚0.09844784735427344 /loss/FRt1/detection_loss‚2.5994548332449554 1loss/SRqqt1/detection_loss‚1.1349318997768154 % _timestamp‚1.7858090551227586e+09ðÊÆøú€ô /loss/FBt2/detection_loss‚1.2106137262391192 _runtime‚3333.414256138 /loss/FRt2/detection_loss‚2.8674357995181015 1loss/SRqqt2/detection_loss‚1.9041442931407975 epoch‚0 _step‚611 0loss/FRt1/assignment_loss‚2.6343213625856348 /loss/FRt2/assignment_loss‚3.223201909535368 2loss/SRqqt1/assignment_loss‚1.1208272155706904 &loss/total_loss‚19.933128201402177 trainer/global_step‚7649 1loss/FBt1/assignment_loss‚0.09844784735427344 /loss/FRt1/detection_loss‚2.5994548332449554 1loss/SRqqt1/detection_loss‚1.1349318997768154 % _timestamp‚1.7858090551227586e+09 2loss/SRqqt2/assignment_loss‚1.6432833756497573 loss/FBt2/assignment_loss‚0 /loss/FBt1/detection_loss‚1.4964659387866657ã£GYœ‚(:‘ ª‘ÅÓ¸»æÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.96777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©d5£«‚‚(:   ª‘ÅÓœŽžÕ disk.dm-1.out‚ 162.4609375 disk.sda2.in‚0 disk.dm-0.in‚ 32.41796875 disk.sda3.in‚ 0.00390625 disk.rbd4.in‚0 disk.sda.in‚ 0.00390625 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.out‚ 1.619140625 disk.drbd1743.in‚0 cpu‚13.951519220226157% disk./.usageGB‚1242.6047630310059 disk.sda.out‚ 162.4609375" disk.dm-0.out‚12031.9384765625 disk.rbd3.in‚0 disk.sda1.in‚0$ memory_percent‚27.46031880378723 proc.cpu.threads‚29 disk.rbd2.in‚0 disk.sda3.out‚ 162.4609375 disk.rbd5.in‚0 disk.rbd2.out‚ 1.681640625! disk.nvme0c0n1.in‚ 32.41796875' disk.nvme0c0n1.out‚12031.9384765625 disk.drbd1743.out‚ 3.75390625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 32.41796875 disk.rbd0.in‚0# proc.memory.rssMB‚ 11267.6484375( disk./.usagePercent‚83.3922482618522 disk.rbd4.out‚ 1.533203125 disk.sda2.out‚0% disk.nvme0n1.out‚12031.9384765625 disk.sda1.out‚0 disk.rbd5.out‚ 1.533203125* proc.memory.availableMB‚47539.60546875) proc.memory.percent‚17.19306707382202 network.sent‚ 7.121896e+06 network.recv‚440998؇¹2ƒ- _wandbruntime‚3352  _runtime‚3352Ç…küÉ„Ã ' lr-AdamW/pg1‚0.00019842006375037167 ' lr-AdamW/pg2‚0.00019842006375037167 trainer/global_step‚7699 % _timestamp‚1.7858090742531602e+09 _runtime‚3352.544977728 _step‚612²ü¥šÎ…È _step‚612 ' lr-AdamW/pg1‚0.00019842006375037167 ' lr-AdamW/pg2‚0.00019842006375037167 trainer/global_step‚7699 % _timestamp‚1.7858090742531602e+09 _runtime‚3352.544977728äDÇ2†- _wandbruntime‚3352  _runtime‚3352Èoô&}‡ÌO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.smal¥ÇèU&l_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22058238 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135662291798124 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132474292 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6958571147264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128903888580832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138963602009600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51155679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30317295 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132500648598528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚132188600802304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139459062070784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚7699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚132188617841664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132500623694848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138963602009600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22057987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6775088636928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52375535 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6775088636928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6775001207296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚613 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51155677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3352.549928717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132464634 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30317297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6958438375936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚132188600802304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6775001207296 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52375282 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139459219745792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 % _timestamp‚1.7858090742543178e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138963706478592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51155677 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81318613 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6758506059776èâdï×'ˆÑO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6775088636928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52375535 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6775088636928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6775001207296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _step‚613 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51155677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3352.549928717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132464634 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30317297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6958438375936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚132188600802304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6775001207296 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52375282 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139459219745792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 % _timestamp‚1.7858090742543178e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138963706478592 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51155677 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81318613 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6758506059776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22058238 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚135662291798124 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132474292 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6958571147264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128903888580832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138963602009600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51155679 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30317295 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132500648598528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚132188600802304 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139459062070784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 trainer/global_step‚7699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚132188617841664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132500623694848 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138963602009600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22057987 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032塲Ÿ|2‰- _wandbruntime‚3352  _runtime‚3352-°gÜ&ŠÖM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81329150 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128920628753152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132481800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135679909443666 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30321234 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚7699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81319480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6759383533110 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132205767565312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51162320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6775968263680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138981752868352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52382124 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138981648392192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132481800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6959336393728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132205784604672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139477167205376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128920611713792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51162322 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858090745908022e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132491472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52382379 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6775968263680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22060892 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30321232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135680012286262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6775880826880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22061145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0B`Ê£ ‹ÛM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6775880826880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22061145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81329150 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128920628753152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132481800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135679909443666 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30321234 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚7699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81319480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6759383533110 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 SDDeviceStatsMonitor.on_train_batch_en¢ê´…>d/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132205767565312 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51162320 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6775968263680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138981752868352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52382124 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚614 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138981648392192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132481800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6959336393728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132205784604672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139477167205376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128920611713792 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51162322 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858090745908022e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132491472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52382379 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6775968263680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22060892 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30321232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135680012286262 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251æ=+4£2Œ- _wandbruntime‚3352  _runtime‚3352Óõ4 ƒ /loss/FRt1/assignment_loss‚2.163300949979473 3loss/FBt1/assignment_loss‚0.0003296872155260644 4loss/FBt2/assignment_loss‚4.6149351352450396e-06 1loss/SRqqt1/detection_loss‚0.9953008104115725 /loss/FBt1/detection_loss‚1.3428425807214808 %loss/total_loss‚16.52357591951455 _step‚615 2loss/SRqqt1/assignment_loss‚1.0459912158607039 2loss/SRqqt2/assignment_loss‚0.8662082409542619 /loss/FRt1/detection_loss‚2.3466422213493168 .loss/FRt2/detection_loss‚2.407937429469041 1loss/SRqqt2/detection_loss‚1.8406482152076382 _runtime‚ 3352.88353476 epoch‚0 trainer/global_step‚7699 % _timestamp‚1.7858090745920315e+09 0loss/FRt2/assignment_loss‚2.6119183891951434 /loss/FBt2/detection_loss‚0.9024515642152567÷Œ Žˆ /loss/FRt1/assignment_loss‚2.163300949979473 3loss/FBt1/assignment_loss‚0.0003296872155260644 4loss/FBt2/assignment_loss‚4.6149351352450396e-06 1loss/SRqqt1/detection_loss‚0.9953008104115725 /loss/FBt1/detection_loss‚1.3428425807214808 %loss/total_loss‚16.52357591951455 _step‚615 2loss/SRqqt1/assignment_loss‚1.0459912158607039 2loss/SRqqt2/assignment_loss‚0.8662082409542619 /loss/FRt1/detection_loss‚2.3466422213493168 .loss/FRt2/detection_loss‚2.407937429469041 1loss/SRqqt2/detection_loss‚1.8406482152076382 _runtime‚ 3352.88353476 epoch‚0 trainer/global_step‚7699 % _timestamp‚1.7858090745920315e+09 0loss/FRt2/assignment_loss‚2.6119183891951434 /loss/FBt2/detection_loss‚0.9024515642152567ç2v?½‘‚(:† ¹‘ÅÓüÅäÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚124.335' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚69.075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hc˜l­‚(:¢  ¹‘ÅÓäħÖ disk.rbd5.in‚0 disk.sda.out‚ 163.23046875% memory_percent‚27.459681034088135 disk.dm-0.in‚ 32.41796875 disk.rbd3.in‚0% disk.nvme0n1.out‚12032.1923828125 disk.rbd5.out‚ 1.533203125 cpu‚13.952371065717271 disk.drbd1743.in‚0 network.recv‚443442! disk.nvme0c0n1.in‚ 32.41796875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.681640625) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29% disk./.usageGB‚1242.6047630310059 disk.drbd1743.out‚ 3.7734375 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.rbd2.in‚0" disk.dm-0.out‚12032.1923828125 disk.rbd3.out‚0 disk.rbd4.out‚ 1.533203125 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0) proc.memory.availableMB‚ 47540.0234375# proc.memory.rssMB‚ 11267.6484375 disk.nvme0n1.in‚ 32.41796875 disk.sda1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚12032.1923828125 disk.dm-1.in‚ 0.00390625 disk.sda2.in‚0 network.sent‚ 7.166854e+06( disk./.usagePercent‚83.3922482618522 disk.sda.in‚ 0.00390625 disk.rbd2.out‚ 1.681640625 disk.sda3.out‚ 163.23046875 disk.dm-1.out‚ 163.23046875 disk.sda1.out‚0ÝXuþ2‘- _wandbruntime‚3374  _runtime‚3374®1¢É’à trainer/global_step‚7749 % _timestamp‚1.7858090958258224e+09 _runtime‚3374.117493916 _step‚616 ' lr-AdamW/pg1‚0.00019970867307463695 ' lr-AdamW/pg2‚0.00019970867307463695ß IÒÎ“È ' lr-AdamW/pg1‚0.00019970867307463695 ' lr-AdamW/pg2‚0.00019970867307463695 trainer/global_step‚7749 % _timestamp‚1.7858090958258224e+09 _runtime‚3374.117493916 _step‚616èAÁÞ!2”- _wandbruntime‚3374  _runtime‚3374KÒèoÒ'•ÌO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133046938952704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30514147 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819084772352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22201519 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51487827 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139866040764416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6818997342720 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81836959 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133334444 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51487829 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140364797866496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136543188877632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81846615 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81846615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140364631802880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _runtime‚3374.121978379 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6802308720160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133046938952704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133361004438528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133046955992064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133360979534848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139865936295424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129740897196832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133046955992064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7003793427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30514145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819084772352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52715409 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚7749 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818997342720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51487829 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129740880157472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133324786 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51487827 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139866040764416 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7003652268032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6802394523284 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139865936295424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚136543291720116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858090958268242e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887Þ”U×'–ÑO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51487827 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139866040764416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6818997342720 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81836959 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133334444 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51487829 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140364797866496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136543188877632 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81846615 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81846615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚140364631802880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _runtime‚3374.121978379 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6802308720160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133046938952704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133361004438528 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133046955992064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133360979534848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139865936295424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129740897196832 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133046955992064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7003793427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30514145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819084772352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52715409 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 trainer/global_step‚7749 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818997342720 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51487829 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129740880157472 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133324786 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51487827 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139866040764416 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7003652268032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6802394523284 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139865936295424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚136543291720116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858090958268242e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133046938952704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30514147 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819084772352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22201519 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121é5¶Gk2—- _wandbruntime‚3374  _runtime‚3374%Z˜ÖM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51494470 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22204295 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22204068 _step‚618 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129757620329792 KEDevicã&Ù!eStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139883982678016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52722379 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133378186651648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7004553882112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81847482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133064122755072 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133378211555328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136560909365770 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133341952 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81857152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819876962304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136560806523174 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139884087154176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140382896104448 _runtime‚3374.464598598 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133064105715712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51494472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30518084 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819876962304 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133351624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚7749 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133351624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52722150 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133064105715712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133341952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858090961725779e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30518082 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6803271996618 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448,h½á&™ÛM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139883982678016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52722379 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133378186651648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7004553882112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81847482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133064122755072 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133378211555328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136560909365770 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133341952 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81857152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819876962304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136560806523174 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139884087154176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140382896104448 _runtime‚3374.464598598 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚133064105715712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51494472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30518084 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819876962304 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133351624 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 trainer/global_step‚7749 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133351624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52722150 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133064105715712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133341952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 % _timestamp‚1.7858090961725779e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30518082 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6803271996618 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51494470 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22204295 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22204068 _step‚618 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129757620329792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ê·B 2š- _wandbruntime‚3374  _runtime‚3374!gé×› %loss/total_loss‚17.96368005025475 2loss/SRqqt1/assignment_loss‚1.0100954600603944 3loss/FBt2/assignment_loss‚5.534713665156232e-08 /loss/FRt2/detection_loss‚2.5394685104321897 1loss/SRqqt2/assignment_loss‚0.776526366138742 .loss/FRt1/detection_loss‚2.465802962096726 /loss/FBt1/detection_loss‚2.1562794859592733 epoch‚0 % _timestamp‚1.7858090961739047e+09 0loss/FRt1/assignment_loss‚2.2878863615233724 2loss/FBt1/assignment_loss‚0.012312389901915948 1loss/SRqqt1/detection_loss‚1.0070671608223567 1loss/SRqqt2/detection_loss‚1.8408185435192925 /loss/FBt2/detection_loss‚1.0182635284644286 trainer/global_step‚7749 _step‚619 /loss/FRt2/assignment_loss‚2.849159225988924 _runtime‚3374.465376037œóY œ† 1loss/SRqqt2/assignment_loss‚0.776526366138742 .loss/FRt1/detection_loss‚2.465802962096726 /loss/FBt1/detection_loss‚2.1562794859592733 epoch‚0 % _timestamp‚1.7858090961739047e+09 0loss/FRt1/assignment_loss‚2.2878863615233724 2loss/FBt1/assignment_loss‚0.012312389901915948 1loss/SRqqt1/detection_loss‚1.0070671608223567 1loss/SRqqt2/detection_loss‚1.8408185435192925 /loss/FBt2/detection_loss‚1.0182635284644286 trainer/global_step‚7749 _step‚619 /loss/FRt2/assignment_loss‚2.849159225988924 _runtime‚3374.465376037 %loss/total_loss‚17.96368005025475 2loss/SRqqt1/assignment_loss‚1.0100954600603944 3loss/FBt2/assignment_loss‚5.534713665156232e-08 /loss/FRt2/detection_loss‚2.5394685104321897ë»Èœ‚(:‘ È‘ÅÓÝ€ÚÒ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.474' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚60.81888888888889 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XèÁ'£ž‚(:˜  È‘ÅÓ°“”Ø disk.drbd1743.in‚0 disk.nvme0n1.in‚ 32.41796875% disk.nvme0n1.out‚12094.7685546875 disk.rbd5.in‚0% disk./.usageGB‚1242.6047630310059 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚0 disk.rbd3.in‚0 disk.rbd5.out‚ 1.619140625! disk.nvme0c0n1.in‚ 32.41796875 disk.sda.in‚ 0.00390625 network.sent‚ 7.211604e+06 disk.rbd2.out‚ 1.7138671875 disk.rbd3.out‚0 disk.sda.out‚ 163.78125 proc.cpu.threads‚29 disk.rbd2.in‚0 disk.sda2.in‚0" disk.dm-0.out‚12094.7685546875' disk.nvme0c0n1.out‚12094.7685546875 disk.sda3.in‚ 0.00390625 disk.dm-1.out‚ 163.78125 disk.rbd1.out‚ 1.681640625 disk.sda2.out‚0 disk.drbd1743.out‚3.78125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚445187 disk.rbd0.in‚0$ memory_percent‚27.46085524559021* proc.memory.availableMB‚47539.25390625 disk.sda3.out‚ 163.78125 disk.rbd4.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375 cpu‚13.952021683740542( disk./.usagePercent‚83.3922482618522 disk.dm-0.in‚ 32.41796875 disk.rbd4.in‚0) proc.memory.percent‚17.19306707382202b ÄjœŸ‚(:‘ בÅÓÜ£ðÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚155.758' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.53222222222223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ã]®° ‚(:¥  בÅÓìÞýÕ disk.sda1.out‚0 disk.rbd2.out‚ 1.7138671875 disk.dm-1.out‚ 164.70703125 disk.nvme0n1.in‚ 32.41796875 disk.sda1.in‚0 disk.drbd1743.out‚ 3.80078125 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd5.in‚0 disk.rbd4.out‚ 1.533203125# proc.memory.rssMB‚ 11267.6484375 cpu‚13.952424951699845$ disk./.usageGB‚1243.164680480957 disk.sda2.out‚0 disk.dm-0.in‚ 32.41796875 disk.rbd0.in‚0) proc.memory.availableMB‚ 47539.2265625) disk./.usagePercent‚83.42982479172046 disk.sda.in‚ 0.00390625 disk.sda.out‚ 164.70703125 disk.rbd2.in‚0 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 164.70703125 disk.drbd1743.in‚0" disk.dm-0.out‚12103.0380859375 network.sent‚ 7.256452e+06 disk.dm-1.in‚ 0.00390625 disk.rbd3.out‚0 disk.rbd5.out‚ 1.681640625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.7138671875% disk.nvme0n1.out‚12103.0380859375 disk.rbd0.out‚0) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29 network.recv‚447388! disk.nvme0c0n1.in‚ 32.41796875' disk.nvme0c0n1.out‚12103.0380859375 disk.rbd4.in‚0% memory_percent‚27.460896968841553%ˆÁ2¡- _wandbruntime‚3394  _runtime‚3394ÄH†âɢà % _timestamp‚1.7858091158377783e+09 _runtime‚3394.129502713 _step‚620 ' lr-AdamW/pg1‚0.00020099728239890224 ' lr-AdamW/pg2‚0.00020099728239890224 trainer/global_step‚7799œ¦úÎ£È ' lr-AdamW/pg2‚0.00020099728239890224 trainer/global_step‚7799 % _timestamp‚1.7858091158377783e+09 _runtime‚3394.129502713 _step‚620 ' lr-AdamW/pg1‚0.00020099728239890224칂#2¤- _wandbruntime‚3394  _runtime‚3394̲­3"¥ÌO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130577905812832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6846197183668 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚3394.133933265 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140768375050240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133905294142464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137424085957140 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133905277103104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53056387 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22345122 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7858091158388178e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140768270581248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6846282986792 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140768375050240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134194596 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30710997 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134221335374848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141270339774464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133905277103104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82374617 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6863080907776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6862993478144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6862993478144 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130577905812832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6846197183668 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _runtime‚3394.133933265 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140768375050240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚133905294142464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137424085957140 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133905277103104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53056387 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22345122 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7858091158388178e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140768270581248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6846282986792 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140768375050240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134194596 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30710997 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134221335374848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141270339774464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133905277103104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82374617 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6863080907776 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6862993478144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6862993478144 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6847160460126 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82375484 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51826622 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51826620 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6863960534528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140786421440000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134211776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140786421440000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6863960534528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82385154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141288446999040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858091161728504e+09 _step‚622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82385154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51826622 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 trainer/global_step‚7799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 _runtime‚3394.464860973 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140786316963840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53062940 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7049744742400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133922460905472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82375484 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53063199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141288287234048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130594645985152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚223480084wŽ‰á&©ÛM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82375484 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51826622 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51826620 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6863960534528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140786421440000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134211776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140786421440000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6863960534528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82385154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141288446999040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858091161728504e+09 _step‚622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82385154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51826622 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 trainer/global_step‚7799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 _runtime‚3394.464860973 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140786316963840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53062940 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7049744742400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133922460905472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82375484 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53063199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141288287234048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130594645985152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22348008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134202104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30714932 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6847074656890 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134202104 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134238567395328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6847160460126 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392îOö*2ª- _wandbruntime‚3394  _runtime‚3394ä2Y«‚ 2loss/SRqqt2/assignment_loss‚0.8278055540427114 2loss/FBt1/assignment_loss‚0.004392069662455829 .loss/FRt2/detection_loss‚2.637740780431104 0loss/SRqqt2/detection_loss‚2.368583237018549 /loss/FBt1/detection_loss‚1.9488797392696142 /loss/FBt2/detection_loss‚0.8808490311694813 _runtime‚3394.465884262 _step‚623 epoch‚0 0loss/FRt1/assignment_loss‚2.1018577575683595 /loss/FRt2/assignment_loss‚2.981010230190783 2loss/SRqqt1/assignment_loss‚0.9259139947343532 4loss/FBt2/assignment_loss‚2.1621421308674174e-06 1loss/SRqqt1/detection_loss‚1.0077029464974705 trainer/global_step‚7799 $ _timestamp‚1.785809116174399e+09 /loss/FRt1/detection_loss‚2.4017213276587426 &loss/total_loss‚18.086458830385755°ãí¬‡ _step‚623 epoch‚0 0loss/FRt1/assignment_loss‚2.1018577575683595 /loss/FRt2/assignment_loss‚2.981010230190783 2loss/SRqqt1/assignment_loss‚0.9259139947343532 4loss/FBt2/assignment_loss‚2.1621421308674174e-06 1loss/SRqqt1/detection_loss‚Õ7 .þ1.0077029464974705 trainer/global_step‚7799 $ _timestamp‚1.785809116174399e+09 /loss/FRt1/detection_loss‚2.4017213276587426 &loss/total_loss‚18.086458830385755 2loss/SRqqt2/assignment_loss‚0.8278055540427114 2loss/FBt1/assignment_loss‚0.004392069662455829 .loss/FRt2/detection_loss‚2.637740780431104 0loss/SRqqt2/detection_loss‚2.368583237018549 /loss/FBt1/detection_loss‚1.9488797392696142 /loss/FBt2/detection_loss‚0.8808490311694813 _runtime‚3394.465884262ï³vgÜœ­‚(:‘ æ‘ÅÓòÃôÒ gpu.0.gpu‚17.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.96777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g¸Ðr«®‚(:   æ‘Åӻ͔Ö! disk.nvme0c0n1.in‚ 32.41796875 disk.rbd2.out‚ 1.77685546875 disk.sda3.in‚ 0.00390625 disk.dm-0.in‚ 32.41796875 disk.sda1.out‚0$ memory_percent‚27.46061682701111) disk./.usagePercent‚83.44670362446045 disk.sda.in‚ 0.00390625 disk.rbd1.out‚ 1.7138671875 disk.dm-1.out‚ 165.46875 disk.sda2.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚12668.65771484375 disk.sda3.out‚ 165.46875 disk.nvme0n1.in‚ 32.41796875 disk.sda1.in‚0 disk.rbd4.out‚ 1.619140625 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.drbd1743.out‚ 3.80859375 disk.rbd5.in‚0 cpu‚13.952770005429569 disk.rbd2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚12668.65771484375 disk.rbd5.out‚ 1.681640625 disk.rbd0.out‚0) proc.memory.percent‚17.19306707382202 network.recv‚448377 disk.sda.out‚ 165.46875 disk.rbd3.in‚0 disk.rbd0.in‚0* proc.memory.availableMB‚47539.41015625 network.sent‚ 7.258969e+06$ disk./.usageGB‚1243.416187286377# disk.dm-0.out‚12668.65771484375 disk.rbd4.in‚0# proc.memory.rssMB‚ 11267.6484375h3àx2¯- _wandbruntime‚3414  _runtime‚3414Ë»»åÉ°Ã _step‚624 ' lr-AdamW/pg1‚0.00020228589172316757 ' lr-AdamW/pg2‚0.00020228589172316757 trainer/global_step‚7849 % _timestamp‚1.7858091359156625e+09 _runtime‚3414.207323419æiÎ±È ' lr-AdamW/pg1‚0.00020228589172316757 ' lr-AdamW/pg2‚0.00020228589172316757 trainer/global_step‚7849 % _timestamp‚1.7858091359156625e+09 _runtime‚3414.207323419 _step‚624ð—ž2²- _wandbruntime‚3414  _runtime‚3414O1ªÒ'³ÌO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141670604867072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6907077043200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134763632292864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6906989613568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82902619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _runtime‚3414.211912709 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52152129 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141670709336064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135081691214848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135054748 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6907077043200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53397382 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135081716118528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30907845 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131414914428832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131414897389472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22489275 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138305085879132 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135045090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141670709336064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52152127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22489535 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141670604867072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82892963 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82892963 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142175705211392 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138304983036648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6906989613568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30907847 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7094148865024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6890085647176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134763615253504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30907847 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7094148865024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6890085647176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134763615253504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141670604867072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6907077043200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134763632292864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6906989613568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82902619 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 _runtime‚3414.211912709 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52152129 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141670709336064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135081691214848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135054748 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6907077043200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53397382 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135081716118528 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30907845 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131414914428832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131414897389472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22489275 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138305085879132 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135045090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141670709336064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52152127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22489535 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141670604867072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640ñlÚy„2µ- _wandbruntime‚3414  _runtime‚3414‹ŠúΘ%¶ÔM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135098923235328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 3414.5604734 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135098898331648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135071928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚138322600682190 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131431654601152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82903486 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30911782 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135062256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6907956669952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134780799055872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142193810722816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6890963120398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6907869233152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7095051446784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7858091362684126e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6907956669952 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _step‚626 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52158770 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135071928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141688755725824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134780799055872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53403975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52158772 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135062256 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6907956669952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134780799055872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142193810722816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6890963120398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6907869233152 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7095051446784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7858091362684126e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6907956669952 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _step‚626 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52158770 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135071928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚141688755725824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134780799055872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53403975 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52158772 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 3414.5604734 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135098898331648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135071928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚138322600682190 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131431654601152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82903486 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670òî.5ú2¸- _wandbruntime‚3414  _runtime‚3414Â~í6¹ÿ 0loss/FRt1/assignment_loss‚2.4601794556639662 $ _timestamp‚1.785809136269829e+09 2loss/SRqqt1/assignment_loss‚0.8042636229174918 1loss/FBt1/assignment_loss‚0.03624050648942604 2loss/FBt2/assignment_loss‚0.004902525331232989 .loss/FRt1/detection_loss‚2.283137695200142 1loss/SRqqt1/detection_loss‚1.1007794322462046 /loss/FBt1/detection_loss‚1.4526884204741302 /loss/FBt2/detection_loss‚0.7777649950846259 trainer/global_step‚7849 /loss/FRt2/assignment_loss‚2.774727769436375 2loss/SRqqt2/assignment_loss‚1.0225336134687744 _step‚627 /loss/FRt2/detection_loss‚2.5143506791322463 1loss/SRqqt2/detection_loss‚1.9291867129928009 %loss/total_loss‚17.16075542843742 epoch‚0 _runtime‚3414.561646902#-*ò º„ /loss/FRt2/detection_loss‚2.5143506791322463 1loss/SRqqt2/detection_loss‚1.9291867129928009 %loss/total_loss‚17.16075542843742 epoch‚0 _runtime‚3414.561646902 0loss/FRt1/assignment_loss‚2.4601794556639662 $ _timestamp‚1.785809136269829e+09 2loss/SRqqt1/assignment_loss‚0.8042636229174918 1loss/FBt1/assignment_loss‚0.03624050648942604 2loss/FBt2/assignment_loss‚0.004902525331232989 .loss/FRt1/detection_loss‚2.283137695200142 1loss/SRqqt1/detection_loss‚1.1007794322462046 /loss/FBt1/detection_loss‚1.4526884204741302 /loss/FBt2/detection_loss‚0.7777649950846259 trainer/global_step‚7849 /loss/FRt2/assignment_loss‚2.774727769436375 2loss/SRqqt2/assignment_loss‚1.0225336134687744 _step‚627ó‰ï œ»‚(:‘ õ‘ÅÓ®œâÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚148.485' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.49166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ<Ï«¼‚(:   õ‘ÅÓᄪ×( disk.nvme0c0n1.out‚12732.96728515625 disk.rbd2.out‚ 1.77685546875 disk.dm-1.out‚ 167.7421875 disk.rbd1.in‚0 disk.sda.out‚ 167.7421875 disk.drbd1743.in‚0 disk.drbd1743.out‚3.8125 disk.dm-0.in‚ 37.5390625& disk.nvme0n1.out‚12732.96728515625 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1243.4152374267578 disk.rbd5.out‚ 1.7138671875 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 167.7421875 disk.rbd0.out‚0 disk.dm-1.in‚ 0.00390625% memory_percent‚27.460622787475586# proc.memory.rssMB‚ 11267.6484375 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 37.5390625 cpu‚13.952997220092499 proc.cpu.threads‚29 disk.sda3.in‚ 0.00390625) disk./.usagePercent‚83.44663987858442 disk.nvme0n1.in‚ 37.5390625' proc.memory.availableMB‚ 47539.40625) proc.memory.percent‚17.19306707382202 disk.rbd4.out‚ 1.619140625 disk.rbd2.in‚0 disk.rbd1.out‚ 1.77685546875 disk.sda2.in‚0 network.sent‚ 7.303857e+06 disk.rbd4.in‚0 disk.sda2.out‚0 network.recv‚450304# disk.dm-0.out‚12732.96728515625 disk.rbd3.out‚0 disk.sda1.out‚0yÿE^œ½‚(:‘ „’ÅÓðâìÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚122.395' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.99722222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÁy0¡¾‚(:–  „’ÅÓâ¾ÄÕ disk.sda2.out‚0 disk.rbd5.in‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 168.39453125 disk.rbd3.in‚0! disk.nvme0n1.out‚ 12734.203125 disk.sda.in‚ 0.00390625 disk.rbd1.in‚0$ memory_percent‚27.46036648750305$ disk./.usageGB‚1243.415241241455# disk.nvme0c0n1.out‚ 12734.203125 disk.rbd2.in‚0 disk.sda2.in‚0 cpu‚13.953565162334202 proc.cpu.threads‚29 disk.nvme0c0n1.in‚ 37.5390625 disk.sda3.out‚ 168.39453125 disk.dm-0.in‚ 37.5390625 disk.nvme0n1.in‚ 37.5390625 disk.rbd5.out‚ 1.7138671875 disk.rbd0.out‚0 disk.sda.out‚ 168.39453125 disk.sda3.in‚ 0.00390625 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚ 1.83984375 disk.drbd1743.in‚0 disk.drbd1743.out‚ 3.83984375 disk.dm-0.out‚ 12734.203125# proc.memory.rssMB‚ 11267.6484375 disk.rbd4.in‚0 disk.rbd4.out‚ 1.681640625 network.recv‚452186) disk./.usagePercent‚83.44664013459196 disk.sda1.out‚0) proc.memory.percent‚17.19306707382202 network.sent‚ 7.348713e+06* proc.memory.availableMB‚47539.57421875 disk.rbd1.out‚ 1.77685546875:‡Í$2¿- _wandbruntime‚3435  _runtime‚3435Üz7êÉÀà ' lr-AdamW/pg1‚0.00020357450104743286 ' lr-AdamW/pg2‚0.00020357450104743286 trainer/global_step‚7899 % _timestamp‚1.7858091576774192e+09 _runtime‚3435.969136441 _step‚628KàHƒÎÁÈ % _timestamp‚1.7858091576774192e+09 _runtime‚3435.969136441 _step‚628 ' lr-AdamW/pg1‚0.00020357450104743286 ' lr-AdamW/pg2‚0.00020357450104743286 trainer/global_step‚7899ôIj\'2Â- _wandbruntime‚3435  _runtime‚3435«Tö\Ñ'ÃËO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6950985748992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135621953403904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31104697 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135621970443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83420965 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6933974110684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _runtime‚ 3435.97394388 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142573043621888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚7899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135942047054848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7858091576784616e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83420965 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139185880116156 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83430621 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83430621 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52484279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132251906005472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52484277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135905242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135914900 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142573043621888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6951073178624 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7139276137472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142572939152896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31104695 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6950985748992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135942071958528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142572939152896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132251923044832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52484279 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52484277 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135621970443264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22632278 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143081348096000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139185982958640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130XÌêâåÄÐO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31104697 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚135621970443264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83420965 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6933974110684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 _runtime‚ 3435.97394388 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142573043621888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 trainer/global_step‚7899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135942047054848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 % _timestamp‚1.7858091576784616e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83420965 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139185880116156 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83430621 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83430621 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52484279 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132251906005472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52484277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135905242 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135914900 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142573043621888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6951073178624 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7139276137472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142572939152896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31104695 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6950985748992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135942071958528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142572939152896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132251923044832 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_poøV§Hñ ol.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52484279 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52484277 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚135621970443264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22632278 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143081348096000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139185982958640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6950985748992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚135621953403904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0õ·– 2Å- _wandbruntime‚3436  _runtime‚3436AÁYÜ&ÆÖM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135922408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53743360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132268646177792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52490920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142590985535488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6951865368576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83441158 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143099303673856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135639120166912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135959254171648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚7899 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142591090011648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7140049502208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142591090011648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135932080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22634985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6951952805376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22634728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858091580163171e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52490922 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135959279075328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31108634 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135922408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52490920 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132268663217152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31108632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83431488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135639120166912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6951952805376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6934937387142 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132268663217152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31108632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83431488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚135639120166912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6951952805376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6934937387142 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53743360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132268646177792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52490920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142590985535488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6951865368576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83441158 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143099303673856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135639120166912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135959254171648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 trainer/global_step‚7899 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚142591090011648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7140049502208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142591090011648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135932080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22634985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6951952805376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22634728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858091580163171e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52490922 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135959279075328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31108634 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135922408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52490920 EVµÎÏÈ trainer/global_step‚7949 % _timestamp‚1.7858091775117512e+09 _runtime‚3455.803404678 _step‚632 ' lr-AdamW/pg1‚0.00020486311037169817 ' lr-AdamW/pg2‚0.00020486311037169817øLEÚ2Ð- _wandbruntime‚3455  _runtime‚3455§ž1IÑÌO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136765394 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143475273438720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54076125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31301547 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143986728436224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83958623 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6995069314048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52816427 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83948967 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6977862574192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6995069314048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6994981884416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143986881916928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143475273438720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22774330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143475377907712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52816429 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 _runtime‚3455.805070764 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52816427 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136480291554304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7184325541376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚633 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83948967 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7858091775127816e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133088914621472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136480308593664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 287Pÿ6Ɖ1874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7949 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140066777195664 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22774578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136480291554304 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6977948377316 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136480308593664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143475377907712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7184454118400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83958623 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54075875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31301545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143475273438720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22774330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143475377907712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52816429 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 _runtime‚3455.805070764 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52816427 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136480291554304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7184325541376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚633 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83948967 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7858091775127816e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133088914621472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136480308593664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7949 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚140066777195664 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22774578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136480291554304 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6977948377316 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136480308593664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143475377907712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7184454118400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83958623 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54075875 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31301545 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6995069314048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52816427ù'|¦2Ó- _wandbruntime‚3456  _runtime‚3456»• ‡Ü&ÔÖM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136792232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143493319821312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136497458317312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144004995662336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83969160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136497475356672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54082689 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31305484 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136782560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83969160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31305482 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136782560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52823070 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133105671833152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54082952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚634 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83959490 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83959490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6995948940800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6995861504000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144004831703040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136819634915328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143493424297472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143493424297472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52823072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858091778697677e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133105654793792 _runtime‚3456.161906014 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6978740047414 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136792232 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52823072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7858091778697677e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133105654793792 _runtime‚3456.161906014 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6978740047414 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136792232 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136497458317312 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144004995662336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83969160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136497475356672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54082689 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31305484 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136782560 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83969160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31305482 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136782560 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52823070 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133105671833152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54082¯X]r952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚634 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83959490 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83959490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6995948940800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6995861504000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144004831703040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136819634915328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143493424297472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640ú‹¾×y2Ö- _wandbruntime‚3456  _runtime‚3456ücµT×þ .loss/FRt1/detection_loss‚2.602897253192839 .loss/FRt2/detection_loss‚2.468498449731659 .loss/FBt1/detection_loss‚1.064966843886809 epoch‚0 trainer/global_step‚7949 % _timestamp‚1.7858091778712118e+09 _step‚635 0loss/FRt2/assignment_loss‚2.9392942760016894 2loss/SRqqt2/assignment_loss‚1.0712521129686918 0loss/FRt1/assignment_loss‚2.4161528133955157 1loss/SRqqt1/detection_loss‚0.9584486025449788 .loss/FBt2/detection_loss‚1.081426909722557 _runtime‚3456.162691828 2loss/SRqqt1/assignment_loss‚0.9512946099685183 1loss/FBt1/assignment_loss‚0.03316917219275178 2loss/FBt2/assignment_loss‚0.022643119965102078 0loss/SRqqt2/detection_loss‚1.958228186890483 &loss/total_loss‚17.568272350461594]ĬD ؃ _step‚635 0loss/FRt2/assignment_loss‚2.9392942760016894 2loss/SRqqt2/assignment_loss‚1.0712521129686918 0loss/FRt1/assignment_loss‚2.4161528133955157 1loss/SRqqt1/detection_loss‚0.9584486025449788 .loss/FBt2/detection_loss‚1.081426909722557 _runtime‚3456.162691828 2loss/SRqqt1/assignment_loss‚0.9512946099685183 1loss/FBt1/assignment_loss‚0.03316917219275178 2loss/FBt2/assignment_loss‚0.022643119965102078 0loss/SRqqt2/detection_loss‚1.958228186890483 &loss/total_loss‚17.568272350461594 .loss/FRt1/detection_loss‚2.602897253192839 .loss/FRt2/detection_loss‚2.468498449731659 .loss/FBt1/detection_loss‚1.064966843886809 epoch‚0 trainer/global_step‚7949 % _timestamp‚1.7858091778712118e+09ûou't›Ù‚(: ¢’ÅÓ¦½ØÒ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚137.96' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚76.64444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u7.£­Ú‚(:¢  ¢’ÅÓ˜Å’Ö network.sent‚ 7.396149e+06 disk.drbd1743.in‚0# disk.dm-0.out‚12799.70556640625& disk.nvme0n1.out‚12799.70556640625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 37.5390625 disk.sda.out‚ 171.41015625 disk.rbd2.in‚0 disk.dm-1.out‚ 171.41015625 disk.rbd1.out‚ 1.83984375 disk.rbd5.in‚0( disk.nvme0c0n1.out‚12799.70556640625 disk.rbd1.in‚0 disk.rbd5.out‚ 1.77685546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 171.41015625$ disk./.usageGB‚1243.415370941162 disk.dm-0.in‚ 37.5390625 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 11267.6484375 disk.drbd1743.out‚ 3.8671875 disk.nvme0n1.in‚ 37.5390625 disk.rbd4.out‚ 1.7138671875 disk.dm-1.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚27.4608314037323* proc.memory.availableMB‚47539.26953125 network.recv‚455294) disk./.usagePercent‚83.44664883884813 disk.sda.in‚ 0.00390625) proc.memory.percent‚17.19306707382202 cpu‚13.953740598105766 disk.rbd3.in‚0 proc.cpu.threads‚29ö)iÑ2Û- _wandbruntime‚3476  _runtime‚3476A ËóÉÜà ' lr-AdamW/pg1‚0.00020615171969596348 ' lr-AdamW/pg2‚0.00020615171969596348 trainer/global_step‚7999 % _timestamp‚1.7858091984953136e+09 _runtime‚3476.788986263 _step‚636(æݧÎÝÈ % _timestamp‚1.7858091984953136e+09 _runtime‚3476.788986263 _step‚636 ' lr-AdamW/pg1‚0.00020615171969596348 ' lr-AdamW/pg2‚0.00020615171969596348 trainer/global_step‚7999üj°Â2Þ- _wandbruntime‚3476  _runtime‚3476¾²‹ìÒ'ßÌO trainer/global_step‚7999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84486625 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137338646744064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84476969 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144377607724544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7229443681792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144377712193536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7038978019840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53148579 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53148577 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31498397 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137662758734848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3476.789065852 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31498395 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133925940276832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137338629704704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137338646744064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54416979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7021751037700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858091984960005e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84476969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22918324 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7021836840824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚140947777117656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7038978019840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7039065449472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137338629704704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144377607724544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7039065449472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144892202416640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137635204 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22918582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1307ƒ§×'àÑO YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚133925923237472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144377712193536 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84486625 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53148579 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7021836840824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚140947777117656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7038978019840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7039065449472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137338629704704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144377607724544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7039065449472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144892202416640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137635204 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22918582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚7999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84486625 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137338646744064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84476969 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144377607724544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7229443681792 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144377712193536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7038978019840 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53148579 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53148577 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31498397 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137662758734848 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3476.789065852 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31498395 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133925940276832 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137338629704704 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137338646744064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54416979 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7021751037700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858091984960005e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84476969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22918324 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54423503 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84497162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858091988269722e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7999 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53155222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31502332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137652384 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54423754 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 _runtime‚3477.119023269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53155222 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7022628510922 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7039945076224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144395758583296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144395654107136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84497162 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84487492 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137679990755328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137642712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144910471051776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22921171 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144910317578240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133942680449152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_po²äÕÖ ol.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22921420 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137355813507072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140965291920714 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133942663409792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚638 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53155220 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137355796467712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137652384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53155220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137679965851648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144910317578240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚133942680449152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22921420 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137355813507072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚140965291920714 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133942663409792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _step‚638 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53155220 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137355796467712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137652384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53155220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137679965851648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54423503 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84497162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858091988269722e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7999 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53155222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31502332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137652384 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54423754 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 _runtime‚3477.119023269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53155222 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7022628510922 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7039945076224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144395758583296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144395654107136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84497162 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84487492 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137679990755328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137642712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144910471051776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22921171 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0þì5 2ä- _wandbruntime‚3477  _runtime‚3477{­ã÷ åƒ epoch‚0 0loss/FRt1/assignment_loss‚2.1341996029728936 1loss/FBt1/assignment_loss‚0.22102873423076996 .loss/FRt1/detection_loss‚2.307329005960907 /loss/FRt2/detection_loss‚2.8272415738399714 trainer/global_step‚7999 % _timestamp‚1.7858091988282027e+09 2loss/SRqqt2/assignment_loss‚1.1555645901900835 1loss/SRqqt1/detection_loss‚0.9453214527901124 1loss/SRqqt2/detection_loss‚1.6984582507075408 /loss/FBt1/detection_loss‚2.1696620262227952 _runtime‚3477.119697394 /loss/FRt2/assignment_loss‚3.054833627726934 2loss/SRqqt1/assignment_loss‚0.9427224868908525 4loss/FBt2/assignment_loss‚0.00023582268651194165 /loss/FBt2/detection_loss‚0.9464636675788668 _step‚639 &loss/total_loss‚18.403060841798236áöÜÝæˆ _step‚639 &loss/total_loss‚18.403060841798236 epoch‚0 0loss/FRt1/assignment_loss‚2.1341996029728936 1loss/FBt1/assignment_loss‚0.22102873423076996 .loss/FRt1/detection_loss‚2.307329005960907 /loss/FRt2/detection_loss‚2.8272415738399714 trainer/global_step‚7999 % _timestamp‚1.7858091988282027e+09 2loss/SRqqt2/assignment_loss‚1.1555645901900835 1loss/SRqqt1/detection_loss‚0.9453214527901124 1loss/SRqqt2/detection_loss‚1.6984582507075408 /loss/FBt1/detection_loss‚2.1696620262227952 _runtime‚3477.119697394 /loss/FRt2/assignment_loss‚3.054833627726934 2loss/SRqqt1/assignment_loss‚0.9427224868908525 4loss/FBt2/assignment_loss‚0.00023582268651194165 /loss/FBt2/detection_loss‚0.9464636675788668ÿZCpœç‚(:‘ ±’ÅÓേÓ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚156.048' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.69333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è‚(:ž  ±’ÅÓÞî€Ö disk.rbd2.out‚ 1.83984375 disk.rbd1.in‚0 disk.drbd1743.out‚3.875 disk.sda1.in‚0 proc.cpu.threads‚29) disk./.usagePercent‚83.44665139892348 disk.dm-0.in‚ 37.5390625 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚13.954215086460282% disk./.usageGB‚1243.4154090881348( disk.nvme0c0n1.out‚12799.91650390625 disk.rbd3.in‚0 disk.rbd5.out‚ 1.83984375 disk.rbd4.out‚ 1.7138671875) proc.memory.percent‚17.19306707382202 disk.nvme0c0n1.in‚ 37.5390625 disk.sda3.in‚ 0.00390625 disk.nvme0n1.in‚ 37.5390625& disk.nvme0n1.out‚12799.91650390625 network.sent‚ 7.441162e+06 disk.rbd2.in‚0 disk.dm-1.in‚ 0.00390625 disk.rbd4.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.sda.out‚ 172.22265625 disk.sda3.out‚ 172.22265625 disk.rbd1.out‚ 1.83984375# disk.dm-0.out‚12799.91650390625 disk.rbd0.out‚0% memory_percent‚27.459675073623657* proc.memory.availableMB‚47540.02734375 disk.sda.in‚ 0.00390625 disk.dm-1.out‚ 172.22265625 disk.sda2.out‚0 disk.drbd1743.in‚0 network.recv‚457624ˆo_’œé‚(:‘ À’ÅÓØ°ÖÒ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.96777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³3;©ê‚(:ž  À’ÅÓÇÄ´Õ network.sent‚ 7.485958e+06) disk./.usagePercent‚83.44665139892348( disk.nvme0c0n1.out‚12862.56396484375 disk.rbd2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚12862.56396484375# proc.memory.rssMB‚ 11267.6484375 disk.rbd4.in‚0 disk.dm-1.in‚ 0.00390625* proc.memory.availableMB‚47540.02734375) proc.memory.percent‚17.19306707382202 cpu‚13.954584991156256% disk./.usageGB‚1243.4154090881348 disk.rbd0.out‚0 disk.rbd4.out‚ 1.77685546875 disk.rbd3.in‚0 network.recv‚459584 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.00390625 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 37.5390625 proc.cpu.threads‚29 disk.dm-1.out‚ 173.015625 disk.rbd1.out‚ 1.83984375 disk.dm-0.in‚ 37.5390625 disk.nvme0c0n1.in‚ 37.5390625 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0# disk.dm-0.out‚12862.56396484375% memory_percent‚27.459675073623657 disk.rbd5.out‚ 1.83984375 disk.sda.out‚ 173.015625 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 173.015625 disk.drbd1743.out‚ 3.90234375j”î2ë- _wandbruntime‚3496  _runtime‚3496f¦¢“Éìà ' lr-AdamW/pg1‚0.00020744032902022877 ' lr-AdamW/pg2‚0.00020744032902022877 trainer/global_step‚8049 % _timestamp‚1.7858092183875802e+09 _runtime‚3496.679248095 _step‚640ÓÄÏ|ÎíÈ ' lr-AdamW/pg1‚0.00020744032902022877 ' lr-AdamW/pg2‚0.00020744032902022877 trainer/global_step‚8049 % _timestamp‚1.7858092183875802e+09 _runtime‚3496.679248095 _step‚640€’ó?12î- _wandbruntime‚3496  _runtime‚3496‡X¤QÒ'ïÌO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7082974155264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54756371 _runtime‚3496.683943313 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53480729 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7065725304332 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138196967855104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23061386 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53480727 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7065639501208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138196984894464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145280046479360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141828674197164 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7274619890688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145797898432000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53480727 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134762931853472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858092183885856e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145279942010368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7274758953472 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85014627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7083061584896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23061126 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138485698 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚145280046479360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7082974155264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85004971 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138495356 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31695245 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7083061584896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚8049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31695247 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85014627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138523139478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚641 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145279942010368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138196967855104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138523114574848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54756633 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134762948892832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138196984894464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134762931853472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 % _timestamp‚1.7858092183885856e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145279942010368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7274758953472 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85014627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7083061584896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23061126 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138485698 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚145280046479360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7082974155264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85004971 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138495356 YKDeviceStatsMonitor.TÅÞþÎon_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31695245 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7083061584896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚8049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31695247 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85014627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138523139478528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚641 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145279942010368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138196967855104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138523114574848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54756633 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134762948892832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138196984894464 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145816002084864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 _step‚642 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54763200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31699184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53487370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134779672025792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23064274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54763458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141846291842818 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141846189000222 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138512536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138512536 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7083853774848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138540321691648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7066602777666 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7083853774848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138540321691648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7066602777666 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145816002084864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 _step‚642 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54763200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31699184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53487370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134779672025792 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23064274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54763458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141846291842818 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141846189000222 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138512536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138512536 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7127057720320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135599957508832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7126970290688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85542629 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146182276296192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85532973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31892095 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139345850 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55096380 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3517.883937523 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85532973 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139055323044864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139055306005504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146182380765184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23204283 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85542629 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139383470414848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚645 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146182380765184ŒfŒÌ×'þÑO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53812877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53812879 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7319894239744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7858092395890388e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7127057720320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135599957508832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7126970290688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85542629 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146182276296192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85532973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31892095 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139345850 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55096380 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3517.883937523 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85532973 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139055323044864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139055306005504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146182380765184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23204283 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85542629 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139383470414848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚645 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146182380765184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135599940469472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146182276296192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146703225591808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142709571276672 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146703389558272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7126970290688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53812877 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139372688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7320794642432 _step‚646 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55102915 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146200322678784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31896034 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31896032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7127849910272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146721331021312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139072472768512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7127849910272 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135616680641792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53819520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139372688 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142727086079730 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚8099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7127937347072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146721497077760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85553166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85543496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7110491241174 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚3518.216844862 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53819522 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55103165 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚3518.216844862 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53819522 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55103165 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139372688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7320794642432 _step‚646 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55102915 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146200322678784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31896034 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31896032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7127849910272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146721331021312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139072472768512 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7127849910272 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135616680641792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53819520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139372688 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142727086079730 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚8099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7127937347072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146721497077760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85553166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85543496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7110491241174 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392†ºêy2‚- _wandbruntime‚3518  _runtime‚3518¿õlP ƒ„ 1loss/SRqqt2/assignment_loss‚1.076040924258996 .loss/FRt1/detection_loss‚2.534630850752195 &loss/total_loss‚18.105572515862306 trainer/global_step‚8099 _step‚647 2loss/SRqqt1/assignment_loss‚1.0804014891844744 3loss/FBt2/assignment_loss‚4.705628036934908e-08 1loss/SRqqt2/detection_loss‚1.6896210723556577 epoch‚0 0loss/FRt1/assignment_loss‚2.7623024940490724 /loss/FRt2/assignment_loss‚2.711978138747968 4loss/FBt1/assignment_loss‚0.00016315892016141982 % _timestamp‚1.7858092399263248e+09 _runtime‚3518.217822887 /loss/FRt2/detection_loss‚2.3337531212129092 1loss/SRqqt1/detection_loss‚1.0780291432938747 /loss/FBt1/detection_loss‚2.2104508553498556 /loss/FBt2/detection_loss‚0.6282012206808615Uʺ„‰ 1loss/SRqqt2/assignment_loss‚1.076040924258996 .loss/FRt1/detection_loss‚2.534630850752195 &loss/total_loss‚18.105572515862306 trainer/global_step‚8099 _step‚647 2loss/SRqqt1/assignment_loss‚1.0804014891844744 3loss/FBt2/assignment_loss‚4.705628036934908e-08 1loss/SRqqt2/detection_loss‚1.6896210723556577 epoch‚0 0loss/FRt1/assignment_loss‚2.7623024940490724 /loss/FRt2/assignment_loss‚2.711978138747968 4loss/FBt1/assignment_loss‚0.00016315892016141982 % _timestamp‚1.7858092399263248e+09 _runtime‚3518.217822887 /loss/FRt2/detection_loss‚2.3337531212129092 1loss/SRqqt1/detection_loss‚1.0780291432938747 /loss/FBt1/detection_loss‚2.2104508553498556 /loss/FBt2/detection_loss‚0.6282012206808615‡jó_œ…‚(:‘ Þ’ÅÓÞÞÉÒ gpu.0.gpu‚14.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚121.229' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.34944444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦?¹*˜†‚(:  Þ’ÅÓŒæ‘Õ network.recv‚462524 disk.drbd1743.out‚ 3.9296875 disk.rbd1.out‚ 1.83984375 disk.sda2.in‚0( disk./.usagePercent‚83.4467061845358 disk.dm-0.in‚ 37.5390625 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 37.5390625) proc.memory.percent‚17.19306707382202 disk.drbd1743.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚12863.52294921875 disk.rbd4.out‚ 1.83984375 disk.sda3.in‚0.4375 network.sent‚ 7.533342e+06 disk.rbd5.out‚ 1.83984375( disk.nvme0c0n1.out‚12863.52294921875 disk.sda.out‚ 174.859375 disk.rbd0.in‚0 disk.rbd1.in‚0% memory_percent‚27.461153268814087* proc.memory.availableMB‚47539.05859375 proc.cpu.threads‚29 disk.sda3.out‚ 174.859375# disk.dm-0.out‚12863.52294921875 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0.4375 disk.dm-1.out‚ 174.859375 disk.nvme0n1.in‚ 37.5390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0.4375 disk.rbd2.out‚ 1.83984375 disk.sda2.out‚0% disk./.usageGB‚1243.4162254333496 disk.rbd0.out‚0# proc.memory.rssMB‚ 11267.6484375 cpu‚13.954687164221088i}ø2‡- _wandbruntime‚3537  _runtime‚3537Î3WƒÇˆÁ trainer/global_step‚8149 % _timestamp‚1.7858092596030967e+09 _runtime‚3537.894845676 _step‚648 & lr-AdamW/pg1‚0.0002100175476687594 & lr-AdamW/pg2‚0.0002100175476687594Ô gTÌ‰Æ & lr-AdamW/pg2‚0.0002100175476687594 trainer/global_step‚8149 % _timestamp‚1.7858092596030967e+09 _runtime‚3537.894845676 _step‚648 & lr-AdamW/pg1‚0.0002100175476687594ˆë‰L†2Š- _wandbruntime‚3537  _runtime‚3537ˆ •Ò'‹ÌO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23345937 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86060975 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140243826254848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139913661195264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55434884 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55434627 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _runtime‚3537.899998373 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147084610582016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139913644155904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54145029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚140243851158528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140206002 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147084715051008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7153502231348 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32088945 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147084715051008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7170966426112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7170966426112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143590365513696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8149 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147084610582016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54145029 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54145027 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7171053855744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858092596042013e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54145027 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86060975 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7365032249344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7171053855744 _step‚649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86070631 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140215660 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23345682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139913644155904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86070631 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55434884 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55434627 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 _runtime‚3537.899998373 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147084610582016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139913644155904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54145029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚140243851158528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140206002 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147084715051008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7153502231348 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32088945 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147084715051008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7170966426112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7170966426112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143590365513696 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8149 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚147084610582016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54145029 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54145027 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7171053855744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858092596042013e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54145027 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86060975 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7365032249344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7171053855744 _step‚649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86070631 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140215660 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23345682 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139913644155904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86070631 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147102761440768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140232840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139930810918912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7365935562240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7154293901446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140223168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86081168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚147102656964608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54151670 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚147626993837568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55441411 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7365800700928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7154379704682 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143608086001834 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23348529 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136453706297152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147626834072576MÎpÒà&ÚM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23348529 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136453706297152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147626834072576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54151672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86071498 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147102761440768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140232840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚139930810918912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7365935562240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7154293901446 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140223168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86081168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚147102656964608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54151670 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚147626993837568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55441411 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7365800700928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7154379704682 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143608086001834Š_ õ–2- _wandbruntime‚3538  _runtime‚3538ãÂ/•‘‚ 1loss/SRqqt2/assignment_loss‚0.943315372412855 3loss/FBt2/assignment_loss‚8.307859134977965e-06 %loss/total_loss‚18.44135685916214 epoch‚0 % _timestamp‚1.7858092599535077e+09 0loss/FRt1/assignment_loss‚2.4148730757908945 0loss/FRt2/assignment_loss‚3.2479373400027933 /loss/FRt1/detection_loss‚2.3406084770193467 1loss/SRqqt1/detection_loss‚0.9237340978450246 _runtime‚3538.244975474 2loss/SRqqt1/assignment_loss‚1.0481524551287293 /loss/FRt2/detection_loss‚2.7396341682626653 0loss/SRqqt2/detection_loss‚1.616560326110233 .loss/FBt1/detection_loss‚2.104824895770954 /loss/FBt2/detection_loss‚1.0616257178364321 _step‚651 3loss/FBt1/assignment_loss‚8.262512307867707e-05 trainer/global_step‚8149Yßõ— ’‡ 0loss/FRt1/assignment_loss‚2.4148730757908945 0loss/FRt2/assignment_loss‚3.2479373400027933 /loss/FRt1/detection_loss‚2.3406084770193467 1loss/SRqqt1/detection_loss‚0.9237340978450246 _runtime‚3538.244975474 2loss/SRqqt1/assignment_loss‚1.0481524551287293 /loss/FRt2/detection_loss‚2.7396341682626653 0loss/SRqqt2/detection_loss‚1.616560326110233 .loss/FBt1/detection_loss‚2.104824895770954 /loss/FBt2/detection_loss‚1.0616257178364321 _step‚651 3loss/FBt1/assignment_loss‚8.262512307867707e-05 trainer/global_step‚8149 1loss/SRqqt2/assignment_loss‚0.943315372412855 3loss/FBt2/assignment_loss‚8.307859134977965e-06 %loss/total_loss‚18.44135685916214 epoch‚0 % _timestamp‚1.7858092599535077e+09‹B<"ÿœ“‚(:‘ í’ÅÓØ’þÒ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚157.115' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚87.28611111111113 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L5ùž”‚(:“  í’ÅÓçÕÔÕ) proc.memory.percent‚17.19306707382202 network.recv‚464617) disk./.usagePercent‚83.44670106438512 disk.rbd5.in‚0 disk.rbd2.in‚0 disk.rbd5.out‚ 1.83984375 disk.rbd0.in‚0 disk.sda.in‚0.4375 disk.sda3.in‚0.4375 disk.sda3.out‚ 175.48046875 disk.rbd3.in‚0$ memory_percent‚27.46073007583618 disk.rbd4.out‚ 1.83984375 disk.nvme0c0n1.in‚ 37.5390625 disk.dm-1.out‚ 175.48046875% disk./.usageGB‚1243.4161491394043 disk.nvme0n1.in‚ 37.5390625& disk.nvme0n1.out‚12925.99169921875 disk.rbd4.in‚0 disk.rbd2.out‚ 1.83984375 disk.sda2.out‚0# disk.dm-0.out‚12925.99169921875# proc.memory.rssMB‚ 11267.6484375 disk.dm-1.in‚0.4375 disk.rbd1.out‚ 1.83984375 disk.dm-0.in‚ 37.5390625) proc.memory.availableMB‚ 47539.3359375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 175.48046875 proc.cpu.threads‚29 network.sent‚ 7.578083e+06( disk.nvme0c0n1.out‚12925.99169921875 disk.drbd1743.in‚0 disk.drbd1743.out‚ 3.95703125 disk.rbd3.out‚0 cpu‚13.955178959276298 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0ÈïÜ$•‚(:… ü’ÅÓ—‘€Ó gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚85.48 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XþeÇš–‚(:  ü’ÅÓ‹¤¸Ö# memory_percent‚27.4605393409729 network.sent‚ 7.622863e+06) proc.memory.availableMB‚ 47539.4609375 proc.cpu.threads‚29% disk./.usageGB‚1243.4162063598633& disk.nvme0n1.out‚12926.29443359375 disk.sda1.in‚0 disk.rbd5.out‚ 1.83984375 disk.sda.out‚ 176.3203125 disk.dm-1.out‚ 176.3203125 disk.rbd1.in‚0 disk.sda3.in‚0.4375 disk.sda3.out‚ 176.3203125) disk./.usagePercent‚83.44670490449812( disk.nvme0c0n1.out‚12926.29443359375 disk.rbd2.out‚ 1.83984375 disk.rbd4.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 37.5390625 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd4.out‚ 1.83984375 disk.dm-1.in‚0.4375 disk.sda2.in‚0) proc.memory.percent‚17.19306707382202 disk.drbd1743.in‚0 disk.drbd1743.out‚ 3.96484375# disk.dm-0.out‚12926.29443359375 disk.rbd2.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0 disk.nvme0n1.in‚ 37.5390625 disk.rbd5.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 11267.6484375 cpu‚13.955527799107571 network.recv‚466524 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 37.5390625 disk.sda.in‚0.4375TØÀ2—- _wandbruntime‚3557  _runtime‚3557zlR¶Ç˜Á & lr-AdamW/pg1‚0.0002113061569930247 & lr-AdamW/pg2‚0.0002113061569930247 trainer/global_step‚8199 % _timestamp‚1.7858092793027685e+09 _runtime‚3557.594433169 _step‚652i›×šÌ™Æ & lr-AdamW/pg1‚0.0002113061569930247 & lr-AdamW/pg2‚0.0002113061569930247 trainer/global_step‚8199 % _timestamp‚1.7858092793027685e+09 _runtime‚3557.594433169 _step‚652ŒÖ÷”¾2š- _wandbruntime‚3557  _runtime‚3557žŸâ Ò'›ÌO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86588977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55773936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140771982306304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54477179 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137273957701472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54477177 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144471365435688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7197390694856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32285797 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55774193 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚8199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147986944867840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7214962561536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23488141 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148514266719232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3557.598938521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86588977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141104182094848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54477179 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144471262593204 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147987049336832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7214962561536 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7410084624384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140771982306304 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 71973048917324oª`œÑO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144471262593204 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147987049336832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7214962561536 \ODeviceStatø%ÏTw$sMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7410084624384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140771982306304 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7197304891732 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86588977 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55773936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140771982306304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54477179 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137273957701472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54477177 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚144471365435688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7197390694856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32285797 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55774193 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 trainer/global_step‚8199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147986944867840 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7214962561536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23488141 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148514266719232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3557.598938521 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86588977 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141104182094848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54477179-6þ¤2- _wandbruntime‚3557  _runtime‚3557-ê;hÜ&žÖM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86609170 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86609170 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3557.926044177 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7215929617920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7410987205120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7215842181120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141083320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54483820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140789166108672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7411115774976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140789149069312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚140789149069312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141092992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144488880238746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55780983 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148532376416768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55780737 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54483822 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54483820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚654 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148005095726592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148004991250432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141121414115328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137290697873792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7215929617920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7198182364954 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141083320 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7410987205120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7215842181120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141083320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54483820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140789166108672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7411115774976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚140789149069312 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚140789149069312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141092992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144488880238746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55780983 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148532376416768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55780737 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54483822 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54483820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚654 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148005095726592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148004991250432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141121414115328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137290697873792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7215929617920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7198182364954 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141083320 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148889279153664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7259046126592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141630320456704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141935964 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7241279158364 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54809327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145352159672712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7241193355240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.785809299014482e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87116979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87126635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148889383622656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚657 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141964562838528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3577.307045632 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32482645 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23630910 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141630320456704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23631155 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141964537934848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32482645 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23630910 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141630320456704 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23631155 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141964537934848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148889279153664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7259046126592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141630320456704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141935964 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7241279158364 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54809327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145352159672712‘üÙ•2«- _wandbruntime‚3577  _runtime‚3577ò Ü&¬ÖM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54815970 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145369880160850 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56120290 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141647487219712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87127502 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7259838316544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23633960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141981745051648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56120544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32486582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87137172 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149438049980416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87137172 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138127723529152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚658 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54815972 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7242156631698 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7242070828462 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148907325536256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141943472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7259925753344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7259925753344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3577.635094849 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141981769955328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32486584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148907430012416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148907430012416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149437896506880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7456280025088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138127706489792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141953144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54815972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23633708 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141647504259072 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858092993430994e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141953144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54815970 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141647487219712 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87127502 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141943472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7456151455232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145369777318254 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121ÿFÛ˜ó­ÛM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7259925753344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3577.635094849 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 cODeviceStatsMonitÜÜ´Kîor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141981769955328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32486584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148907430012416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148907430012416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149437896506880 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7456280025088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138127706489792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141953144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54815972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23633708 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141647504259072 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 % _timestamp‚1.7858092993430994e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141953144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54815970 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141647487219712 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87127502 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141943472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7456151455232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145369777318254 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54815970 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145369880160850 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56120290 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141647487219712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87127502 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7259838316544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23633960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141981745051648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56120544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32486582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87137172 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149438049980416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87137172 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138127723529152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _step‚658 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54815972 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7242156631698 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7242070828462 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148907325536256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141943472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7259925753344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024’`@$J2®- _wandbruntime‚3577  _runtime‚3577øÖ_n ¯ƒ /loss/FBt2/detection_loss‚1.2789218773175361 3loss/FBt2/assignment_loss‚5.029734656468756e-06 /loss/FRt2/detection_loss‚2.4977347050086562 -loss/FBt1/detection_loss‚3.08873135011111 &loss/total_loss‚19.020609681368125 epoch‚0 trainer/global_step‚8249 3loss/FBt1/assignment_loss‚6.691292267646531e-05 .loss/FRt1/detection_loss‚2.402453561398116 _step‚659 0loss/FRt2/assignment_loss‚2.7105647088533424 1loss/SRqqt1/detection_loss‚0.9790850069139387 % _timestamp‚1.7858092993449247e+09 _runtime‚3577.636368348 0loss/FRt1/assignment_loss‚2.4840278097174386 2loss/SRqqt1/assignment_loss‚0.9292804149266046 2loss/SRqqt2/assignment_loss‚0.7679480450031789 1loss/SRqqt2/detection_loss‚1.8817902594608695# ¡°ˆ 0loss/FRt1/assignment_loss‚2.4840278097174386 2loss/SRqqt1/assignment_loss‚0.9292804149266046 2loss/SRqqt2/assignment_loss‚0.7679480450031789 1loss/SRqqt2/detection_loss‚1.8817902594608695 /loss/FBt2/detection_loss‚1.2789218773175361 3loss/FBt2/assignment_loss‚5.029734656468756e-06 /loss/FRt2/detection_loss‚2.4977347050086562 -loss/FBt1/detection_loss‚3.08873135011111 &loss/total_loss‚19.020609681368125 epoch‚0 trainer/global_step‚8249 3loss/FBt1/assignment_loss‚6.691292267646531e-05 .loss/FRt1/detection_loss‚2.402453561398116 _step‚659 0loss/FRt2/assignment_loss‚2.7105647088533424 1loss/SRqqt1/detection_loss‚0.9790850069139387 % _timestamp‚1.7858092993449247e+09 _runtime‚3577.636368348“‡‚-œ±‚(:‘ š“ÅÓ¥¡ýÒ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚139.847' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚77.69277777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ÚÀþ˜²‚(:  š“ÅÓ­ýäÖ disk.sda2.in‚0" disk.dm-0.out‚12989.2685546875 disk.sda1.in‚0# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202 disk.sda3.out‚ 177.94921875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 37.5390625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚0.4375 disk.rbd2.in‚0( proc.memory.availableMB‚ 47538.921875 proc.cpu.threads‚29 network.sent‚ 7.670362e+06 network.recv‚469614% disk./.usageGB‚1243.4162826538086 disk.sda2.out‚0% disk.nvme0n1.out‚12989.2685546875 disk.rbd4.in‚0 disk.sda.out‚ 177.94921875 disk.dm-1.in‚0.4375 disk.rbd3.out‚0# memory_percent‚27.4613618850708 cpu‚13.956396402333738 disk.drbd1743.in‚0 disk.rbd5.in‚0 disk.rbd4.out‚ 1.83984375 disk.rbd2.out‚ 1.83984375) disk./.usagePercent‚83.44671002464882 disk.drbd1743.out‚ 3.9921875 disk.nvme0n1.in‚ 37.5390625 disk.sda1.out‚0 disk.dm-1.out‚ 177.94921875 disk.dm-0.in‚ 37.5390625 disk.rbd3.in‚0 disk.rbd5.out‚ 1.83984375' disk.nvme0c0n1.out‚12989.2685546875 disk.sda3.in‚0.4375qBûó2³- _wandbruntime‚3596  _runtime‚3596`„/ÃÉ´Ã _step‚660 ' lr-AdamW/pg1‚0.00021388337564155532 ' lr-AdamW/pg2‚0.00021388337564155532 trainer/global_step‚8299 % _timestamp‚1.7858093186373932e+09 _runtime‚3596.929045063DÎµÈ ' lr-AdamW/pg2‚0.00021388337564155532 trainer/global_step‚8299 % _timestamp‚1.7858093186373932e+09 _runtime‚3596.929045063 _step‚660 ' lr-AdamW/pg1‚0.00021388337564155532”íÛP,2¶- _wandbruntime‚3596  _runtime‚3596 ™–Ð'·ÊO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7303042262016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55141477 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56453508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23773767 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142786458 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142488658607104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _runtime‚3596.933929797 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55141479 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142488658607104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142824893774848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚149791717908480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87654637 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142796116 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146233056752220 _step‚661 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142824918678528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55141477 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146233159594704 trainer/global_step‚8299 # _timestamp‚1.78580931863836e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149791613439488 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32679495 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150325436761088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150325281183232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149791613439488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55141479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56453262$ÎvÕ'¸ÏO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149791613439488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55141479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56453262 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7303042262016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55141477 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56453508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23773767 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142786458 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142488658607104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 _runtime‚3596.933929797 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55141479 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142488658607104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142824893774848 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚149791717908480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87654637 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142796116 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146233056752220 _step‚661 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142824918678528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55141477 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146233159594704 trainer/global_step‚8299 # _timestamp‚1.78580931863836e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149791613439488 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32679495 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150325436761088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚150325281183232•Ø|½…2¹- _wandbruntime‚3597  _runtime‚3597õ—aºÕM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 $ _timestamp‚1.785809318976121e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 27682406hä5«z$40 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142505842409472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23776892 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149809764298240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138964732145152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7303921888768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55148120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23776641 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142803624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7501290291200 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142813296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142505825370112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149809764298240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146250674397762 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142803624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56460073 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142813296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55148122 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55148120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7303921888768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149809659822080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142505842409472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7501425152512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32683432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87655504™"¾à&»ÚM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142813296 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142505825370112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149809764298240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146250674397762 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142803624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56460073 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142813296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55148122 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55148120 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7303921888768 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149809659822080 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142505842409472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7501425152512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32683432 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87655504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 $ _timestamp‚1.785809318976121e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142505842409472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23776892 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149809764298240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138964732145152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7303921888768 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55148120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23776641 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142803624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7501290291200 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0–DåZ2¼- _wandbruntime‚3597  _runtime‚3597õÔeû½ /loss/FRt1/assignment_loss‚2.649674372828525 2loss/SRqqt1/assignment_loss‚1.0060159711690704 1loss/SRqqt2/assignment_loss‚1.224653024681259 .loss/FRt2/detection_loss‚2.430212348994642 1loss/SRqqt1/detection_loss‚1.0299747814089546 /loss/FBt2/detection_loss‚0.6557389076064842 epoch‚0 trainer/global_step‚8299 .loss/FRt1/detection_loss‚2.299932375712239 % _timestamp‚1.7858093189773319e+09 0loss/FRt2/assignment_loss‚3.0003966494361953 4loss/FBt2/assignment_loss‚1.7318214911636704e-05 _runtime‚3597.268861901 _step‚663 2loss/FBt1/assignment_loss‚0.014866238803733722 1loss/SRqqt2/detection_loss‚1.7637131392609242 /loss/FBt1/detection_loss‚3.8119567036628723 %loss/total_loss‚19.88715183177981¾—…ì ¾† 2loss/FBt1/assignment_loss‚0.014866238803733722 1loss/SRqqt2/detection_loss‚1.7637131392609242 /loss/FBt1/detection_loss‚3.8119567036628723 %loss/total_loss‚19.88715183177981 /loss/FRt1/assignment_loss‚2.649674372828525 2loss/SRqqt1/assignment_loss‚1.0060159711690704 1loss/SRqqt2/assignment_loss‚1.224653024681259 .loss/FRt2/detection_loss‚2.430212348994642 1loss/SRqqt1/detection_loss‚1.0299747814089546 /loss/FBt2/detection_loss‚0.6557389076064842 epoch‚0 trainer/global_step‚8299 .loss/FRt1/detection_loss‚2.299932375712239 % _timestamp‚1.7858093189773319e+09 0loss/FRt2/assignment_loss‚3.0003966494361953 4loss/FBt2/assignment_loss‚1.7318214911636704e-05 _runtime‚3597.268861901 _step‚663—²¡ ñœ¿‚(:‘ ©“ÅÓ“äËÒ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚129.479' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.93277777777779 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„õØ™À‚(:Ž  ©“ÅÓµõ…Õ disk.drbd1743.in‚0 proc.cpu.threads‚29 network.recv‚471755' disk.nvme0c0n1.out‚12989.5458984375 disk.dm-1.in‚0.4375 disk.sda2.in‚0 disk.rbd0.out‚0 cpu‚13.956755827465074 disk.sda.in‚0.4375 disk.sda.out‚ 178.55078125 disk.rbd2.in‚0 disk.sda3.in‚0.4375 disk.rbd5.out‚ 1.83984375# proc.memory.rssMB‚ 11267.6484375 disk.rbd2.out‚ 1.83984375) proc.memory.percent‚17.19306707382202% disk./.usageGB‚1243.4036598205566 disk.rbd1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚12989.5458984375 disk.sda1.out‚0 disk.rbd4.in‚0 disk.rbd4.out‚ 1.83984375' proc.memory.availableMB‚ 47538.90625) disk./.usagePercent‚83.44586289571801 disk.sda2.out‚0 disk.dm-0.in‚ 37.5390625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 37.5390625 disk.rbd5.in‚0 disk.rbd0.in‚0$ memory_percent‚27.46138572692871 network.sent‚ 7.715101e+06 disk.sda3.out‚ 178.55078125 disk.dm-1.out‚ 178.55078125 disk.rbd1.out‚ 1.83984375 disk.drbd1743.out‚ 3.99609375" disk.dm-0.out‚12989.5458984375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 37.5390625ØùËœÁ‚(:‘ ¸“ÅÓ³ø×Ò gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚42.0 gpu.0.powerWatts‚156.048' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.69333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]®¡’‚(:‡  ¸“ÅÓö¹²Õ network.recv‚473504 disk.sda.out‚ 179.28125 disk.sda2.in‚0 disk.dm-0.in‚ 37.5390625 disk.rbd0.in‚0$ memory_percent‚27.46140956878662( proc.memory.availableMB‚ 47538.890625% disk./.usageGB‚1243.4036674499512 disk.rbd4.in‚0 disk.drbd1743.in‚0 cpu‚13.95817321585113( disk.nvme0c0n1.out‚12990.70654296875 disk.sda.in‚0.4375 disk.sda1.in‚0# proc.memory.rssMB‚ 11267.6484375 network.sent‚ 7.759876e+06 disk.rbd4.out‚ 1.83984375 disk.rbd2.out‚ 1.83984375 disk.dm-1.in‚0.4375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 1.83984375 disk.sda3.out‚ 179.28125# disk.dm-0.out‚12990.70654296875 disk.rbd3.out‚0) disk./.usagePercent‚83.44586340773307 disk.nvme0c0n1.in‚ 37.5390625 disk.rbd2.in‚0 disk.drbd1743.out‚ 4.0234375 disk.rbd3.in‚0 proc.cpu.threads‚29 disk.sda3.in‚0.4375 disk.dm-1.out‚ 179.28125 disk.rbd1.out‚ 1.83984375 disk.sda2.out‚0 disk.nvme0n1.in‚ 37.5390625& disk.nvme0n1.out‚12990.70654296875 disk.rbd5.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.19306707382202çË3X2Ã- _wandbruntime‚3616  _runtime‚3616‹•å¸ÇÄÁ & lr-AdamW/pg1‚0.0002151719849658206 & lr-AdamW/pg2‚0.0002151719849658206 trainer/global_step‚8349 % _timestamp‚1.7858093384114144e+09 _runtime‚3616.703093299 _step‚664ç©ÑdÌÅÆ & lr-AdamW/pg1‚0.0002151719849658206 & lr-AdamW/pg2‚0.0002151719849658206 trainer/global_step‚8349 % _timestamp‚1.7858093384114144e+09 _runtime‚3616.703093299 _step‚664˜{R˜‡2Æ- _wandbruntime‚3616  _runtime‚3616‡þc)¥ÇÌO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56793621 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23917276 _step‚665 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150694052194304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55473629 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚3616.707935003 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143646610 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7346950967808 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55473627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143685274518528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143685249614848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88172983 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23917547 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150694052194304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147113953831728 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150693947725312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88182639 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150693947725312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143346996757504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7346950967808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7329056085380 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143347013796864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56793894 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7328970282256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚8349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55473627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143685274518528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143685249614848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88172983 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23917547 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150694052194304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147113953831728 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150693947725312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88182639 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150693947725312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143346996757504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7346950967808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7329056085380 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143347013796864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56793894 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7328970282256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 trainer/global_step‚8349 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150694052194304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55473629 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 _runtime‚3616.707935003 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143646610 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7346950967808 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143663776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88193176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150712098584064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55480270 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151249116160512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7347918024192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143663776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143673448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143364163520512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32880282 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150712098584064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7347830587392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23920374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7329933558714 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150711994107904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7546510149632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139801723721792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56800422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139801740761152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858093387433403e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88193176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55480270 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143702456731648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7347830587392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143673448¨Ã*gá&ËÛM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143663776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143673448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143364163520512 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32880282 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150712098584064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7347830587392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23920374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 149279232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7329933558714 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150711994107904 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7546510149632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139801723721792 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56800422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139801740761152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858093387433403e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88193176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55480270 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143702456731648 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7347830587392 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143673448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55480272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56800658 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 _runtime‚3617.035333302 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚147131571477270 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8349 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143663776 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88193176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150712098584064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55480270 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151249116160512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7347918024192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0šÒlÞñ2Ì- _wandbruntime‚3617  _runtime‚3617Ã1IªÍ /loss/FRt1/assignment_loss‚2.581178241155364 3loss/FBt2/assignment_loss‚7.909739602456827e-06 0loss/SRqqt1/detection_loss‚0.977038145959624 1loss/SRqqt2/detection_loss‚1.6822129355780049 _runtime‚3617.036119046 3loss/FBt1/assignment_loss‚3.999214391114947e-05 /loss/FRt2/detection_loss‚2.6732865982241445 .loss/FBt2/detection_loss‚0.496769857280823 0loss/FRt2/assignment_loss‚2.6875~åW]998679693645 1loss/SRqqt2/assignment_loss‚1.268234007078789 .loss/FRt1/detection_loss‚2.404722108251669 /loss/FBt1/detection_loss‚2.7517566166818144 _step‚667 2loss/SRqqt1/assignment_loss‚1.1832348833119632 &loss/total_loss‚18.706081163375075 epoch‚0 trainer/global_step‚8349 % _timestamp‚1.7858093387446127e+09× Óç Ά /loss/FRt1/assignment_loss‚2.581178241155364 3loss/FBt2/assignment_loss‚7.909739602456827e-06 0loss/SRqqt1/detection_loss‚0.977038145959624 1loss/SRqqt2/detection_loss‚1.6822129355780049 _runtime‚3617.036119046 3loss/FBt1/assignment_loss‚3.999214391114947e-05 /loss/FRt2/detection_loss‚2.6732865982241445 .loss/FBt2/detection_loss‚0.496769857280823 0loss/FRt2/assignment_loss‚2.6875998679693645 1loss/SRqqt2/assignment_loss‚1.268234007078789 .loss/FRt1/detection_loss‚2.404722108251669 /loss/FBt1/detection_loss‚2.7517566166818144 _step‚667 2loss/SRqqt1/assignment_loss‚1.1832348833119632 &loss/total_loss‚18.706081163375075 epoch‚0 trainer/global_step‚8349 % _timestamp‚1.7858093387446127e+09›â]„É‘Ï‚(:† Ç“ÅÓ¹ùÁÒ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚145.233' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚80.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚtÔ[šÐ‚(:  Ç“ÅÓ¸€–Ö disk.dm-0.in‚ 37.5390625# disk.dm-0.out‚13053.26904296875 disk.rbd1.out‚ 1.83984375* proc.memory.availableMB‚47538.87109375 disk.drbd1743.out‚ 4.04296875 disk.rbd3.in‚0 disk.sda1.out‚0 disk.rbd5.out‚ 1.83984375 disk.rbd1.in‚0# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29 disk.rbd4.out‚ 1.83984375 disk.nvme0c0n1.in‚ 37.5390625 disk.sda.out‚ 180.0703125& disk.nvme0n1.out‚13053.26904296875 network.recv‚475155) disk./.usagePercent‚83.44586033564268 disk.rbd0.in‚0 disk.dm-1.out‚ 180.0703125( disk.nvme0c0n1.out‚13053.26904296875 disk.sda2.out‚0$ memory_percent‚27.46143937110901) proc.memory.percent‚17.19306707382202 disk.rbd3.out‚0 disk.rbd5.in‚0 disk.rbd4.in‚0 disk.sda3.in‚0.4375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0.4375 disk.rbd2.in‚0 disk.sda1.in‚0 network.sent‚ 7.76265e+06$ disk./.usageGB‚1243.403621673584 disk.drbd1743.in‚0 cpu‚13.958570245203752 disk.sda.in‚0.4375 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 180.0703125 disk.nvme0n1.in‚ 37.5390625,¿Ó2Ñ- _wandbruntime‚3636  _runtime‚3636„ÁHìÇÒÁ & lr-AdamW/pg1‚0.0002164605942900859 & lr-AdamW/pg2‚0.0002164605942900859 trainer/global_step‚8399 % _timestamp‚1.7858093579727442e+09 _runtime‚3636.264412481 _step‚668P°2 ÌÓÆ trainer/global_step‚8399 % _timestamp‚1.7858093579727442e+09 _runtime‚3636.264412481 _step‚668 & lr-AdamW/pg1‚0.0002164605942900859 & lr-AdamW/pg2‚0.0002164605942900859œžµ 2Ô- _wandbruntime‚3636  _runtime‚3636sÂÔÑ'ÕËO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚151596386480128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144545630358528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7390947103232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57134512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24061315 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55805779 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152136445480960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33073197 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151596386480128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144205334907904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140621992165472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151596282011136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144205334907904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7391034532864 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88710641 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55805777 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152136601058816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55805777 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 $ _timestamp‚1.785809357973721e+09 _runtime‚3636.265619255 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7391034532864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7372944548888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚669 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33073195 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7590840026112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88700985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144545605454848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7372858745764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88700985 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144205351947264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144506762 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151596282011136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚140622009204832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚8399 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7390947103232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88710641 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55805779 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144205351947264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24061070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147994850911236 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144516420 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55805779 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152136445480960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33073197 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚151596386480128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144205334907904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140621992165472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151596282011136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚144205334907904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7391034532864 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88710641 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55805777 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152136601058816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55805777 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 $ _timestamp‚1.785809357973721e+09 _runtime‚3636.265619255 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7391034532864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 155577856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7372944548888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 _step‚669 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33073195 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7590840026112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88700985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144545605454848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 130674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7372858745764 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88700985 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚144205351947264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144506762 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151596282011136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚140622009204832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 trainer/global_step‚8399 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7390947103232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88710641 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55805779 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136êÂF2×- _wandbruntime‚3636  _runtime‚3636›¾žH#ØÕM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57141114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7391826722816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144533600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57141368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148012571399374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7591741591040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144523928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144562812571648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144523928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144222518710272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140638732337792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88711508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚3636.606658903 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55812422 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151614432869888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24064234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33077134 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚140638749377152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7373822022222 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144562837475328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144222501670912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7591876452352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88721178 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚8399 $ _timestamp‚1.785809358314706e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55812422 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚WÅô“24063982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚151614432869888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152154554162688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚151614328393728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88711508 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7391826722816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55812420³`+àà&ÙÚM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144533600 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7373736218986 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33077132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144222518710272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140638732337792 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88711508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚3636.606658903 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55812422 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151614432869888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24064234 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33077134 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚140638749377152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7373822022222 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144562837475328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144222501670912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7591876452352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88721178 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚8399 $ _timestamp‚1.785809358314706e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55812422 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24063982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚151614432869888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152154554162688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚151614328393728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88711508 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7391826722816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55812420 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57141114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7391826722816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 _step‚670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144533600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57141368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148012571399374 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7591741591040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144523928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144562812571648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144523928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7416747209272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145063690097664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24204754 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7636026887168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89238643 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24204496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148875850833228 _runtime‚3657.288925299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145366914 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152498616296960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57474801 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33270045 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152498720765952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7434943238656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56137929 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145376572 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56137927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89238643 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56137929 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7435030668288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7416833012396 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7435030668288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145405961294848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7858093789936457e+09 _step‚673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7416747209272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145063690097664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24204754 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7636026887168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89238643 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24204496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148875850833228 _runtime‚3657.288925299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145366914 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152498616296960 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57474801 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33270045 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152498720765952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7434943238656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56137929 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N@DeviceStatsMonitor.on_tÅß^—Ýrain_batch_start/allocation.all.allocated‚ 145376572 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56137927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89238643 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56137929 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7435030668288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7416833012396 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7435030668288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145405961294848¡ÛQúC2å- _wandbruntime‚3657  _runtime‚3657„ÖPÚ&æÔM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7435910295040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7435822858240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24207305 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153060103343616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7417624682494 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145384080 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145393752 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145393752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56144572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7435910295040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153060252622848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33273984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145423193315328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145423168411648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152516767155712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56144572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145080856860672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145080839821312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141475740953792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145384080 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145080856860672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57481516 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 3657.62088453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56144570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148893365636286 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89239510 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145080839821312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148893468478882 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7417710485730 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141475757993152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152516662679552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57481287 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89249180 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670jmû¾ß&çÙM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56144572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7435910295040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153060252622848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33273984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚145423193315328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145423168411648 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152516767155712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56144572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145080856860672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145080839821312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141475740953792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145384080 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145080856860672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57481516 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 3657.62088453 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56144570 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148893365636286 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89239510 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145080839821312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148893468478882 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7417710485730 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141475757993152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152516662679552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57481287 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89249180 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7435910295040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7435822858240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24207305 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153060103343616 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7417624682494 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 124375552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145384080 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145393752 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145393752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121¢Y¥¶=2è- _wandbruntime‚3657  _runtime‚3657ãåc¥ñéë /loss/FRt2/assignment_loss‚2.944171335962084 1loss/SRqqt1/detection_loss‚1.1354102573480787 $loss/total_loss‚18.8028049419745 trainer/global_step‚8449 /loss/FRt2/detection_loss‚2.6134648027105465 .loss/FBt1/detection_loss‚1.812481570057571 .loss/FBt2/detection_loss‚1.538436574359926 _runtime‚3657.621983001 0loss/FRt1/assignment_loss‚2.3899958208203316 2loss/SRqqt1/assignment_loss‚1.0661630573329817 3loss/FBt1/assignment_loss‚6.467042985569549e-06 loss/FBt2/assignment_loss‚0 -loss/FRt1/detection_loss‚2.44397876765579 1loss/SRqqt2/detection_loss‚1.7048771189791816 epoch‚0 _step‚675 2loss/SRqqt2/assignment_loss‚1.1538191697050242 % _timestamp‚1.7858093793301182e+09$å¨öêð .loss/FBt1/detection_loss‚1.812481570057571 .loss/FBt2/detection_loss‚1.538436574359926 _runtime‚3657.621983001 0loss/FRt1/assignment_loss‚2.3899958208203316 2loss/SRqqt1/assignment_loss‚1.0661630573329817 3loss/FBt1/assignment_loss‚6.467042985569549e-06 loss/FBt2/assignment_loss‚0 -loss/FRt1/detection_loss‚2.44397876765579 1loss/SRqqt2/detection_loss‚1.7048771189791816 epoch‚0 _step‚675 2loss/SRqqt2/assignment_loss‚1.1538191697050242 % _timestamp‚1.7858093793301182e+09 /loss/FRt2/assignment_loss‚2.944171335962084 1loss/SRqqt1/detection_loss‚1.1354102573480787 $loss/total_loss‚18.8028049419745 trainer/global_step‚8449 /loss/FRt2/detection_loss‚2.6134648027105465£y@¾»œë‚(:‘ å“ÅÓ˜ÄñÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚158.183' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚87.87944444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·} ì‚(:•  å“ÅÓ៥Õ disk.rbd3.in‚0 disk.nvme0n1.in‚ 37.5390625 disk.rbd5.out‚ 1.83984375 disk.sda.out‚ 181.48828125 disk.rbd2.out‚ 1.83984375% memory_percent‚27.461272478103638 disk.rbd4.out‚ 1.83984375 disk.nvme0c0n1.in‚ 37.5390625) disk./.usagePercent‚83.44586161568034% disk./.usageGB‚1243.4036407470703 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0.4375) proc.memory.percent‚17.19306707382202 disk.rbd1.out‚ 1.83984375 disk.rbd5.in‚0 proc.cpu.threads‚29 network.recv‚485511# disk.dm-0.out‚13053.78662109375 network.sent‚ 7.855734e+06 disk.dm-0.in‚ 37.5390625 disk.rbd4.in‚0 disk.sda3.out‚ 181.48828125 cpu‚13.959373176548041 disk.sda.in‚0.4375 disk.drbd1743.in‚0 disk.drbd1743.out‚ 4.05859375& disk.nvme0n1.out‚13053.78662109375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-1.in‚0.4375 disk.sda2.out‚0( disk.nvme0c0n1.out‚13053.78662109375* proc.memory.availableMB‚47538.98046875# proc.memory.rssMB‚ 11267.6484375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 181.48828125 disk.sda2.in‚0ûÈîœí‚(:‘ ô“ÅÓü³ôÒ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚128.941' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚71.63388888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01”-¹–î‚(:‹  ô“ÅÓÉŽ†×# disk.dm-0.out‚13116.31787109375 disk.sda3.in‚0.4375 disk.sda3.out‚ 182.078125% memory_percent‚27.461647987365723 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 47538.734375 network.recv‚487527 disk.drbd1743.in‚0( disk.nvme0c0n1.out‚13116.31787109375 disk.rbd1.in‚0 cpu‚13.959786124732402& disk.nvme0n1.out‚13116.31787109375 disk.rbd5.in‚0) proc.memory.percent‚17.19306707382202$ disk./.usageGB‚1243.220718383789 disk.nvme0n1.in‚ 37.5390625 disk.sda1.in‚0 disk.rbd5.out‚ 1.83984375 disk.rbd4.out‚ 1.83984375 disk.sda.out‚ 182.078125 disk.rbd3.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0.4375 disk.dm-1.out‚ 182.078125 proc.cpu.threads‚29 network.sent‚ 7.900605e+06 disk.drbd1743.out‚ 4.0859375 disk.rbd3.in‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 37.5390625 disk.sda.in‚0.4375 disk.rbd2.out‚ 1.83984375 disk.rbd1.out‚ 1.83984375) disk./.usagePercent‚83.43358554239845 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.dm-0.in‚ 37.5390625 Nã2ï- _wandbruntime‚3676  _runtime‚3676vDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7478939374080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153947556371968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858093986395054e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56470079 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153947716144128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146266342038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146236724 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145922028248064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142296009397472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145922028248064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33466895 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153400950582784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153400950582784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57815288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149756747912736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89766645 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7479026803712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153401055051776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146266317134848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56470077 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145922011208704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7478939374080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032Vm¬g×'ôÑO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56470077 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145922011208704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7478939374080 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7681374105600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚142296026436832 trainer/global_step‚8499 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7460721475904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145922011208704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚149756645070252 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56470079 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153401055051776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7478939374080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153947556371968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 % _timestamp‚1.7858093986395054e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56470079 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153947716144128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146266342038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146236724 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145922028248064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142296009397472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145922028248064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33466895 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153400950582784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153400950582784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57815288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149756747912736 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89766645 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7479026803712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153401055051776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146266317134848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¥Yî· 2õ- _wandbruntime‚3677  _runtime‚3677̘ÝÛ&öÕM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33470834 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89777182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7461598949238 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚8499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56476722 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145939177971712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56476720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153418996965376 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153419101441536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57822056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149774262715794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24351224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153965658975232 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56476720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145939177971712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145939195011072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142312749569792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153419101441536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56476722 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57822321 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7479818993664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146253904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146244232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33470832 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146253904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89767512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146283549155328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89777182 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149774365558390 _runtime‚ 3677.26715508 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146244232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146283524251648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145939195011072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7479818993664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 % _timestamp‚1.7858093989752045e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7461513146002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7479906430464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142312766609152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130•ø²q÷ÚM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7479906430464 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153419101441536 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57822056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149774262715794 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24351224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153965658975232 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56476720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145939177971712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145939195011072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142312749569792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153419101441536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56476722 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57822321 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7479818993664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146253904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146244232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 166056448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33470832 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 146253904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89767512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146283549155328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89777182 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149774365558390 _runtime‚ 3677.26715508 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146244232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146283524251648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145939195011072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7479818993664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 % _timestamp‚1.7858093989752045e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7461513146002 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7479906430464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142312766609152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_po"°5o ol.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33470834 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89777182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7461598949238 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 trainer/global_step‚8499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56476722 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145939177971712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56476720 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153418996965376 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146780349359104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24491960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7522935509504 trainer/global_step‚8549 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154303284868608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7726530932736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56802227 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7522935509504 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154303284868608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146780366398464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146780349359104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56802229 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33663747 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146780366398464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150637644992244 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147126672974848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90294647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154303389337600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154303389337600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58155469 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33663745 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7504609939412 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90284991 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154853228811264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150637542149760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7523022939136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56802229 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147126697878528 _step‚681 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154853073233408/TÖ'‚ÐO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154853073233408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146780349359104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24491960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7522935509504 trainer/global_step‚8549 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154303284868608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7726530932736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56802227 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7522935509504 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154303284868608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146780366398464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146780349359104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56802229 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33663747 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146780366398464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150637644992244 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147126672974848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90294647 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154303389337600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154303389337600 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58155469 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33663745 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7504609939412 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90284991 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154853228811264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150637542149760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7523022939136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56802229 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147126697878528 _step‚681©Ê¦ü2ƒ- _wandbruntime‚3696  _runtime‚3696|K4à„ÕM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7523815129088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56808872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33667684 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147143904995328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90295514 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150655262637898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147114056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56808870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154871182701568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7523902565888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154321331251200 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143149775225152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154321435727360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90295514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24494542 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146797516122112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7505401609510 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146797533161472 J@DeviceStatsMonitor.on_trsª=°ûain_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58162224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147114056 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858094181297958e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚154321435727360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146797533161472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150655159795302 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 trainer/global_step‚8549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58162482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154871340369408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146797516122112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147143880091648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90305184 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7727302609920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Û÷×à&…ÚM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24494798 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33667682 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7523902565888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7727435374080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147114056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56808870 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154871182701568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7523902565888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚154321331251200 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143149775225152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154321435727360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90295514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24494542 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146797516122112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7505401609510 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146797533161472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58162224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147114056 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858094181297958e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚154321435727360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146797533161472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150655159795302 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 trainer/global_step‚8549 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58162482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154871340369408 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146797516122112 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147143880091648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90305184 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7727302609920 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7523815129088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56808872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33667684 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147143904995328 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90295514 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150655262637898 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 _step‚682 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155205723623424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147638687509504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7566931644928 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33860597 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155205619154432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24635070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155205723623424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚147638704548864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90822649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 $ _timestamp‚1.785809437314454e+09 _runtime‚3715.609943987 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57134379 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147957028 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚685 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147638704548864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24635322 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7567019074560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58495665 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58495919 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7771696785408éÛExÐO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58495919 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7771696785408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7548498402920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7566931644928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 128577024 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚8599 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90812993 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57134379 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143970043668832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90822649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57134377 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155205723623424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147638687509504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7566931644928 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33860597 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155205619154432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24635070 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155205723623424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚147638704548864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90822649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 $ _timestamp‚1.785809437314454e+09 _runtime‚3715.609943987 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57134379 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147957028 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 _step‚685 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147638704548864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24635322 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7567019074560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58495665­)¹l2‘- _wandbruntime‚3715  _runtime‚3715ªJjoÜ&’ÖM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90823516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚8599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚155223770013184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7567898701312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚3715.942180987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90833186 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7772604380672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151536159717406 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143986766801792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147974208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33864532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147655854272512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24637926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33864534 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858094376502495e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7549290073018 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7549375876254 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24638185 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57141020 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7567811264512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90823516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7567811264512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155776703353856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148004235931648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147964536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147655871311872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151536056874810 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155223770013184 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58502458 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7567898701312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148004260835328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57141022 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0zXgÎá&“ÛM _step‚686 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚155223665537024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147655871311872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155776703353856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148004235931648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147964536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147655871311872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151536056874810 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155223770013184 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58502458 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7567898701312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148004260835328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57141022 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90823516 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 trainer/global_step‚8599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚155223770013184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7567898701312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _runtime‚3715.942180987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90833186 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7772604380672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151536159717406 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143986766801792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147974208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33864532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147655854272512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24637926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33864534 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 % _timestamp‚1.7858094376502495e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7549290073018 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7549375876254 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24638185 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57141020 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7567811264512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90823516 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7567811264512 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 trainer/global_step‚8649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91350651 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34057445 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7610927780352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91340995 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocateî a)ûd‚ 3021996032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156107953440256 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156107953440256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57466529 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24778822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7816723826176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚689 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91340995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7611015209984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7816862888960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144807035245472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58836269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156664108481024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144807052284832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7610927780352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚3734.955911534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57466527 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57466529 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156108057909248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592386866428 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148807522 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152399439151260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148497042699264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152399336308776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148817180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156664272447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156108057909248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7611015209984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24778567 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148847409558528ÐÛ¡×'žÑO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34057445 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7610927780352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91340995 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156107953440256 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156107953440256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57466529 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24778822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7816723826176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 _step‚689 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91340995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7611015209984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7816862888960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚144807035245472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58836269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156664108481024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚144807052284832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7610927780352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 _runtime‚3734.955911534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57466527 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57466529 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156108057909248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592386866428 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148807522 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152399439151260 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚148497042699264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152399336308776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148817180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156664272447488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156108057909248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7611015209984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24778567 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148847409558528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34057447 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592301063304 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚148497025659904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 163966464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57466527 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 trainer/global_step‚8649 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91350651 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121±é«dš›Ÿ‚(: °”ÅÓÙüÒ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.86' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚66.58888888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Ü|ñž ‚(:“  °”ÅÓ︔Ö disk.sda1.in‚0 disk.drbd1743.out‚ 4.14453125) proc.memory.percent‚17.19306707382202 disk.rbd5.out‚ 1.83984375 disk.rbd4.out‚ 1.83984375 disk.sda3.out‚ 185.08984375 disk.rbd3.in‚0& disk.nvme0n1.out‚14472.73388671875# proc.memory.rssMB‚ 11267.6484375 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 37.5390625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.83984375 disk.sda3.in‚0.4375 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0* proc.memory.availableMB‚47538.31640625 network.sent‚ 8.037695e+06 network.recv‚494859) disk./.usagePercent‚83.51534180446562 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 185.08984375 disk.rbd5.in‚0( disk.nvme0c0n1.out‚14472.73388671875 disk.dm-1.out‚ 185.08984375 disk.nvme0n1.in‚ 37.5390625$ memory_percent‚27.46228575706482% disk./.usageGB‚1244.4389457702637 disk.sda2.out‚0 cpu‚13.96169392244865 proc.cpu.threads‚29 disk.drbd1743.in‚0 disk.sda.in‚0.4375 disk.dm-1.in‚0.4375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd4.in‚0 disk.dm-0.in‚ 37.5390625# disk.dm-0.out‚14472.73388671875úÈ2¡- _wandbruntime‚3735  _runtime‚3735€ªÜ&¢ÖM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57473172 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156126104299008 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7858094569959726e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156126104299008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58842688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34061384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚3735.287999134 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148514209462272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91361188 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58842942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144823775417792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7593178536526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24781558 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7817628487680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7611894836736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148514209462272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57473170 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148864591771648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148834360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚8649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148824688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57473172ü|×Ͷ£ÛM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24781558 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7817628487680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7611894836736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148514209462272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57473170 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148864591771648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148834360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 trainer/global_step‚8649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148824688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57473172 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57473172 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156126104299008 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 % _timestamp‚1.7858094569959726e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156126104299008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58842688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34061384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 _runtime‚3735.287999134 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚148514209462272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91361188 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58842942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144823775417792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7593178536526²Œ‹Q2¤- _wandbruntime‚3735  _runtime‚3735„=ø«¥€ /loss/FBt1/detection_loss‚1.5638399024804432 epoch‚0 _runtime‚ 3735.28854028 2loss/SRqqt2/assignment_loss‚0.9305766559431146 2loss/FBt2/assignment_loss‚0.017544597177764666 1loss/SRqqt2/detection_loss‚1.5647639052056876 _step‚691 /loss/FBt2/detection_loss‚0.8670214058714919 &loss/total_loss‚17.185458876882226 % _timestamp‚1.7858094569970646e+09 /loss/FRt2/assignment_loss‚2.850696158409119 1loss/SRqqt1/detection_loss‚0.9136194347085643 trainer/global_step‚8649 0loss/FRt1/assignment_loss‚2.5319517730327137 2loss/SRqqt1/assignment_loss‚0.9030459458538909 3loss/FBt1/assignment_loss‚1.196760411090736e-05 -loss/FRt1/detection_loss‚2.38384151492226 .loss/FRt2/detection_loss‚2.658545615673065 ¨s ¦… /loss/FRt2/assignment_loss‚2.850696158409119 1loss/SRqqt1/detection_loss‚0.9136194347085643 trainer/global_step‚8649 0loss/FRt1/assignment_loss‚2.5319517730327137 2loss/SRqqt1/assignment_loss‚0.9030459458538909 3loss/FBt1/assignment_loss‚1.196760411090736e-05 -loss/FRt1/detection_loss‚2.38384151492226 .loss/FRt2/detection_loss‚2.658545615673065 /loss/FBt1/detection_loss‚1.5638399024804432 epoch‚0 _runtime‚ 3735.28854028 2loss/SRqqt2/assignment_loss‚0.9305766559431146 2loss/FBt2/assignment_loss‚0.017544597177764666 1loss/SRqqt2/detection_loss‚1.5647639052056876 _step‚691 /loss/FBt2/detection_loss‚0.8670214058714919 &loss/total_loss‚17.185458876882226 % _timestamp‚1.7858094569970646e+09³ÿ-„x›§‚(: ¿”ÅÓĦýÒ gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚67.34444444444443 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍZ¾æ›¨‚(:  ¿”ÅÓ½üÜØ) disk./.usagePercent‚83.51520970457796 disk.dm-0.in‚ 37.5390625 disk.nvme0n1.in‚ 37.5390625 disk.rbd0.out‚0 disk.rbd2.out‚ 1.83984375 disk.sda3.in‚0.4375# disk.dm-0.out‚14734.42138671875& disk.nvme0n1.out‚14734.42138671875 cpu‚13.962148585568247 disk.rbd4.out‚ 1.83984375 disk.nvme0c0n1.in‚ 37.5390625( disk.nvme0c0n1.out‚14734.42138671875 disk.rbd2.in‚0 disk.dm-1.out‚ 185.8984375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.rbd3.in‚0 disk.drbd1743.in‚0 disk.drbd1743.out‚ 4.1640625 disk.sda1.out‚0 disk.rbd5.out‚ 1.83984375 disk.sda.in‚0.4375# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202 network.sent‚ 8.071663e+06* proc.memory.availableMB‚47538.74609375 network.recv‚496785% disk./.usageGB‚1244.4369773864746 disk.rbd5.in‚0 disk.dm-1.in‚0.4375 disk.sda2.out‚0 proc.cpu.threads‚29 disk.rbd0.in‚0 disk.sda.out‚ 185.8984375 disk.sda3.out‚ 185.8984375 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd4.in‚0 disk.rbd3.out‚0$ memory_percent‚27.46163010597229žÖÛ2©- _wandbruntime‚3754  _runtime‚3754´GœÉªÃ _runtime‚3754.118644896 _step‚692 ' lr-AdamW/pg1‚0.00022419225023567776 ' lr-AdamW/pg2‚0.00022419225023567776 trainer/global_step‚8699 % _timestamp‚1.7858094758269873e+09r·S°Î«È _step‚692 ' lr-AdamW/pg1‚0.00022419225023567776 ' lr-AdamW/pg2‚0.00022419225023567776 trainer/global_step‚8699 % _timestamp‚1.7858094758269873e+09 _runtime‚3754.118644896´O±2¬- _wandbruntime‚3754  _runtime‚3754o÷â Ñ'­ËO YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149677332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24922562 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153280336230768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34254295 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149707765398528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59176859 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚8699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91868997 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59176620 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7861864504320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7636275329936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157010287726080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149355363810304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157569604999168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149707740494848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7636189526812 _runtime‚3754.122925305 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91878653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7655011345408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157010287726080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149667674 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153280233388284 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7655011345408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149355380849664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145644060900832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57798679 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7654923915776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34254297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145644043861472 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57798677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157569758479872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _step‚693 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149355363810304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91868997 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 $ _timestamp‚1.785809475827992e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24922325ÊÖ¼ÿÖ'®ÐO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149677332 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24922562 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153280336230768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34254295 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149707765398528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59176859 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 trainer/global_step‚8699 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91868997 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59176620 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7861864504320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7636275329936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157010287726080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149355363810304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157569604999168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149707740494848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7636189526812 _runtime‚3754.122925305 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91878653 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7655011345408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157010287726080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149667674 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153280233388284 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7655011345408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚149355380849664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145644060900832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57798679 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7654923915776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34254297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145644043861472 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57798677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157569758479872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 _step‚693 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149355363810304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91868997 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 $ _timestamp‚1.785809475827992e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24922325 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712µv8(N2¯- _wandbruntime‚3754  _runtime‚3754‚¢m& °ÖM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚694 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153297851033826 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7637067000034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149694512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157587878631424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145660801073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858094761598477e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91879520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149724972515328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59183431 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91879520 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149684840 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YAªÚ¶GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7655890972160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157028438584832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7655803535360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149372547612672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91889190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34258232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145660784033792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚3754.451943791 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149694512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8699 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57805322 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149724947611648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130%ØÄká&±ÛM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7655803535360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153297953876422 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149372530573312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7862906116096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57805322 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145660784033792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 _runtime‚3754.451943791 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149694512 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8699 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57805322 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149724947611648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚694 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153297851033826 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7637067000034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149694512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157587878631424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145660801073152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 % _timestamp‚1.7858094761598477e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91879520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149724972515328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59183431 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91879520 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149684840 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7655890972160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157028438584832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7655803535360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149372547612672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91889190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34258232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59517419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158475124761600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150568096334848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146481069516832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25066026 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150213701960704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7858094949520426e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157912726480896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150537484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158475284533760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7907028426752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7680163793444 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157912622011904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58130829 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59517171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25066272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150568121238528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154161130467792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7698920051200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154161233310276 trainer/global_step‚8749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58130829 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146481052477472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25066026 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150213701960704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7858094949520426e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157912726480896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150537484 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158475284533760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7907028426752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7680163793444 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157912622011904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 159772160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YLDeviceStatsMonitor.on_train_batch_starHÉ¯Ó t/inactive_split_bytes.large_pool.peak‚86278144 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58130829 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59517171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25066272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150568121238528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154161130467792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7698920051200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154161233310276 trainer/global_step‚8749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58130829 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚146481052477472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59517419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158475124761600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150568096334848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚146481069516832 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150230885763072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158493393585152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158493235917312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858094952798367e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154178850955930 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7680955463542 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150544992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150585303451648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚146497792649792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34455082 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58137472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34455084 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157930772870656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150230885763072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7699799670784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚157930772870656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58137470 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92417192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7699799670784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚3773.571859933 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58137472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7699887107584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157930668394496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150230868723712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150554664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7907932465664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154178748113334 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25068910 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59523992 trainer/global_step‚8749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150230868723712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150554664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7907932465664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚154178748113334 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 _step‚698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25068910 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59523992 trainer/global_step‚8749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150230868723712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150230885763072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158493393585152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158493235917312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858094952798367e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚154178850955930 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7680955463542 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150544992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150585303451648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚146497792649792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34455082 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58137472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34455084 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856ºÐ¹2À- _wandbruntime‚3773  _runtime‚3773–k· Á„ %loss/total_loss‚18.25119052396147 _runtime‚3773.572571267 0loss/FRt2/assignment_loss‚3.0384623162886677 2loss/SRqqt1/assignment_loss‚0.8952689855457185 3loss/FBt1/assignment_loss‚0.0006197583063164794 3loss/FBt2/assignment_loss‚8.217197997926936e-05 .loss/FRt2/detection_loss‚2.619755432886236 epoch‚0 1loss/SRqqt1/detection_loss‚0.9342184457081285 /loss/FBt1/detection_loss‚1.7533967724690835 trainer/global_step‚8749 % _timestamp‚1.7858094952810407e+09 _step‚699 /loss/FRt1/detection_loss‚2.4768055602010475 /loss/FBt2/detection_loss‚1.2222350709761183 0loss/FRt1/assignment_loss‚2.4111553365925706 2loss/SRqqt2/assignment_loss‚0.7805646364935316 1loss/SRqqt2/detection_loss‚2.1186260365140743&i‰ epoch‚0 1loss/SRqqt1/detection_loss‚0.9342184457081285 /loss/FBt1/detection_loss‚1.7533967724690835 trainer/global_step‚8749 % _timestamp‚1.7858094952810407e+09 _step‚699 /loss/FRt1/detection_loss‚2.4768055602010475 /loss/FBt2/detection_loss‚1.2222350709761183 0loss/FRt1/assignment_loss‚2.4111553365925706 2loss/SRqqt2/assignment_loss‚0.7805646364935316 1loss/SRqqt2/detection_loss‚2.1186260365140743 %loss/total_loss‚18.25119052396147 _runtime‚3773.572571267 0loss/FRt2/assignment_loss‚3.0384623162886677 2loss/SRqqt1/assignment_loss‚0.8952689855457185 3loss/FBt1/assignment_loss‚0.0006197583063164794 3loss/FBt2/assignment_loss‚8.217197997926936e-05 .loss/FRt2/detection_loss‚2.619755432886236»WüTzœÃ‚(:‘ Ý”ÅÓêÃÈÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚155.221' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.23388888888888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õà´…Ä‚(:’  Ý”ÅÓ”¼ôÕ network.sent‚ 8.151375e+06 disk.sda1.out‚0 disk.rbd5.in‚0 disk.nvme0c0n1.in‚ 37.7734375 disk.rbd1.in‚0 disk.dm-0.in‚ 37.7734375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 37.7734375( disk.nvme0c0n1.out‚14794.10205078125 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 191.61328125 disk.drbd1743.in‚0) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29 network.recv‚500746& disk.nvme0n1.out‚14794.10205078125 disk.rbd0.in‚0 disk.sda.in‚0.4375 disk.rbd1.out‚ 1.83984375 cpu‚13.963042584572081 disk.rbd4.in‚0 disk.dm-1.in‚0.4375 disk.sda2.in‚0# proc.memory.rssMB‚ 11267.6484375 disk.sda1.in‚0 disk.sda3.in‚0.4375$ memory_percent‚27.46208906173706) proc.memory.availableMB‚ 47538.4453125( disk./.usagePercent‚83.5156825504937 disk.rbd0.out‚0 disk.rbd4.out‚ 1.83984375 disk.sda2.out‚0# disk.dm-0.out‚14794.10205078125 disk.rbd3.out‚0 disk.sda.out‚ 191.61328125 disk.rbd2.in‚0 disk.dm-1.out‚ 191.61328125 disk.drbd1743.out‚ 4.19921875% disk./.usageGB‚1244.4440231323242 disk.rbd5.out‚ 1.83984375¢²’92Å- _wandbruntime‚3792  _runtime‚3792×5‡–ÉÆà ' lr-AdamW/pg2‚0.00022676946888420838 trainer/global_step‚8799 % _timestamp‚1.7858095143362014e+09 _runtime‚3792.627861047 _step‚700 ' lr-AdamW/pg1‚0.00022676946888420838Œ¢ŸÎÎÇÈ ' lr-AdamW/pg2‚0.00022676946888420838 trainer/global_step‚8799 % _timestamp‚1.7858095143362014e+09 _runtime‚3792.627861047 _step‚700 ' lr-AdamW/pg1‚0.00022676946888420838¼ FÞG2È- _wandbruntime‚3792  _runtime‚3792v¼Èy'ÉÌO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151072057150464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158815060766720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743003616256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147318078132832 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 N@DeviceStatsMonitor.on_train_batch_s¤Á-e«$tart/active_bytes.all.current‚ 104468992 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7742916186624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7952364712960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7723966453828 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _runtime‚3792.631928589 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25209134 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _step‚701 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92925001 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58462977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158815060766720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159380679922176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158814956297728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151428477078528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58462979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151387978 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7742916186624 trainer/global_step‚8799 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858095143371587e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151072057150464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25209392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158814956297728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151428477078528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58462979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151387978 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7742916186624 trainer/global_step‚8799 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 % _timestamp‚1.7858095143371587e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151072057150464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25209392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7742916186624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7952364712960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7723966453828 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 _runtime‚3792.631928589 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25209134 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 _step‚701 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92925001 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58462977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158815060766720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159380679922176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151089223913472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151405144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92935524 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7743795806208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58469622 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92945194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25211975 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92935524 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155059645192842 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159398787907584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158833107156480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151089223913472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58469620 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7743883243008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147334818305152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59863907 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34651934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92945194 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147334801265792 $ _timestamp‚1.785809514668511e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7743795806208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34651932 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7953263477248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25212227 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚702 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151089206874112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151445684195328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚3792.960434858 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7953128615936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151414816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59864161 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151089206874112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151414816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 159764992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8799~ÀÄqN ÍÚM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 _step‚702 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151089206874112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151445684195328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 _runtime‚3792.960434858 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7953128615936 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151414816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59864161 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151089206874112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151414816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 TFDeviceStatsMonitor.on_train_batch_en,K‰û’d/inactive_split_bytes.all.current‚ 159764992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8799 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151089223913472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151405144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92935524 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7743795806208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58469622 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92945194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25211975 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92935524 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155059645192842 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159398787907584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158833107156480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151089223913472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58469620 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7743883243008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147334818305152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59863907 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34651934 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92945194 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147334801265792 $ _timestamp‚1.785809514668511e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7743795806208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34651932 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7953263477248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25212227¾ȨìÒ2Î- _wandbruntime‚3792  _runtime‚3792³mcl Ï„ _step‚703 0loss/FRt1/assignment_loss‚2.4894027363422304 3loss/FBt1/assignment_loss‚0.0002640182336397364 4loss/FBt2/assignment_loss‚1.0276655639872529e-07 &loss/total_loss‚18.033209621338006 0loss/SRqqt2/detection_loss‚1.708934702186121 trainer/global_step‚8799 /loss/FRt2/assignment_loss‚2.953115675338479 2loss/SRqqt2/assignment_loss‚1.0625849281334214 .loss/FBt1/detection_loss‚2.181338295371582 % _timestamp‚1.7858095146697578e+09 2loss/SRqqt1/assignment_loss‚1.0053555047873295 /loss/FRt1/detection_loss‚2.3686139685816543 /loss/FRt2/detection_loss‚2.4647105810947196 1loss/SRqqt1/detection_loss‚1.0671035410610563 /loss/FBt2/detection_loss‚0.7317855674412166 epoch‚0 _runtime‚3792.961266116%0?Љ trainer/global_step‚8799 /loss/FRt2/assignment_loss‚2.953115675338479 2loss/SRqqt2/assignment_loss‚1.0625849281334214 .loss/FBt1/detection_loss‚2.181338295371582 % _timestamp‚1.7858095146697578e+09 2loss/SRqqt1/assignment_loss‚1.0053555047873295 /loss/FRt1/detection_loss‚2.3686139685816543 /loss/FRt2/detection_loss‚2.4647105810947196 1loss/SRqqt1/detection_loss‚1.0671035410610563 /loss/FBt2/detection_loss‚0.7317855674412166 epoch‚0 _runtime‚3792.961266116 _step‚703 0loss/FRt1/assignment_loss‚2.4894027363422304 3loss/FBt1/assignment_loss‚0.0002640182336397364 4loss/FBt2/assignment_loss‚1.0276655639872529e-07 &loss/total_loss‚18.033209621338006 0loss/SRqqt2/detection_loss‚1.708934702186121¿ aiœÑ‚(:‘ ì”ÅÓ‘ÂãÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚158.946' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚88.30333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îXt¹ŸÒ‚(:”  ì”ÅÓ虉Ö disk.rbd2.in‚0# disk.dm-0.out‚14794.35986328125 disk.rbd1.out‚ 1.83984375 disk.sda2.out‚0 disk.nvme0n1.in‚ 37.7734375 disk.sda2.in‚0 disk.drbd1743.out‚ 4.20703125 disk.rbd5.out‚ 1.83984375 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.nvme0c0n1.in‚ 37.7734375( disk.nvme0c0n1.out‚14794.35986328125 disk.rbd2.out‚ 1.83984375 disk.rbd1.in‚0 disk.drbd1743.in‚0 network.sent‚ 8.196162e+06 disk.sda.out‚ 192.34765625 disk.dm-1.out‚ 192.34765625 disk.rbd3.in‚0 disk.rbd3.out‚0% memory_percent‚27.462780475616455) proc.memory.availableMB‚ 47537.9921875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0.4375 disk.sda3.out‚ 192.34765625 disk.dm-0.in‚ 37.7734375) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29& disk.nvme0n1.out‚14794.35986328125 disk.rbd4.out‚ 1.83984375# proc.memory.rssMB‚ 11267.6484375 cpu‚13.963504771817044 network.recv‚502535% disk./.usageGB‚1244.4380226135254 disk.sda1.in‚0 disk.sda.in‚0.4375 disk.dm-1.in‚0.4375) disk./.usagePercent‚83.51527985064233 disk.rbd5.in‚0Ø{œÓ‚(:‘ û”ÅÓ—†äÒ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚132.821' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚73.78944444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L”0”Ô‚(:‰  û”ÅÓèÙ¬× disk.rbd0.in‚0 proc.cpu.threads‚29% disk./.usageGB‚1244.4390754699707 disk.dm-0.in‚ 37.7734375& disk.nvme0n1.out‚14796.92333984375# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202 disk.rbd3.in‚0( disk.nvme0c0n1.out‚14796.92333984375 disk.sda.out‚ 193.109375 disk.rbd2.in‚0 disk.dm-1.in‚0.4375 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1743.out‚ 4.2265625 network.recv‚504887 disk.sda.in‚0.4375 disk.sda3.in‚0.4375# disk.dm-0.out‚14796.92333984375 disk.nvme0n1.in‚ 37.7734375 disk.sda1.in‚0 disk.rbd5.out‚ 1.83984375( proc.memory.availableMB‚ 47538.046875 disk.rbd5.in‚0 disk.rbd4.in‚0 network.sent‚ 8.241077e+06 disk.nvme0c0n1.in‚ 37.7734375 disk.dm-1.out‚ 193.109375 disk.rbd1.out‚ 1.83984375$ memory_percent‚27.46269702911377 cpu‚13.9638911702904) disk./.usagePercent‚83.51535050872178 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 1.83984375 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 193.109375 disk.rbd3.out‚0¬®`­2Õ- _wandbruntime‚3811  _runtime‚3811oA=ÅÉÖà ' lr-AdamW/pg1‚0.00022805807820847366 ' lr-AdamW/pg2‚0.00022805807820847366 trainer/global_step‚8849 % _timestamp‚1.7858095331865504e+09 _runtime‚3811.478197658 _step‚7049• ÐÎ×È ' lr-AdamW/pg1‚0.00022805807820847366 ' lr-AdamW/pg2‚0.00022805807820847366 trainer/global_step‚8849 % _timestamp‚1.7858095331865504e+09 _runtime‚3811.478197658 _step‚704À.Z,Ó2Ø- _wandbruntime‚3811  _runtime‚3811q¯­Ð'ÙÊO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151930395300864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159717395052544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58795129 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚3811.482928615 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148155069709472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚705 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152248130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58795127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160286184299008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151930378261504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60197885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8849 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151930395300864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7786999751680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7767854917336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25353040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148155086748832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7786912322048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159717395052544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60198145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159717290583552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93453003 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58795129 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7767940720460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151930378261504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155923027469292 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 # _timestamp‚1.78580953318748e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159717290583552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152288808014848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7997517444096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93462659 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7997376284160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58795127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160286350362624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93462659 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93453003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34844845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Od«YÕ'ÚÏO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151930378261504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155923027469292 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 # _timestamp‚1.78580953318748e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159717290583552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152288808014848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7997517444096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93462659 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7997376284160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58795127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160286350362624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 141159936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93462659 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93453003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34844845 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151930395300864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159717395052544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58795129 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 _runtime‚3811.482928615 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148155069709472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚705 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152248130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58795127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160286184299008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151930378261504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60197885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 166063616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚8849 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151930395300864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7786999751680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7767854917336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25353040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148155086748832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7786912322048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159717395052544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60198145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159717290583552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93453003 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58795129 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7767940720460 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251ÁZ-µ™2Û- _wandbruntime‚3811  _runtime‚3811¼Î ˆvÜÖM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/aNsFf&ctive.large_pool.freed‚58801770 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159735441442304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93463526 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _runtime‚3811.809321532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152265296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93473196 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159735336966144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7787791941632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25356108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152306015131648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7768732390558 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159735441442304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152306040035328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155940645114946 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152265296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58801772 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚8849 % _timestamp‚1.7858095335173914e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152274968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58801770 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7768818193794 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7787879378432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93473196 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152274968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7787791941632 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151947562063872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60204892 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34848782 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7998280790016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58801772 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60204644 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚706 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7787879378432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148171826921152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93463526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151947545024512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151947562063872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121ZAz3á&ÝÛM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7768732390558 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159735441442304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152306040035328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155940645114946 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152265296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58801772 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚8849 % _timestamp‚1.7858095335173914e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152274968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58801770 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7768818193794 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7787879378432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93473196 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152274968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7787791941632 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151947562063872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60204892 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34848782 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7998280790016 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58801772 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60204644 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚706 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7787879378432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148171826921152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93463526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151947545024512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151947562063872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58801770 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159735441442304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93463526 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 _runtime‚3811.809321532 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152265296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93473196 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159735336966144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7787791941632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25356108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152306015131648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÂÆ´F2Þ- _wandbruntime‚3811  _runtime‚3811{Ä·#ßÿ .loss/FRt2/detection_loss‚2.545659723215633 _step‚707 0loss/FRt1/assignment_loss‚1.8050858621244077 4loss/FBt2/assignment_loss‚0.00010444287141075925 0loss/SRqqt1/detection_loss‚1.090050473588137 % _timestamp‚1.7858095335186143e+09 .loss/FRt1/detection_loss‚2.379445995077675 1loss/SRqqt2/detection_loss‚1.6585578026456966 .loss/FBt1/detection_loss‚2.098287702848514 .loss/FBt2/detection_loss‚0.554437716913526 epoch‚0 trainer/global_step‚8849 _runtime‚3811.810102656 2loss/SRqqt1/assignment_loss‚0.9322121353759665 1loss/FBt1/assignment_loss‚0.07385549099635587 %loss/total_loss‚16.61787837501351 0loss/FRt2/assignment_loss‚2.6952987790107725 2loss/SRqqt2/assignment_loss‚0.7848822503454156€m< à„ 0loss/FRt1/assignment_loss‚1.8050858621244077 4loss/FBt2/assignment_loss‚0.00010444287141075925 0loss/SRqqt1/detection_loss‚1.090050473588137 % _timestamp‚1.7858095335186143e+09 .loss/FRt1/detection_loss‚2.379445995077675 1loss/SRqqt2/detection_loss‚1.6585578026456966 .loss/FBt1/detection_loss‚2.098287702848514 .loss/FBt2/detection_loss‚0.554437716913526 epoch‚0 trainer/global_step‚8849 _runtime‚3811.810102656 2loss/SRqqt1/assignment_loss‚0.9322121353759665 1loss/FBt1/assignment_loss‚0.07385549099635587 %loss/total_loss‚16.61787837501351 0loss/FRt2/assignment_loss‚2.6952987790107725 2loss/SRqqt2/assignment_loss‚0.7848822503454156 .loss/FRt2/detection_loss‚2.545659723215633 _step‚707îLÀ®›á‚(: Š•ÅÓÇõÁÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚151.55' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚84.19444444444444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥$ý•—â‚(:Œ  Š•ÅÓƺÍ× network.sent‚ 8.243568e+06% disk./.usageGB‚1244.4401512145996 disk.sda1.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.19306707382202 cpu‚13.964473455812746 disk.sda2.out‚0# proc.memory.rssMB‚ 11267.6484375 disk.nvme0n1.in‚38.21875 disk.rbd4.in‚0 disk.rbd1.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚29 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd1743.out‚4.234375 disk.dm-0.in‚38.21875 disk.rbd5.in‚0 disk.rbd5.out‚ 1.83984375 disk.drbd1743.in‚0# disk.dm-0.out‚14798.78759765625$ memory_percent‚27.46289372444153) disk./.usagePercent‚83.51542270284644& disk.nvme0n1.out‚14798.78759765625 disk.rbd4.out‚ 1.83984375 disk.sda.out‚ 193.72265625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 193.72265625 disk.dm-1.out‚ 193.72265625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚38.21875( disk.nvme0c0n1.out‚14798.78759765625 disk.sda3.in‚0.4375 disk.dm-1.in‚0.4375 disk.sda2.in‚0 network.recv‚505756 disk.sda1.out‚0 disk.sda.in‚0.4375* proc.memory.availableMB‚47537.91796875­JqŽ2ã- _wandbruntime‚3830  _runtime‚3830ºÌÖÉäà trainer/global_step‚8899 % _timestamp‚1.7858095525836933e+09 _runtime‚3830.875322744 _step‚708 ' lr-AdamW/pg1‚0.00022934668753273895 ' lr-AdamW/pg2‚0.00022934668753273895en£²ÎåÈ ' lr-AdamW/pg1‚0.00022934668753273895 ' lr-AdamW/pg2‚0.00022934668753273895 trainer/global_step‚8899 % _timestamp‚1.7858095525836933e+09 _runtime‚3830.875322744 _step‚708Ä1ÓÚQ2æ- _wandbruntime‚3830  _runtime‚3830¶Tdù÷#çÌO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8042533886464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7830995887104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152788716411904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7811829183968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93990661 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161191697741312 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160619624869376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152788716411904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59127277 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160619729338368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156803821706316 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858095525846083e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160619624869376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59127279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60536998 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7811743380844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93990661 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152788733451264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25495303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚3830.879934586 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35041695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156803924548800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59127279 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60537264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93981005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59127277 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7830995887104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Fz¯×'èÑO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7811743380844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93990661 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152788733451264 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25495303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚3830.879934586 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35041695 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156803924548800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59127279 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60537264 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93981005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59127277 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7830995887104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8042533886464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7830995887104 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152788716411904 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7811829183968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93990661 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161191697741312 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160619624869376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152788716411904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59127277 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160619729338368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156803821706316 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 % _timestamp‚1.7858095525846083e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160619624869376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59127279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60536998Åt7ø2é- _wandbruntime‚3831  _runtime‚3831+ô.ÒÜ&êÖM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153166370971648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93991528 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160637775728128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152805883174912 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7812620854066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59133922 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94001198 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60544092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35045632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152805883174912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8043440636928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚8899 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59133922 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153135120 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59133920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152805900214272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25498458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149008818497792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858095529150274e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94001198 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7812706657302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35045634 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152805900214272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153166395875328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7831788077056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149008835537152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3831.206976616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7831875513856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7831875513856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161209811608576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160637775728128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60543829 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153135120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25498197 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153125448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160637671251968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153125448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚710 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161209965082112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536ºª@qá&ëÛM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8043569206784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59133920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93991528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153135120 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59133920 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152805900214272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25498458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149008818497792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 % _timestamp‚1.7858095529150274e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94001198 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7812706657302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35045634 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152805900214272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153166395875328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7831788077056 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149008835537152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3831.206976616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7831875513856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7831875513856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161209811608576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160637775728128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60543829 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153135120 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25498197 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153125448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160637671251968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153125448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚710 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161209965082112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153166370971648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93991528 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160637775728128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152805883174912 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7812620854066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59133922 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94001198 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60544092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35045632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152805883174912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8043440636928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 trainer/global_step‚8899 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59133922 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154009519694848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161521959155200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157684821628308 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7855717647476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153647054562304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153647071601664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153978092 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94509007 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25639078 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35238547 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161521959155200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162097416054784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149829103980832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149829086941472 _runtime‚3850.005939696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚713ê½ÝK×'öÑO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154009519694848 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161521959155200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157684821628308 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7855717647476 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153647054562304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153647071601664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153978092 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94509007 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25639078 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35238547 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161521959155200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162097416054784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149829103980832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149829086941472 _runtime‚3850.005939696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚713 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7874904592896 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161540005537792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153995272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59466070 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153664238364672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35242484 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153664238364672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7875871649280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7856509317574 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8088774152704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59466072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154026726811648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35242482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7856595120810 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60884170 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162115370297344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157702439273962 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60884436 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8088643485696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161540110013952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162115525868032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚8949 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7875784212480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94519530 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161540110013952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59466070 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7875784212480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59466072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153664221325312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25641688Ò{W¥ùÛM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 uiî<&KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161540110013952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59466070 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7875784212480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59466072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153664221325312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25641688 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153664221325312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153995272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 % _timestamp‚1.7858095720414016e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161540005537792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153995272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59466070 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚153664238364672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35242484 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153664238364672 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7875871649280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7856509317574 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 155570688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8088774152704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59466072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154026726811648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35242482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7856595120810 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60884170 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162115370297344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157702439273962 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60884436 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8088643485696 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161540110013952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162115525868032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 trainer/global_step‚8949 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7875784212480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94519530 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Êv˜2ú- _wandbruntime‚3850  _runtime‚3850BMû˜ûÿ /loss/FRt2/detection_loss‚2.4685887722081916 _runtime‚3850.334579905 2loss/SRqqt1/assignment_loss‚1.0645241044374585 1loss/SRqqt2/assignment_loss‚1.009660552289071 .loss/FBt1/detection_loss‚1.279605496581644 _step‚715 /loss/FRt1/assignment_loss‚2.102565105957321 1loss/FBt1/assignment_loss‚0.02098399636221302 3loss/FBt2/assignment_loss‚0.0016817301129888943 0loss/FBt2/detection_loss‚0.44337438081244107 &loss/total_loss‚16.489275120869543 trainer/global_step‚8949 # _timestamp‚1.78580957204309e+09 0loss/FRt2/assignment_loss‚2.7405073185299718 /loss/FRt1/detection_loss‚2.4066996915808208 1loss/SRqqt1/detection_loss‚0.9564043764319075 0loss/SRqqt2/detection_loss‚1.994679595565512 epoch‚0`:% ü„ 1loss/SRqqt1/detection_loss‚0.9564043764319075 0loss/SRqqt2/detection_loss‚1.994679595565512 epoch‚0 /loss/FRt2/detection_loss‚2.4685887722081916 _runtime‚3850.334579905 2loss/SRqqt1/assignment_loss‚1.0645241044374585 1loss/SRqqt2/assignment_loss‚1.009660552289071 .loss/FBt1/detection_loss‚1.279605496581644 _step‚715 /loss/FRt1/assignment_loss‚2.102565105957321 1loss/FBt1/assignment_loss‚0.02098399636221302 3loss/FBt2/assignment_loss‚0.0016817301129888943 0loss/FBt2/detection_loss‚0.44337438081244107 &loss/total_loss‚16.489275120869543 trainer/global_step‚8949 # _timestamp‚1.78580957204309e+09 0loss/FRt2/assignment_loss‚2.7405073185299718 /loss/FRt1/detection_loss‚2.4066996915808208Ë›â«÷œý‚(:‘ ¨•ÅÓ¹§ÃÒ gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚122.822' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚68.23444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¡¼×šþ‚(:  ¨•ÅÓǵ×Ö# disk.dm-0.out‚14802.52392578125 disk.nvme0c0n1.in‚ 38.2265625 disk.sda3.in‚0.4375 disk.rbd1.out‚ 1.83984375$ memory_percent‚27.46339440345764 network.sent‚ 8.333404e+06 disk.sda2.in‚0& disk.nvme0n1.out‚14802.52392578125 disk.rbd4.out‚ 1.83984375 disk.sda.in‚0.4375 disk.sda.out‚ 195.1484375 disk.dm-1.out‚ 195.1484375 disk.rbd3.out‚0 proc.cpu.threads‚29% disk./.usageGB‚1244.4439430236816 disk.rbd5.out‚ 1.83984375 disk.rbd4.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 11267.6484375 cpu‚13.965477003384095) disk./.usagePercent‚83.51567717433548 disk.nvme0n1.in‚ 38.2265625 disk.sda1.in‚0 disk.drbd1743.out‚4.265625 disk.rbd0.out‚0* proc.memory.availableMB‚47537.58984375 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0) proc.memory.percent‚17.19306707382202 disk.rbd3.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚14802.52392578125 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 195.1484375 disk.dm-1.in‚0.4375 network.recv‚509977 disk.dm-0.in‚ 38.2265625qkî–2ÿ- _wandbruntime‚3869  _runtime‚3869HêFÚɀà ' lr-AdamW/pg1‚0.00023192390618126957 ' lr-AdamW/pg2‚0.00023192390618126957 trainer/global_step‚8999 % _timestamp‚1.7858095908850636e+09 _runtime‚3869.176740767 _step‚716¹]ÐÉÎÈ _step‚716 ' lr-AdamW/pg1‚0.00023192390618126957 ' lr-AdamW/pg2‚0.00023192390618126957 trainer/global_step‚8999 % _timestamp‚1.7858095908850636e+09 _runtime‚3869.176740767ÌÞjΨ2‚- _wandbruntime‚3869  _runtime‚3869’xuLÒ'ƒÌO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59791579 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163002957369344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95046665 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8133056930816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154505409752064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25781360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150666112596832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154505392712704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35435395 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154505409752064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154838244 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 trainer/global_step‚8999 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59791579 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚717 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162424397910016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154828586 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154505392712704 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162424293441024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158565718707816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95046665 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95037009 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25781126 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3869.180927331 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35435397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 % _timestamp‚1.7858095908860903e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7918988157952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7918900728320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032Rû_p,#„ÑO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154505409752064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154838244 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 trainer/global_step‚8999 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59791579 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚717 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162424397910016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154828586 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154505392712704 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162424293441024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158565718707816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95046665 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95037009 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25781126 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3869.180927331 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35435397 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 % _timestamp‚1.7858095908860903e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7918988157952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7918900728320 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59791579 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163002957369344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95046665 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8133056930816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154505409752064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25781360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.lÍn׫on_train_batch_start/requested_bytes.large_pool.allocated‚150666112596832 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154505392712704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35435395 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2Í%«¤²2…- _wandbruntime‚3869  _runtime‚3869¢ûöÛ&†ÕM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150682852769152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95057202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35439334 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162442444299776 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61223275 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95047532 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154855424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150682835729792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚718 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162442339823616 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25783943 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154522576515072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158583233510874 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 $ _timestamp‚1.785809591220857e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162442444299776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154845752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154522559475712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61223532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚8999 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95047532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154855424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7919780347904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25784198 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163021069976064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2L”à&‡ÚM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61223275 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95047532 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154855424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150682835729792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _step‚718 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162442339823616 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25783943 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154522576515072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158583233510874 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 $ _timestamp‚1.785809591220857e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162442444299776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154845752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154522559475712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61223532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚8999 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95047532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154855424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7919780347904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25784198 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163021069976064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150682852769152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95057202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35439334 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162442444299776 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Χ_èÌ2ˆ- _wandbruntime‚3869  _runtime‚3869ºzb‚ ‰ƒ trainer/global_step‚8999 % _timestamp‚1.7858095912220933e+09 0loss/FRt1/assignment_loss‚2.4140136980674636 2loss/SRqqt2/assignment_loss‚1.1396428680238433 /loss/FBt1/detection_loss‚1.2074105097827585 epoch‚0 0loss/FRt2/assignment_loss‚2.7772471451423537 1loss/FBt2/assignment_loss‚0.01660862149717133 /loss/FRt1/detection_loss‚2.6624902243345554 1loss/SRqqt1/detection_loss‚0.9423249330574355 &loss/total_loss‚17.108643464758636 _runtime‚3869.513878275 _step‚719 2loss/SRqqt1/assignment_loss‚0.8188478942045017 /loss/FBt2/detection_loss‚0.3546183422440663 2loss/FBt1/assignment_loss‚0.013745789407110465 /loss/FRt2/detection_loss‚2.7759254612553286 1loss/SRqqt2/detection_loss‚1.9857679777420485>+Šˆ 0loss/FRt1/assignment_loss‚2.4140136980674636 2loss/SRqqt2/assignment_loss‚1.1396428680238433 /loss/FBt1/detection_loss‚1.2074105097827585 epoch‚0 0loss/FRt2/assignment_loss‚2.7772471451423537 1loss/FBt2/assignment_loss‚0.01660862149717133 /loss/FRt1/detection_loss‚2.6624902243345554 1loss/SRqqt1/detection_loss‚0.9423249330574355 &loss/total_loss‚17.108643464758636 _runtime‚3869.513878275 _step‚719 2loss/SRqqt1/assignment_loss‚0.8188478942045017 /loss/FBt2/detection_loss‚0.3546183422440663 2loss/FBt1/assignment_loss‚0.013745789407110465 /loss/FRt2/detection_loss‚2.7759254612553286 1loss/SRqqt2/detection_loss‚1.9857679777420485 trainer/global_step‚8999 % _timestamp‚1.7858095912220933e+09ϽñAí‘‹‚(:† ·•ÅÓ½¶ÐÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚126.081' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚70.045 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 »×„žŒ‚(:“  ·•ÅÓ¼‘Õ network.recv‚512271 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 47537.6015625# disk.dm-0.out‚14804.53759765625 disk.rbd5.in‚0 disk.sda.in‚0.4375 disk.sda.out‚ 196.01953125 disk.rbd1.out‚ 1.83984375$ memory_percent‚27.46337652206421 disk.drbd1743.in‚0 disk.dm-0.in‚ 38.2265625 disk.sda1.in‚0 disk.rbd4.in‚0 disk.rbd2.out‚ 1.83984375 disk.sda3.out‚ 196.01953125 disk.dm-1.in‚0.4375 proc.cpu.threads‚29 network.sent‚ 8.378195e+06 disk.rbd3.in‚0 disk.nvme0n1.in‚ 38.2265625 cpu‚13.965902260482883% disk./.usageGB‚1244.4441413879395& disk.nvme0n1.out‚14804.53759765625 disk.nvme0c0n1.in‚ 38.2265625( disk.nvme0c0n1.out‚14804.53759765625 disk.sda1.out‚0 disk.rbd5.out‚ 1.83984375 disk.rbd0.out‚0 disk.rbd4.out‚ 1.83984375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1743.out‚ 4.28515625 disk.rbd2.in‚0 disk.sda3.in‚0.4375# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202) disk./.usagePercent‚83.51569048672725 disk.rbd0.in‚0 disk.dm-1.out‚ 196.01953125Vbœ‚(:‘ Æ•ÅÓò³âÒ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚136.382' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚75.76777777777778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•|‚Ÿ¤Ž‚(:™  Æ•ÅÓÚ¼§Ö disk.sda1.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29 network.recv‚514249 disk.sda2.in‚0 disk.rbd4.in‚0) proc.memory.percent‚17.19306707382202 disk.rbd2.in‚0 disk.sda3.in‚ 0.45703125 disk.dm-1.in‚ 0.45703125 disk.rbd3.in‚0( proc.memory.availableMB‚ 47537.609375 cpu‚13.966273066937742% disk.nvme0n1.out‚15061.3154296875% disk./.usageGB‚1245.0725784301758 disk.rbd2.out‚ 1.83984375 disk.rbd1.out‚ 1.83984375 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 1.83984375' disk.nvme0c0n1.out‚15061.3154296875" disk.dm-0.out‚15061.3154296875 disk.drbd1743.out‚ 4.29296875 disk.dm-0.in‚ 38.2265625 disk.sda.in‚ 0.45703125 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd5.out‚ 1.83984375 network.sent‚ 8.423035e+06 disk.nvme0c0n1.in‚ 38.2265625 disk.sda.out‚ 196.6640625 disk.dm-1.out‚ 196.6640625% memory_percent‚27.463364601135254) disk./.usagePercent‚83.55786542392553 disk.sda3.out‚ 196.6640625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 38.2265625²´®‰2- _wandbruntime‚3888  _runtime‚3888ã  Éà _step‚720 ' lr-AdamW/pg1‚0.00023321251550553488 ' lr-AdamW/pg2‚0.00023321251550553488 trainer/global_step‚9049 % _timestamp‚1.7858096104454029e+09 _runtime‚3888.738952425¦ïà]Î‘È trainer/global_step‚9049 % _timestamp‚1.7858096104454029e+09 _runtime‚3888.738952425 _step‚720 ' lr-AdamW/pg1‚0.00023321251550553488 ' lr-AdamW/pg2‚0.00023321251550553488ÐMœ]o2’- _wandbruntime‚3888  _runtime‚3888æÿaâE“ËO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155730231374848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7962984293376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60123727 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163908490331648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25924654 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163908332656640 _runtime‚3888.739061489 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61556901 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35632245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7943494574492 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35632247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8178234053120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163326732195840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7962984293376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155363730863104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151503121212832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚155363747902464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155730256278528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163326627726848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151503104173472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95565011 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95565011 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163326627726848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60123727 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25924424 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7962896863744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95574667 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚155363747902464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7962896863744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95574667 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60123727 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163908490331648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25924654 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163908332656640 _runtime‚3888.739061489 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61556901 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35632245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7943494574492 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35632247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8178234053120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163326732195840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7962984293376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚155363730863104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151503121212832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚155363747902464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155730256278528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163326627726848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151503104173472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95565011ÑfVWg2•- _wandbruntime‚3889  _runtime‚38892¼Ó®Ü&–ÖM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35636184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7944286244590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60130370 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163344674109440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163926451226112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151519844345792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7963863920128 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25927554 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155705904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7963776483328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155715576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159464130590382 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95585204 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159464233432978 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60130370 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95585204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8179149692928 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61563461 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151519861385152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155747438491648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163344778585600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60130372 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858096107771018e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95575534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61563738 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚163344674109440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163926451226112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151519844345792 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7963863920128 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25927554 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155705904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7963776483328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155715576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159464130590382 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95585204 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159464233432978 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60130370 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95585204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8179149692928 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61563461 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151519861385152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚155747438491648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚163344778585600 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60130372 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7858096107771018e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95575534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61563738 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 136958464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156222086052864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156222069013504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60455877 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35829097 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7987383038000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164229066481664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164228962012672 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96102669 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35829095 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164228962012672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7858096293420768e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156222086052864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156558548 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156222069013504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 _runtime‚3907.636948286 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8223436212224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156222086052864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156222069013504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60455877 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35829097 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7987383038000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164229066481664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164228962012672 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96102669 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35829095 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164228962012672 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7858096293420768e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156222086052864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156558548 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156222069013504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 153480704 _runtime‚3907.636948286 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8223436212224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8007860055552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26071109 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156239235776512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156575728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8007772618752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96113206 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7988260511334 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _step‚726 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7988174708098 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164832001664000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8224207332352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152356870001152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858096296731834e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60462520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164832165623296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160345130512486 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96103536 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60462522 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61904141 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156566056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156575728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8224346387968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156239252815872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156239235776512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _runtime‚3907.965120225 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60462522 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96103536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156239252815872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60462520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35833032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156607819235328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96113206 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0› 0Êá&¥ÛM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7988260511334 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 _step‚726 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7988174708098 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164832001664000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8224207332352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152356870001152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858096296731834e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60462520 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164832165623296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 139055616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160345130512486 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96103536 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60462522 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61904141 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156566056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156575728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8224346387968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156239252815872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156239235776512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _runtime‚3907.965120225 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60462522 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96103536 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156239252815872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60462520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35833032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156607819235328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96113206 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164247112871424 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8007860055552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26071109 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156239235776512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156575728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 163959296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8007772618752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96113206 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670Ö=¨þZ2¦- _wandbruntime‚3907  _runtime‚3907sÝnS §ƒ 0loss/FRt1/assignment_loss‚1.9462520922383955 /loss/FRt2/detection_loss‚2.5173714766278863 /loss/FBt1/detection_loss‚1.3815129024442285 0loss/FRt2/assignment_loss‚2.8069384656846523 2loss/SRqqt1/assignment_loss‚0.9636864397800405 3loss/FBt1/assignment_loss‚4.220516768249638e-05 2loss/FBt2/assignment_loss‚0.000146354171452653 trainer/global_step‚9099 % _timestamp‚1.7858096296743839e+09 2loss/SRqqt2/assignment_loss‚1.0606938172025944 .loss/FRt1/detection_loss‚2.046981516864992 0loss/SRqqt1/detection_loss‚0.964756889700718 &loss/total_loss‚16.971747029804725 _runtime‚3907.965921627 1loss/SRqqt2/detection_loss‚1.6088323873552408 /loss/FBt2/detection_loss‚1.6745324825668442 epoch‚0 _step‚727ÿéJC¨ˆ epoch‚0 _step‚727 0loss/FRt1/assignment_loss‚1.9462520922383955 /loss/FRt2/detection_loss‚2.5173714766278863 /loss/FBt1/detection_loss‚1.3815129024442285 0loss/FRt2/assignment_loss‚2.8069384656846523 2loss/SRqqt1/assignment_loss‚0.9636864397800405 3loss/FBt1/assignment_loss‚4.220516768249638e-05 2loss/FBt2/assignment_loss‚0.000146354171452653 trainer/global_step‚9099 % _timestamp‚1.7858096296743839e+09 2loss/SRqqt2/assignment_loss‚1.0606938172025944 .loss/FRt1/detection_loss‚2.046981516864992 0loss/SRqqt1/detection_loss‚0.964756889700718 &loss/total_loss‚16.971747029804725 _runtime‚3907.965921627 1loss/SRqqt2/detection_loss‚1.6088323873552408 /loss/FBt2/detection_loss‚1.6745324825668442×Y²˜Œœ©‚(:‘ ä•ÅÓïÍôÒ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚126.909' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚70.50500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©’ߘ¨ª‚(:  ä•ÅÓõŸøÖ disk.nvme0n1.in‚ 38.2265625 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 38.2265625( disk.nvme0c0n1.out‚15531.71826171875$ memory_percent‚27.46342420578003) proc.memory.availableMB‚ 47537.5703125# proc.memory.rssMB‚ 11267.6484375 disk.rbd4.in‚0 disk.rbd2.in‚0 disk.sda3.out‚ 198.13671875 disk.sda2.in‚0 proc.cpu.threads‚29 disk.drbd1743.in‚0& disk.nvme0n1.out‚15531.71826171875 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375# disk.dm-0.out‚15531.71826171875 disk.sda1.out‚0 disk.rbd5.in‚0 disk.dm-1.in‚ 0.45703125 disk.dm-1.out‚ 198.13671875 cpu‚13.967081195855796% disk./.usageGB‚1245.0702514648438 disk.dm-0.in‚ 38.2265625 disk.rbd5.out‚ 1.83984375 disk.sda.in‚ 0.45703125 disk.sda.out‚ 198.13671875 disk.sda3.in‚ 0.45703125 disk.rbd1.in‚0) proc.memory.percent‚17.19306707382202 network.sent‚ 8.470565e+06 disk.drbd1743.out‚4.328125 disk.rbd0.out‚0 disk.rbd4.out‚ 1.83984375 disk.rbd2.out‚ 1.83984375 network.recv‚517801) disk./.usagePercent‚83.55770925932966 disk.rbd3.out‚0@?@ò2«- _wandbruntime‚3927  _runtime‚3927:?R]ɬà _runtime‚3927.070355254 _step‚728 ' lr-AdamW/pg1‚0.00023578973415406548 ' lr-AdamW/pg2‚0.00023578973415406548 trainer/global_step‚9149 % _timestamp‚1.7858096487787094e+09|sÎ­È _step‚728 ' lr-AdamW/pg1‚0.00023578973415406548 ' lr-AdamW/pg2‚0.00023578973415406548 trainer/global_step‚9149 % _timestamp‚1.7858096487787094e+09 _runtime‚3927.070355254Øܦë-2®- _wandbruntime‚3927  _runtime‚3927mqŸíÑ'¯ËO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165131296298496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165131400767488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157418700 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8268635066880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153177138444832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60788029 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165719603025408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8050889134592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157080407163904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8050976564224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62236951 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9149 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157080424203264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8050976564224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26211004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96621015 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96621015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153177121405472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96630671 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157409042 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8268508587008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165131296298496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62236699 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161208409946340 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26210754 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157080424203264 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157080407163904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _step‚729 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8031185698384 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157080407163904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 _step‚729 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8031185698384 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165131296298496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165131400767488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157418700 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8268635066880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153177138444832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60788029 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165719603025408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8050889134592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157080407163904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8050976564224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62236951 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚9149 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157080424203264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8050976564224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26211004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96621015 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96621015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153177121405472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96630671 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 LDDeviceStatsMonitor.on_tra—ÑK‡‡in_batch_start/segment.large_pool.allocated‚130 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 151383552 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157409042 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8268508587008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165131296298496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62236699 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161208409946340 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26210754 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 126479872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚157080424203264 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3927.404554213 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60794670 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8051768754176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165149342681088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165737562013184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8051856190976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36029882 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153193861577792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157097590966272 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858096491125555e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8051768754176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96641208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26213840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96641208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157097573926912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157435880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60794672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚9149 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157097590966272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96631538 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157468150171648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8051856190976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62243488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157097573926912 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8269544605696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157426208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8269411841536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8032148974842 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26213606 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8032063171606 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96631538 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157435880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153193878617152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165149447157248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165737719681024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60794672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62243724 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887JÙ¢á&³ÛM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8032148974842 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26213606 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8032063171606 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96631538 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157435880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153193878617152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165149447157248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165737719681024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60794672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62243724 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161226027591994 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165149447157248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3927.404554213 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60794670 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8051768754176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165149342681088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165737562013184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8051856190976 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36029882 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153193861577792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157097590966272 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 % _timestamp‚1.7858096491125555e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8051768754176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96641208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26213840 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96641208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157097573926912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157435880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60794672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚9149 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157097590966272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96631538 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157468150171648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8051856190976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62243488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157097573926912 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8269544605696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157426208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8269411841536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596ÚØOrÑ2´- _wandbruntime‚3927  _runtime‚3927¹a ƒ µƒ /loss/FRt1/assignment_loss‚2.167804267125971 2loss/SRqqt1/assignment_loss‚0.8152216571581969 epoch‚0 trainer/global_step‚9149 _runtime‚3927.405509143 0loss/FRt2/assignment_loss‚2.7787064956294167 2loss/SRqqt2/assignment_loss‚1.0360178870777057 4loss/FBt2/assignment_loss‚1.4346058617222516e-06 1loss/SRqqt1/detection_loss‚0.8600061241951253 1loss/SRqqt2/detection_loss‚2.1467399799893068 $ _timestamp‚1.785809649113763e+09 /loss/FBt1/detection_loss‚2.8502225937942662 /loss/FBt2/detection_loss‚0.7786793361963897 %loss/total_loss‚18.12626186222333 _step‚731 4loss/FBt1/assignment_loss‚0.00027818104717762065 /loss/FRt1/detection_loss‚2.2910601391511807 -loss/FRt2/detection_loss‚2.40152376625273 »Ã?¶ˆ /loss/FRt1/assignment_loss‚2.167804267125971 2loss/SRqqt1/assignment_loss‚0.8152216571581969 epoch‚0 trainer/global_step‚9149 _runtime‚3927.405509143 0loss/FRt2/assignment_loss‚2.7787064956294167 2loss/SRqqt2/assignment_loss‚1.0360178870777057 4loss/FBt2/assignment_loss‚1.4346058617222516e-06 1loss/SRqqt1/detection_loss‚0.8600061241951253 1loss/SRqqt2/detection_loss‚2.1467399799893068 $ _timestamp‚1.785809649113763e+09 /loss/FBt1/detection_loss‚2.8502225937942662 /loss/FBt2/detection_loss‚0.7786793361963897 %loss/total_loss‚18.12626186222333 _step‚731 4loss/FBt1/assignment_loss‚0.00027818104717762065 /loss/FRt1/detection_loss‚2.2910601391511807 -loss/FRt2/detection_loss‚2.40152376625273Û÷œL¤œ·‚(:‘ ó•ÅÓééÕÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚150.257' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.47611111111112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¯–Ш¸‚(:  ó•ÅÓÚæ“Ö disk.rbd3.out‚0$ memory_percent‚27.46303677558899 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.in‚0 disk.dm-1.in‚ 0.45703125 disk.dm-0.in‚ 38.2265625 disk.rbd1.out‚ 1.92578125 disk.sda2.in‚0 network.sent‚ 8.515517e+06 network.recv‚519906 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd5.out‚ 1.83984375 disk.sda.in‚ 0.45703125* proc.memory.availableMB‚47537.82421875 disk.rbd2.in‚0 disk.rbd1.in‚0 cpu‚13.967519712282881 proc.cpu.threads‚29% disk./.usageGB‚1245.0743217468262% disk.nvme0n1.out‚15533.8720703125 disk.sda1.in‚0 disk.rbd2.out‚ 1.92578125 disk.sda3.out‚ 198.92578125 disk.drbd1743.out‚ 4.34765625 disk.sda2.out‚0 disk.rbd5.in‚0 disk.rbd4.out‚ 1.83984375 disk.sda.out‚ 198.92578125 disk.sda3.in‚ 0.45703125 disk.nvme0n1.in‚ 38.2265625 disk.drbd1743.in‚0" disk.dm-0.out‚15533.8720703125# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202) disk./.usagePercent‚83.55798241936868 disk.nvme0c0n1.in‚ 38.2265625' disk.nvme0c0n1.out‚15533.8720703125 disk.dm-1.out‚ 198.92578125‰~S›¹‚(: ‚–ÅÓËÏéÒ gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚155.401' gpu.0.enforcedPowerLimitWatts‚180.0' gpu.0.powerPercent‚86.3338888888889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±¾M¬º‚(:¡  ‚–ÅÓ•¹³Ö) proc.memory.percent‚17.19306707382202 cpu‚13.967916609676804 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0% memory_percent‚27.462655305862427 disk.nvme0n1.in‚ 38.40234375 disk.rbd4.in‚0 disk.sda3.in‚ 0.45703125 disk.dm-1.in‚ 0.45703125 disk.sda1.out‚0 disk.rbd5.out‚ 1.83984375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd4.out‚ 1.83984375! disk.nvme0c0n1.in‚ 38.40234375 disk.rbd1.in‚0 disk.drbd1743.in‚0 network.sent‚ 8.560446e+06) disk./.usagePercent‚83.55580430726737% disk./.usageGB‚1245.0418663024902 disk.dm-1.out‚ 199.83203125 disk.drbd1743.out‚ 4.35546875 disk.dm-0.in‚ 38.40234375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 0.45703125 disk.sda.out‚ 199.83203125 disk.rbd2.out‚ 1.98828125* proc.memory.availableMB‚47538.07421875% disk.nvme0n1.out‚15536.1083984375 disk.rbd2.in‚0 disk.rbd1.out‚ 1.92578125 proc.cpu.threads‚29 network.recv‚521966 disk.rbd5.in‚0 disk.sda3.out‚ 199.83203125" disk.dm-0.out‚15536.1083984375# proc.memory.rssMB‚ 11267.6484375' disk.nvme0c0n1.out‚15536.1083984375­JøØ2»- _wandbruntime‚3946  _runtime‚3946‡‘.XǼÁ & lr-AdamW/pg2‚0.0002370783434783308 trainer/global_step‚9199 % _timestamp‚1.7858096678900642e+09 _runtime‚3946.181721898 _step‚732 & lr-AdamW/pg1‚0.0002370783434783308¢-°Ì½Æ & lr-AdamW/pg1‚0.0002370783434783308 & lr-AdamW/pg2‚0.0002370783434783308 trainer/global_step‚9199 % _timestamp‚1.7858096678900642e+09 _runtime‚3946.181721898 _step‚732Ü1ÓS2¾- _wandbruntime‚3946  _runtime‚3946^Z¹sO ¿ÌO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚166033630584320 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61120179 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97158673 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97149017 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36222797 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158278852 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166625138220544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8075159965016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62576446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158311298894848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166033630584320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157938745314304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166624980545536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154014147060832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97158673 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61120177 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858096678910615e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97149017 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61120179 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8075074161892 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚157938762353664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26353651 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158311323798528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166033735053312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36222797 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158278852 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166625138220544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8075159965016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62576446 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158311298894848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166033630584320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157938745314304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166624980545536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154014147060832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97158673 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61120177 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 % _timestamp‚1.7858096678910615e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008Ý%fÁ2Á- _wandbruntime‚3946  _runtime‚3946µ²ÔFÜ&ÂÖM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚154030870193792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158286360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162106924671502 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158296032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157955929116672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61126820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158328506011648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚734 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8075951635114 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62583222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157955912077312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166051781443072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8095852326400 _runtime‚3946.516266435 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97169210 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62583445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166643245371392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61126822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26356711 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61126820 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166051781443072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61126822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8076037438350 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157955929116672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166643091897856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8095764889600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97159540 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8095852326400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158296032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97169210 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36226732 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158328530915328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8314714456064 trainer/global_step‚9199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8314585886208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154030887233152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158286360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166051676966912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97159540 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121äŠäØuÃÛM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158328530915328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8314714456064 trainer/global_step‚9199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8314585886208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154030887233152 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158286360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166051676966912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97159540 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚154030870193792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158286360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162106924671502 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158296032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157955929116672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61126820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158328506011648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚734 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8075951635114 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62583222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157955912077312 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166051781443072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8095852326400 _runtime‚3946.516266435 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 RÎâù!lEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97169210 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62583445 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166643245371392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61126822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26356711 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61126820 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166051781443072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61126822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8076037438350 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚157955929116672 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166643091897856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8095764889600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97159540 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8095852326400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158296032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97169210 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36226732 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Þ¹ÿ¨2Ä- _wandbruntime‚3946  _runtime‚3946$1Å 2loss/SRqqt1/assignment_loss‚0.9165292731976431 1loss/FBt1/assignment_loss‚0.07908608362457961 .loss/FRt2/detection_loss‚2.634723794537705 &loss/total_loss‚17.709496853383527 epoch‚0 trainer/global_step‚9199 _step‚735 0loss/FRt2/assignment_loss‚2.7622183855477864 3loss/FBt2/assignment_loss‚6.510626917588525e-07 .loss/FRt1/detection_loss‚2.253556418506538 2loss/SRqqt2/assignment_loss‚0.9374813724728301 1loss/SRqqt1/detection_loss‚0.9533855758460337 .loss/FBt1/detection_loss‚1.635730443522334 % _timestamp‚1.7858096682255926e+09 _runtime‚3946.517073237 0loss/FRt1/assignment_loss‚2.6464528658810784 1loss/SRqqt2/detection_loss‚1.7820311857759952 /loss/FBt2/detection_loss‚1.1083008034083133½iGà Ɔ 0loss/FRt1/assignment_loss‚2.6464528658810784 1loss/SRqqt2/detection_loss‚1.7820311857759952 /loss/FBt2/detection_loss‚1.1083008034083133 2loss/SRqqt1/assignment_loss‚0.9165292731976431 1loss/FBt1/assignment_loss‚0.07908608362457961 .loss/FRt2/detection_loss‚2.634723794537705 &loss/total_loss‚17.709496853383527 epoch‚0 trainer/global_step‚9199 _step‚735 0loss/FRt2/assignment_loss‚2.7622183855477864 3loss/FBt2/assignment_loss‚6.510626917588525e-07 .loss/FRt1/detection_loss‚2.253556418506538 2loss/SRqqt2/assignment_loss‚0.9374813724728301 1loss/SRqqt1/detection_loss‚0.9533855758460337 .loss/FBt1/detection_loss‚1.635730443522334 % _timestamp‚1.7858096682255926e+09 _runtime‚3946.517073237ß±«š‘Ç‚(:† ‘–ÅÓô…ÑÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚155.691' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚86.495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†|GtŸÈ‚(:”  ‘–ÅÓÜÀ¡Ö disk.rbd5.out‚ 1.83984375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 38.40234375 disk.rbd2.out‚ 1.98828125# proc.memory.rssMB‚ 11267.6484375) disk./.usagePercent‚83.55568961589202% disk./.usageGB‚1245.0401573181152 disk.drbd1743.out‚ 4.37890625 disk.sda1.in‚0) proc.memory.availableMB‚ 47537.8671875) proc.memory.percent‚17.19306707382202 cpu‚13.968375103904933 proc.cpu.threads‚29 network.sent‚ 8.562917e+06& disk.nvme0n1.out‚15548.20263671875 disk.sda1.out‚0 disk.rbd4.out‚ 1.83984375 disk.dm-0.in‚ 38.40234375 disk.sda3.in‚ 0.45703125 disk.sda3.out‚202.625 network.recv‚523056# disk.dm-0.out‚15548.20263671875 disk.rbd0.out‚0 disk.sda.out‚202.625% memory_percent‚27.462971210479736 disk.rbd1.in‚0( disk.nvme0c0n1.out‚15548.20263671875 disk.rbd2.in‚0 disk.dm-1.out‚202.625 disk.rbd5.in‚0 disk.sda.in‚ 0.45703125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 1.98828125 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.nvme0n1.in‚ 38.40234375 disk.rbd4.in‚0 disk.dm-1.in‚ 0.45703125 disk.sda2.out‚0´t¶€2É- _wandbruntime‚3965  _runtime‚39659çùþÉÊà _runtime‚3965.433257636 _step‚736 ' lr-AdamW/pg1‚0.00023836695280259607 ' lr-AdamW/pg2‚0.00023836695280259607 trainer/global_step‚9249 % _timestamp‚1.7858096871416173e+09óŵ‹ÎËÈ ' lr-AdamW/pg1‚0.00023836695280259607 ' lr-AdamW/pg2‚0.00023836695280259607 trainer/global_step‚9249 % _timestamp‚1.7858096871416173e+09 _runtime‚3965.433257636 _step‚736à(í_2Ì- _wandbruntime‚3965  _runtime‚3965®â¼ˆÒ'ÍÌO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158797100504064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26497321 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159171654734848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162970101262872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97677019 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8138968835072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158797100504064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158797083464704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97677019 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8118962625400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166935964870144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36419647 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392Fž—×'ÎÑO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158797100504064 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158797083464704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97677019 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8118962625400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166935964870144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36419647 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158797100504064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26497321 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159171654734848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162970101262872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97677019 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8138968835072 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62923547 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162987718908414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158814250227712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858096874761183e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97687542 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8119925901858 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167548645770240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159156184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8359918779904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 O@DeviceStª¬þ£õatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159156184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61458972 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 _runtime‚3965.768044532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166954115728896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167548805535232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159146512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97697212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159188861851648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36423584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62923810 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8139848461824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36423582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97687542 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162987821751010 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61458970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8139761025024åìdá&ÑÛM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26500226 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26499965 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62923547 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162987718908414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158814250227712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 % _timestamp‚1.7858096874761183e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97687542 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8119925901858 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167548645770240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159156184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8359918779904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159156184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61458972 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 _runtime‚3965.768044532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166954115728896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167548805535232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159146512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97697212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159188861851648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36423584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62923810 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8139848461824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 134861312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36423582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97687542 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162987821751010 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61458970 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8139761025024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159188886755328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 _step‚738 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61458972 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61458970⣲wó2Ò- _wandbruntime‚3965  _runtime‚3965DÊ·; Ó† 1loss/SRqqt2/detection_loss‚1.8552489490636315 trainer/global_step‚9249 % _timestamp‚1.7858096874772987e+09 _runtime‚3965.769063723 1loss/SRqqt1/assignment_loss‚0.975293274681161 /loss/FRt2/detection_loss‚2.6561719289532415 /loss/FBt1/detection_loss‚1.4080127296814073 /loss/FBt2/detection_loss‚0.7745640879102911 &loss/total_loss‚16.489588775004485 _step‚739 2loss/SRqqt2/assignment_loss‚0.6840965277127301 /loss/FRt1/detection_loss‚2.3982432467095993 1loss/SRqqt1/detection_loss‚0.8240712472492722 0loss/FRt1/assignment_loss‚2.1446627974510193 0loss/FRt2/assignment_loss‚2.7692050359867237 3loss/FBt2/assignment_loss‚2.292483842416879e-07 epoch‚0 4loss/FBt1/assignment_loss‚1.8720357021982813e-05µ„ÂÔ‹ 1loss/SRqqt1/assignment_loss‚0.975293274681161 /loss/FRt2/detection_loss‚2.6561719289532415 /loss/FBt1/detection_loss‚1.4080127296814073 /loss/FBt2/detection_loss‚0.7745640879102911 &loss/total_loss‚16.489588775004485 _step‚739 2loss/SRqqt2/assignment_loss‚0.6840965277127301 /loss/FRt1/detection_loss‚2.3982432467095993 1loss/SRqqt1/detection_loss‚0.8240712472492722 0loss/FRt1/assignment_loss‚2.1446627974510193 0loss/FRt2/assignment_loss‚2.7692050359867237 3loss/FBt2/assignment_loss‚2.292483842416879e-07 epoch‚0 4loss/FBt1/assignment_loss‚1.8720357021982813e-05 1loss/SRqqt2/detection_loss‚1.8552489490636315 trainer/global_step‚9249 % _timestamp‚1.7858096874772987e+09 _runtime‚3965.769063723ã½1®›œÕ‚(:‘  –ÅÓ´€ÊÒ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚83.70055555555555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û²÷½ Ö‚(:•   –ÅӫÀÕ& disk.nvme0n1.out‚15548.39013671875 disk.rbd4.in‚0 disk.rbd2.out‚ 2.0205078125 disk.dm-0.in‚ 38.40234375 disk.rbd3.in‚0 disk.rbd5.out‚ 1.92578125 disk.rbd2.in‚0 disk.rbd1.out‚ 1.98828125 disk.drbd1743.in‚0 disk.drbd1743.out‚ 4.38671875 proc.cpu.threads‚29 network.sent‚ 8.60795e+06) disk./.usagePercent‚83.55569345600505 disk.nvme0n1.in‚ 38.40234375 disk.sda1.in‚0 disk.rbd5.in‚0 disk.rbd4.out‚ 1.83984375( disk.nvme0c0n1.out‚15548.39013671875 disk.sda3.in‚ 0.45703125 network.recv‚525252! disk.nvme0c0n1.in‚ 38.40234375 disk.sda.out‚203.375 disk.dm-1.out‚203.375 disk.sda2.in‚0* proc.memory.availableMB‚47538.07421875 cpu‚13.96877663647421 disk.sda1.out‚0 disk.dm-1.in‚ 0.45703125 disk.rbd1.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚15548.39013671875# proc.memory.rssMB‚ 11267.6484375% disk./.usageGB‚1245.0402145385742 disk.sda.in‚ 0.45703125 disk.sda3.out‚203.375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd3.out‚0% memory_percent‚27.462655305862427) proc.memory.percent‚17.19306707382202Ñðs2×- _wandbruntime‚3984  _runtime‚3984Þ_@ÇØÁ _runtime‚3984.727958504 _step‚740 & lr-AdamW/pg1‚0.0002396555621268614 & lr-AdamW/pg2‚0.0002396555621268614 trainer/global_step‚9299 % _timestamp‚1.7858097064361992e+09Ta•ÌÙÆ & lr-AdamW/pg1‚0.0002396555621268614 & lr-AdamW/pg2‚0.0002396555621268614 trainer/global_step‚9299 % _timestamp‚1.7858097064361992e+09 _runtime‚3984.727958504 _step‚740äSõ¼ 2Ú- _wandbruntime‚3984  _runtime‚3984—°b´Ñ'ÛËO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160032035478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163851101184864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63257441 trainer/global_step‚9299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98205021 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61784479 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26640696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8404063517184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159989498 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167838403624960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159655421615104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98205021 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26640944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8162936892032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36616495 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167838403624960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61784479 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98214677 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159999156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168436231767040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159655438654464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155688164292832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 $ _timestamp‚1.785809706437257e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159655438654464 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98214677 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚160032010574848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8182877540864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168436074092032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167838299155968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159655421615104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155688147253472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚741 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8404196288512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8162851088908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _runtime‚3984.733035751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61784477 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8182964970496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167838299155968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8182964970496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8182877540864 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36616497 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61784477 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63257191 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163850998342380 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167838299155968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159655421615104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155688147253472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 _step‚741 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8404196288512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8162851088908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 _runtime‚3984.733035751 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61784477 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8182964970496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167838299155968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8182964970496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8182877540864 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36616497 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61784477 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63257191 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163850998342380 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160032035478528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163851101184864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63257441 trainer/global_step‚9299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98205021 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61784479 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26640696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8404063517184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159989498 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167838403624960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159655421615104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98205021 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2åx?¶2Ý- _wandbruntime‚3985  _runtime‚3985Á”[Ü&ÞÖM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160006664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163868615987922 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163868718830518 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167856450014720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168454340621824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159672605417472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26643493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36620432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚9299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61791122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858097067721837e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61791122 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61791120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159672588378112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155704887425792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8183844597248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160016336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8183844597248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61791120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98225214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160049242595328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155704904465152 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63263925 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160049217691648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160016336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98215544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26643741 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98215544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168454185051136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159672588378112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63264175nŽ„.á&ßÛM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160049242595328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 130667008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155704904465152 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63263925 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160049217691648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160016336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98215544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26643741 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98215544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168454185051136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159672588378112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63264175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160006664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163868615987922 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163868718830518 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167856450014720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168454340621824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159672605417472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26643493 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36620432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 trainer/global_step‚9299 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61791122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858097067721837e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61791122 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61791120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159672588378112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155704887425792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8183844597248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160016336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8183844597248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61791120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98225214 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168740737910784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169341609089024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168740633441792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156525172908832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160513759765504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168740633441792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160513759765504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160513776804864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8226873676288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156525155869472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98742679 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98733023 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63595496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98733023 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8449375445504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8226961105920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26782151 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8226873676288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62116629 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160513776804864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168740737910784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36813347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36813345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8449242674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858097255717785e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160892391318528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160859308 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62116629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63595741 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8226961105920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚745 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98742679 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4003.866949979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992ï÷Ĉ×'êÑO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164731998264372 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚160892366414848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8206825355540 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160849650 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169341766764032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168740737910784 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169341609089024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168740633441792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156525172908832 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160513759765504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168740633441792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160513759765504 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160513776804864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8226873676288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156525155869472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98742679 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98733023 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63595496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98733023 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8449375445504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8226961105920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26782151 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8226873676288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62116629 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚160513776804864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168740737910784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36813347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36813345 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8449242674176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 % _timestamp‚1.7858097255717785e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚160892391318528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160859308 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62116629 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63595741 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8226961105920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 _step‚745 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98742679 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4003.866949979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658é âªW2ë- _wandbruntime‚4004  _runtime‚4004Ѹ“Û&ìÕM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 _step‚746 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36817284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8227840732672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168758784300544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160530943567872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98743546 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62123270 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8227753295872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164749513067430 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 $ _timestamp‚1.785809725902428e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160909573531648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160530926528512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98753216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36817282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26785180 trainer/global_step‚9349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98743546 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8227840732672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63602464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8450279944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8227753295872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160866816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156541913081152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160866816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164749615910026 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8207617025638 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63602208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160876488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168758784300544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160876488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62123272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160530926528512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160909598435328 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160876488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62123272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160530926528512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160909598435328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36817284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 161862144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8227840732672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168758784300544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160530943567872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98743546 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62123270 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8227753295872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164749513067430 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 $ _timestamp‚1.785809725902428e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160909573531648 UHDeviceStatsMonitor.on_train_batch_end/&h=Ürequested_bytes.small_pool.current‚85803236 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160530926528512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98753216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36817282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26785180 trainer/global_step‚9349 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98743546 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8227840732672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63602464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8450279944192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8227753295872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160866816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚156541913081152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160866816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164749615910026 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8207617025638 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63602208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130ê<{ˆ2î- _wandbruntime‚4004  _runtime‚4004º¤ ï„ % _timestamp‚1.7858097259036596e+09 _runtime‚4004.195132517 /loss/FRt1/detection_loss‚2.4283642582111544 1loss/SRqqt2/detection_loss‚1.6564039243757724 /loss/FBt1/detection_loss‚2.0998983454580107 &loss/total_loss‚18.505347637610477 2loss/FBt1/assignment_loss‚0.000670887868904894 1loss/SRqqt1/detection_loss‚1.0820583307158937 trainer/global_step‚9349 0loss/FRt1/assignment_loss‚2.9053687700977573 2loss/SRqqt1/assignment_loss‚1.0483623104919142 epoch‚0 _step‚747 /loss/FRt2/assignment_loss‚2.780385814961933 2loss/SRqqt2/assignment_loss‚0.9729215592797845 3loss/FBt2/assignment_loss‚0.0001267017543201431 /loss/FRt2/detection_loss‚2.5343252906487104 /loss/FBt2/detection_loss‚0.9964614437463192ë\N?ð‰ /loss/FRt1/detection_loss‚2.4283642582111544 1loss/SRqqt2/detection_loss‚1.6564039243757724 /loss/FBt1/detection_loss‚2.0998983454580107 &loss/total_loss‚18.505347637610477 2loss/FBt1/assignment_loss‚0.000670887868904894 1loss/SRqqt1/detection_loss‚1.0820583307158937 trainer/global_step‚9349 0loss/FRt1/assignment_loss‚2.9053687700977573 2loss/SRqqt1/assignment_loss‚1.0483623104919142 epoch‚0 _step‚747 /loss/FRt2/assignment_loss‚2.780385814961933 2loss/SRqqt2/assignment_loss‚0.9729215592797845 3loss/FBt2/assignment_loss‚0.0001267017543201431 /loss/FRt2/detection_loss‚2.5343252906487104 /loss/FBt2/detection_loss‚0.9964614437463192 % _timestamp‚1.7858097259036596e+09 _runtime‚4004.195132517ë!a3uœñ‚(:‘ ¾–ÅÓ¡¶þÒ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚144.211' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚80.11722222222222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p…) §ò‚(:œ  ¾–ÅÓתÕ disk.sda1.in‚0 disk.rbd4.out‚ 1.98828125 disk.dm-1.out‚ 205.0703125& disk.nvme0c0n1.out‚15551.697265625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.45703125 disk.rbd2.out‚ 2.08349609375 disk.rbd1.in‚0! disk.dm-0.out‚15551.697265625) proc.memory.availableMB‚ 47537.5078125! disk.nvme0c0n1.in‚ 38.40234375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 38.40234375$ disk.nvme0n1.out‚15551.697265625# proc.memory.rssMB‚ 11267.6484375) proc.memory.percent‚17.19306707382202 proc.cpu.threads‚29 disk.drbd1743.out‚4.421875 network.sent‚ 8.697779e+06% disk./.usageGB‚1244.6208381652832 disk.rbd4.in‚0 disk.sda2.in‚0$ memory_percent‚27.46351957321167 disk.rbd5.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 205.0703125 disk.dm-1.in‚ 0.45703125 disk.rbd1.out‚ 2.0205078125 disk.rbd3.out‚0 cpu‚13.969764114883764) disk./.usagePercent‚83.52754875571327 disk.rbd5.out‚ 1.98828125 disk.sda.out‚ 205.0703125 disk.sda2.out‚0 disk.drbd1743.in‚0 disk.dm-0.in‚ 38.40234375 network.recv‚529359 disk.rbd2.in‚0 disk.sda3.in‚ 0.45703125ÃAú­‘ó‚(:† Í–ÅÓȇÞÒ gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚40.0 gpu.0.powerWatts‚155.691' gpu.0.enforcedPowerLimitWatts‚180.0 gpu.0.powerPercent‚86.495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0( E+©ô‚(:ž  Í–ÅÓü¯ÄÕ disk.sda.in‚ 0.45703125$ disk./.usageGB‚1243.914882659912 disk.drbd1743.out‚4.421875 disk.rbd0.in‚0 disk.rbd2.out‚ 2.08349609375 disk.sda3.in‚ 0.45703125 disk.rbd1.out‚ 2.08349609375 network.sent‚ 8.742531e+06 disk.sda2.out‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚15552.1025390625 disk.sda3.out‚ 205.953125 disk.dm-1.out‚ 205.953125% memory_percent‚27.463573217391968* proc.memory.availableMB‚47537.47265625 disk.dm-0.in‚ 38.40234375" disk.dm-0.out‚15552.1025390625 disk.nvme0n1.in‚ 38.40234375 disk.rbd5.in‚0 disk.rbd5.out‚ 1.98828125 disk.rbd2.in‚0) proc.memory.percent‚17.19306707382202 disk.sda2.in‚0 disk.drbd1743.in‚0 disk.rbd4.out‚ 1.98828125 disk.sda.out‚ 205.953125 proc.cpu.threads‚29 disk.sda1.out‚0 disk.rbd4.in‚0 disk.rbd1.in‚0 network.recv‚531217 disk.rbd0.out‚0 disk.dm-1.in‚ 0.45703125# proc.memory.rssMB‚ 11267.6484375 cpu‚13.970024593417113) disk./.usagePercent‚83.48017148941169 disk.rbd3.in‚0% disk.nvme0n1.out‚15552.1025390625 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 38.40234375äíR2õ- _wandbruntime‚4023  _runtime‚4023ÁC"DÈö ' lr-AdamW/pg1‚0.00024223278077539203 ' lr-AdamW/pg2‚0.00024223278077539203 trainer/global_step‚9399 $ _timestamp‚1.785809745218252e+09 _runtime‚4023.511990034 _step‚748Å ¹ŽÍ÷Ç ' lr-AdamW/pg1‚0.00024223278077539203 ' lr-AdamW/pg2‚0.00024223278077539203 trainer/global_step‚9399 $ _timestamp‚1.785809745218252e+09 _runtime‚4023.511990034 _step‚748ìf^,2ø- _wandbruntime‚4023  _runtime‚4023ÑvÒ'ùÌO _runtime‚4023.512088799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚749 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157362164485472 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37010197 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161372097915904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62448779 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8494443856384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8270957241344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161709802 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62448779 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 % _timestamp‚1.7858097452189808e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99270681 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161372097915904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99270681 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62448777 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8494576627712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161372114955264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161372114955264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165612895343880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63934509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63934250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170247166111232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157362181524832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8270869811712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165612792501396 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169642967727616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169642967727616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161752747158528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8250628015924 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37010195 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8270957241344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169642967727616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚169642967727616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161752747158528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8250628015924 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37010195 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8270957241344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8270957241344 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161709802 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62448779 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 % _timestamp‚1.7858097452189808e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99270681 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161372097915904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99270681 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62448777 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8494576627712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161372114955264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161372114955264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165612895343880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63934509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63934250 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170247166111232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157362181524832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8270869811712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165612792501396 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161389281718272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8495352327168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚9399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161726968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99281218 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165630512989534 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8251591292382 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62455422 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26926892 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26927136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99271548 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161726968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161389264678912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99271548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165630410146938 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157378904657792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169661118586368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62455422 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858097455581815e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚750 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37014132 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8271749431296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161389281718272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170265281698816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8271836868096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161769954275328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4023.850266696 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99281218 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63941024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161389264678912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161736640 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8495478799872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161389264678912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99271548 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165630410146938 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 151376384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157378904657792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169661118586368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62455422 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 % _timestamp‚1.7858097455581815e+09 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚750 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37014132 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8271749431296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161389281718272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170265281698816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8271836868096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161769954275328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4023.850266696 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99281218 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63941024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161389264678912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161736640 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8495478799872 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161389281718272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8495352327168 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 trainer/global_step‚9399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161726968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99281218 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165630512989534 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8251591292382 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62455422 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26926892 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26927136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99271548 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161726968 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032îŽ÷l¹2þ- _wandbruntime‚4023  _runtime‚4023~gpáÿ‚ trainer/global_step‚9399 % _timestamp‚1.7858097455593626e+09 1loss/SRqqt2/detection_loss‚2.4106009155511856 /loss/FBt2/detection_loss‚0.7040597418602556 epoch‚0 1loss/SRqqt1/detection_loss‚1.0264192449347096 /loss/FBt1/detection_loss‚2.7087294552475214 %loss/total_loss‚19.81079160127966 .loss/FRt2/assignment_loss‚3.23536922714927 2loss/SRqqt2/assignment_loss‚0.9384988344584902 3loss/FBt1/assignment_loss‚9.327953193860594e-06 .loss/FRt1/detection_loss‚2.685416510031707 _runtime‚4023.851038452 _step‚751 /loss/FRt1/assignment_loss‚2.541230313155962 2loss/SRqqt1/assignment_loss‚0.7869696741049387 4loss/FBt2/assignment_loss‚3.3140003105813776e-06 /loss/FRt2/detection_loss‚2.7734850428321143GA•7 €‡ .loss/FRt2/assignment_loss‚3.23536922714927 2loss/SRqqt2/assignment_loss‚0.9384988344584902 3loss/FBt1/assignment_loss‚9.327953193860594e-06 .loss/FRt1/detection_loss‚2.685416510031707 _runtime‚4023.851038452 _step‚751 /loss/FRt1/assignment_loss‚2.541230313155962 2loss/SRqqt1/assignment_loss‚0.7869696741049387 4loss/FBt2/assignment_loss‚3.3140003105813776e-06 /loss/FRt2/detection_loss‚2.7734850428321143 trainer/global_step‚9399 % _timestamp‚1.7858097455593626e+09 1loss/SRqqt2/detection_loss‚2.4106009155511856 /loss/FBt2/detection_loss‚0.7040597418602556 epoch‚0 1loss/SRqqt1/detection_loss‚1.0264192449347096 /loss/FBt1/detection_loss‚2.7087294552475214 %loss/total_loss‚19.81079160127966ï#̸ ›‚(: Ü–ÅÓ»¬çÓ gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚133.13' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚73.96111111111111 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á""¯‚‚(:¤  Ü–ÅÓ±Œ¯× disk.rbd4.in‚0 disk.sda.out‚ 206.55078125 disk.sda2.out‚0 disk.nvme0n1.in‚ 38.40234375 disk.rbd0.in‚0% disk.nvme0n1.out‚15552.6923828125 disk.rbd5.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 47537.40625 disk.rbd4.out‚ 2.0205078125' disk.nvme0c0n1.out‚15552.6923828125 disk.rbd2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.08349609375 disk.sda2.in‚0 disk.drbd1743.out‚4.453125 disk.dm-0.in‚ 38.40234375) disk./.usagePercent‚83.46588550097847 disk.sda3.in‚ 0.45703125 disk.dm-1.out‚ 206.55078125 disk.rbd3.out‚0 disk.rbd5.out‚ 2.0205078125# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29% disk./.usageGB‚1243.7020111083984! disk.nvme0c0n1.in‚ 38.40234375 disk.rbd2.out‚ 2.146484375 disk.sda3.out‚ 206.55078125 disk.rbd3.in‚0 disk.sda1.out‚0% memory_percent‚27.463674545288086 disk.sda.in‚ 0.45703125 disk.sda1.in‚0 disk.drbd1743.in‚0" disk.dm-0.out‚15552.6923828125) proc.memory.percent‚17.19306707382202 cpu‚13.970271511525413 disk.dm-1.in‚ 0.45703125 network.sent‚ 8.744988e+06 network.recv‚532021^qœ2ƒ- _wandbruntime‚4043  _runtime‚4043XQŽÈ„ _step‚752 ' lr-AdamW/pg1‚0.00024352139009965732 ' lr-AdamW/pg2‚0.00024352139009965732 trainer/global_step‚9449 $ _timestamp‚1.785809764772514e+09 _runtime‚4043.064166469.µ8%Í…Ç _runtime‚4043.064166469 _step‚752 ' lr-AdamW/pg1‚0.00024352139009965732 ' lr-AdamW/pg2‚0.00024352139009965732 trainer/global_step‚9449 $ _timestamp‚1.785809764772514e+09ð”ÇÚC2†- _wandbruntime‚4043  _runtime‚4043°QÀý}'‡ÌO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162230453105664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166493689580904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 _step‚753 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170545302013440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166493792423388 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162569954 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62780929 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8314953376768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162230436066304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27067766 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99798683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62780927 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162230453105664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27067509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8294602282556 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8314865947136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99798683 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170545302013440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162230436066304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162579612 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62780927 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37207045 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170545406482432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64274813 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64274554 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8539592384512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171152670479360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158199190140832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8539725155840 % _timestamp‚1.7858097647734334e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚4043.068929457 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171152828154368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62780929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8294516479432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8314953376768 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8314865947136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99789027 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool—# U.current‚9656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0G­f×'ˆÑO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170545302013440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162230436066304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162579612 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62780927 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37207045 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170545406482432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64274813 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64274554 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8539592384512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171152670479360 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158199190140832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8539725155840 % _timestamp‚1.7858097647734334e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 157675008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 _runtime‚4043.068929457 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 132771328 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171152828154368 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62780929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8294516479432 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8314953376768 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8314865947136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99789027 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162230453105664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166493689580904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 _step‚753 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170545302013440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166493792423388 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162569954 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62780929 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8314953376768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162230436066304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27067766 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99798683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104468992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62780927 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162230453105664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27067509 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8294602282556 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8314865947136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99798683 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158215930313152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27070340 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64281582 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166511410069042 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166511307226446 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62787572 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚9449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162247602829312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99809220 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162630310115328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99809220 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99799550 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62787570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62787570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64281322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858097651084263e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8315833003520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8540492458496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158215913273792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62787572 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8540625222656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162587120 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162247602829312 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8295393952654 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37210982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171170777670144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162247619868672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27070598 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162596792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162596792 _runtime‚4043.400474199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8315745566720 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 è\Òá&‹ÛM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64281322 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 % _timestamp‚1.7858097651084263e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8315833003520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8540492458496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158215913273792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62787572 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8540625222656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162587120 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162247602829312 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8295393952654 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37210982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171170777670144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162247619868672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27070598 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162596792 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162596792 _runtime‚4043.400474199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8315745566720 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158215930313152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27070340 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64281582 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166511410069042 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166511307226446 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62787572 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 trainer/global_step‚9449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162247602829312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99809220 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162630310115328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99809220 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99799550 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62787570 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62787570 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640òíòü'2Œ- _wandbruntime‚4043  _runtime‚4043#¹.þ trainer/global_step‚9449 _step‚755 2loss/SRqqt2/assignment_loss‚0.9645628741476685 3loss/FBt1/assignment_loss‚0.0015850178399550903 .loss/FRt1/detection_loss‚2.539478693258118 0loss/SRqqt2/detection_loss‚1.791899378504604 %loss/total_loss‚16.24934312940376 _runtime‚ 4043.40122719 1loss/FBt2/assignment_loss‚0.01990277080484404 /loss/FRt2/detection_loss‚2.4777256562429315 /loss/FBt2/detection_loss‚0.5433342349441612 epoch‚0 % _timestamp‚1.7858097651097665e+09 /loss/FRt1/assignment_loss‚2.208890667638263 0loss/FRt2/assignment_loss‚2.6636291272499983 2loss/SRqqt1/assignment_loss‚0.8727440521882576 0loss/SRqqt1/detection_loss‚1.038204253418371 /loss/FBt1/detection_loss‚1.1273864031665854ŽŽ…õ Žƒ 0loss/FRt2/assignment_loss‚2.6636291272499983 2loss/SRqqt1/assignment_loss‚0.8727440521882576 0loss/SRqqt1/detection_loss‚1.038204253418371 /loss/FBt1/detection_loss‚1.1273864031665854 trainer/global_step‚9449 _step‚755 2loss/SRqqt2/assignment_loss‚0.9645628741476685 3loss/FBt1/assignment_loss‚0.0015850178399550903 .loss/FRt1/detection_loss‚2.539478693258118 0loss/SRqqt2/detection_loss‚1.791899378504604 %loss/total_loss‚16.24934312940376 _runtime‚ 4043.40122719 1loss/FBt2/assignment_loss‚0.01990277080484404 /loss/FRt2/detection_loss‚2.4777256562429315 /loss/FBt2/detection_loss‚0.5433342349441612 epoch‚0 % _timestamp‚1.7858097651097665e+09 /loss/FRt1/assignment_loss‚2.208890667638263óÌœ‚(:‘ ë–ÅÓ¨ÚæÒ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚149.119' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚82.84388888888888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‡gHÄ‚(:Ÿ  ë–ÅÓÉáŽÕ disk.rbd2.in‚0* proc.memory.availableMB‚47537.16015625 cpu‚13.970515422983041 network.sent‚ 8.789944e+06 disk.drbd1743.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚29 network.recv‚534389# disk.nvme0n1.out‚15553.26171875 disk.rbd5.in‚0 disk.sda.in‚ 0.45703125 disk.dm-1.out‚ 207.57421875# proc.memory.rssMB‚ 11267.6484375 disk.rbd0.out‚0 disk.rbd4.out‚ 2.0205078125)úµ:žæ proc.memory.percent‚17.19306707382202 disk.rbd5.out‚ 2.0205078125 disk.dm-0.out‚15553.26171875 disk.rbd3.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 38.40234375 disk.sda.out‚ 207.57421875 disk.dm-1.in‚ 0.45703125 disk.sda2.out‚0 disk.nvme0n1.in‚ 38.40234375 disk.rbd0.in‚0 disk.rbd2.out‚ 2.146484375 disk.sda3.out‚ 207.57421875 disk.rbd1.in‚0$ disk./.usageGB‚1243.701675415039 disk.dm-0.in‚ 38.40234375 disk.sda1.in‚0 disk.rbd4.in‚0 disk.sda3.in‚ 0.45703125 disk.rbd1.out‚ 2.146484375 disk.sda2.in‚0$ memory_percent‚27.46405005455017) disk./.usagePercent‚83.46586297231546 disk.drbd1743.out‚ 4.47265625% disk.nvme0c0n1.out‚15553.26171875=²2‘- _wandbruntime‚4062  _runtime‚4062ËÚ9Æ’À & lr-AdamW/pg1‚0.0002448099994239226 & lr-AdamW/pg2‚0.0002448099994239226 trainer/global_step‚9499 $ _timestamp‚1.785809784317959e+09 _runtime‚4062.609641863 _step‚756<دHË“Å & lr-AdamW/pg1‚0.0002448099994239226 & lr-AdamW/pg2‚0.0002448099994239226 trainer/global_step‚9499 $ _timestamp‚1.785809784317959e+09 _runtime‚4062.609641863 _step‚756ô¡+6Ú2”- _wandbruntime‚4062  _runtime‚4062Ç ë­Ô'•ÎO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37403897 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37403895 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100317029 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167374689502896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚163088791256064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚163088774216704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8358862082560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4062.613939439 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171447636299264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚163088791256064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27209138 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63113077 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171447740768256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8584916129280 # _timestamp‚1.78580978431893e+09 _step‚757 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100326685 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64613304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163473433934848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163439764 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63113079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171447740768256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8358949512192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 trainer/global_step‚9499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163430106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159036198756832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63113079 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171447636299264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100317029 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚163088774216704 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172058374967808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27209407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163473458838528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159036181717472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887—‹…ýÙ'–ÓO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100317029 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 2768240640 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167374689502896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚163088791256064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚163088774216704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8358862082560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚4062.613939439 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171447636299264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚163088791256064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27209138 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63113077 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171447740768256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8584916129280 # _timestamp‚1.78580978431893e+09 _step‚757 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100326685 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64613304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163473433934848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163439764 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63113079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171447740768256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8358949512192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 trainer/global_step‚9499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163430106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159036198756832 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63113079 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 139062784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171447636299264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100317029 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚163088774216704 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172058374967808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27209407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163473458838528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159036181717472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37403897 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37403895 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024õÕ\¯2—- _wandbruntime‚4062  _runtime‚4062½ g(à&˜ÚM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64620075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27212241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163447272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171465682681856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63119722 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100327552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163105958019072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172076480685056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163105940979712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172076323017216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8585814729728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8585681965568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63119722 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163105958019072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63119720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163490665955328 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167392204305954 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163456944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100337222 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171465787158016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63119720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 trainer/global_step‚9499 _runtime‚4062.944600485 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171465787158016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8339368219398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64619830 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163490641051648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159052921889792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100337222 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8359741702144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27211998 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8359829138944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7858097846526647e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8359829138944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163456944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167392307148550 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8339282416162 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚758 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163447272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171465682681856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63119722 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100327552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 145651200 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163105958019072 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172076480685056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163105940979712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172076323017216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8585814729728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8585681965568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63119722 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163105958019072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63119720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163490665955328 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167392204305954 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163456944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100337222 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171465787158016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63119720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 trainer/global_step‚9499 _runtime‚4062.944600485 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171465787158016 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8339368219398 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64619830 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163490641051648 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159052921889792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100337222 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8359741702144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27211998 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8359829138944 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 % _timestamp‚1.7858097846526647e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8359829138944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163456944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167392307148550 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8339282416162 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚130 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚758 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100845031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚163947129406464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164333789774848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172350075054080 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27353224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172963740475904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159873190333472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚163947112367104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164290258 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚163947129406464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63445227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37600747 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4081.516915411 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168255586582404 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172349970585088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8402858217984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8402945647616 trainer/global_step‚9549žKlk4 ¦ÓO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚538 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 2871874888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164333789774848 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚251 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172350075054080 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 2940455936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 2940455936 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27353224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172963740475904 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159873190333472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚163947112367104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164290258 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 250523136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚163947129406464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63445227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37600747 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 2768240640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4081.516915411 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168255586582404 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172349970585088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8402858217984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11816 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8402945647616 trainer/global_step‚9549 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8382379209572 # _timestamp‚1.78580980322181e+09 _step‚761 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104468992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100845031 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172963900248064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 253755392 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63445229 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100854687 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚121 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87429632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚18 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚130 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11887 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚251 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100845031 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 134868480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚251 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚163947129406464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658ù]ò™2§- _wandbruntime‚4081  _runtime‚4081Eþèß&¨ÙM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164307424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172982007636480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163964279130112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172368121443840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163964296169472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168273101385462 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63451872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37604684 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37604682 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27356067 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8630985841152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172368016967680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8630857271296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8403737837568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8403737837568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚163964279130112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100855554 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172981854162944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164351021795328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159889947545152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100855554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27356302 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172368121443840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64960986 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164350996891648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164317096 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163964296169472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63451872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚762 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63451870 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64960749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8383170879670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 $ _timestamp‚1.785809803551224e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8403825274368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164317096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100865224 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168273204228058 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63451870 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100865224&Ï©"ä&©ÞM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚130 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172368016967680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1291 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8630857271296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8403737837568 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8403737837568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 189288448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚163964279130112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100855554 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172981854162944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164351021795328 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159889947545152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100855554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27356302 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172368121443840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64960986 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164350996891648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164317096 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163964296169472 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63451872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚762 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 250523136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63451870 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64960749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 153473536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8383170879670 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚121 $ _timestamp‚1.785809803551224e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8403825274368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164317096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100865224 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168273204228058 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚130 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63451870 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100865224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164307424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172982007636480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163964279130112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172368121443840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163964296169472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168273101385462 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63451872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37604684 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37604682 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27356067 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8630985841152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 128569856 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856úß÷¾ç2ª- _wandbruntime‚4081  _runtime‚4081ɇ¹«‚ /loss/FBt2/detection_loss‚0.7008477463994337 2loss/SRqqt1/assignment_loss‚0.9419362965316594 2loss/SRqqt2/assignment_loss‚0.8725022500125217 4loss/FBt1/assignment_loss‚4.9868536192339734e-06 .loss/FRt2/detection_loss‚2.415760807300869 .loss/FBt1/detection_loss‚4.564322465409835 % _timestamp‚1.7858098035523515e+09 _step‚763 &loss/total_loss‚19.727582327291667 /loss/FRt1/assignment_loss‚2.037292939513477 /loss/FRt2/assignment_loss‚2.859175121156793 .loss/FRt1/detection_loss‚2.788688846488497 trainer/global_step‚9549 3loss/FBt2/assignment_loss‚9.909057580036125e-06 1loss/SRqqt1/detection_loss‚1.0688120089726496 1loss/SRqqt2/detection_loss‚1.4782289495947314 epoch‚0 _runtime‚4081.843856781[¨v­ ¬‡ 2loss/SRqqt1/assignment_loss‚0.9419362965316594 2loss/SRqqt2/assignment_loss‚0.8725022500125217 4loss/FBt1/assignment_loss‚4.9868536192339734e-06 .loss/FRt2/detection_loss‚2.415760807300869 .loss/FBt1/detection_loss‚4.564322465409835 % _timestamp‚1.7858098035523515e+09 _step‚763 &loss/total_loss‚19.727582327291667 /loss/FRt1/assignment_loss‚2.037292939513477 /loss/FRt2/assignment_loss‚2.859175121156793 .loss/FRt1/detection_loss‚2.788688846488497 trainer/global_step‚9549 3loss/FBt2/assignment_loss‚9.909057580036125e-06 1loss/SRqqt1/detection_loss‚1.0688120089726496 1loss/SRqqt2/detection_loss‚1.4782289495947314 epoch‚0 _runtime‚4081.843856781 /loss/FBt2/detection_loss‚0.7008477463994337ûù[¹êœ­‚(:‘ ˜—ÅÓà““Ó gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚4.1602325439453125) gpu.0.memoryAllocatedBytes‚ 3573612544 gpu.0.temp‚41.0 gpu.0.powerWatts‚154.808' gpu.0.enforcedPowerLimitWatts‚180.0( gpu.0.powerPercent‚86.00444444444445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ç£¤®‚(:™  ˜—ÅÓÅÀÆÖ disk.rbd4.in‚0 disk.sda3.in‚ 0.45703125 disk.rbd5.in‚0 disk.rbd5.out‚ 2.146484375' disk.nvme0c0n1.out‚15622.3876953125 disk.sda3.out‚209.5625 cpu‚13.971865064855683$ disk./.usageGB‚1243.809627532959 disk.drbd1743.in‚0" disk.dm-0.out‚15622.3876953125 disk.rbd0.out‚0% disk.nvme0n1.out‚15622.3876953125 disk.sda.out‚209.5625 disk.dm-1.in‚ 0.45703125 disk.dm-1.out‚209.5625) proc.memory.percent‚17.19306707382202 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0) disk./.usagePercent‚83.47310772952693 disk.rbd3.in‚0 disk.nvme0n1.in‚ 38.40234375 disk.rbd4.out‚ 2.08349609375! disk.nvme0c0n1.in‚ 38.40234375 disk.rbd2.in‚0 disk.rbd1.out‚ 2.146484375$ memory_percent‚27.46325135231018 network.sent‚ 8.882365e+06 network.recv‚539734 disk.dm-0.in‚ 38.40234375 disk.rbd1.in‚0# proc.memory.rssMB‚ 11267.6484375 proc.cpu.threads‚29 disk.drbd1743.out‚ 4.51171875 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.45703125 disk.rbd2.out‚ 2.146484375 disk.sda2.in‚0* proc.memory.availableMB‚47537.68359375è´Ñ2¯- _wandbruntime‚4100  _runtime‚41008¨Ó2Æ°À _step‚764 & lr-AdamW/pg1‚0.0002473872180724532 & lr-AdamW/pg2‚0.0002473872180724532 trainer/global_step‚9599 $ _timestamp‚1.785809822595753e+09 _runtime‚4100.887389925I½¡Ã˱Š_step‚764 & lr-AdamW/pg1‚0.0002473872180724532 & lr-AdamW/pg2‚0.0002473872180724532 trainer/global_step‚9599 $ _timestamp‚1.785809822595753e+09 _runtime‚4100.887389925üåÀG72²- _wandbruntime‚4100  _runtime‚4100uË%ÐO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8446854353408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173869249649152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8426267673080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101373033 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37797595 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173869411518464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8446854353408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65294139 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8446941783040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63777379 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚9599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101373033 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8446941783040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160710198949472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63777377 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8675104034304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165194170518528 _step‚765 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101382689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚169136483661912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚169136380819428 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173252304870912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165160068 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63777377 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173252409339904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164805450517504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63777379Ýb@4Û'´ÕO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 136965632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9658 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65293890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37797597 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 161869312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 2645753856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚85803124 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173252304870912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚164805467556864 YLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚86278144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 102842484 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173252409339904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164805450517504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63777379 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 189288448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 3021996032 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚121 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8446854353408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173869249649152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8426267673080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 3021996032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101373033 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 2712961024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37797595 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173869411518464 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11887 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8446854353408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65294139 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 145651200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 248886712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚130 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8446941783040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚548 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚130 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9656 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 2768240640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63777379 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 2712961024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 3021996032 trainer/global_step‚9599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101373033 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8446941783040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 250523136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160710198949472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63777377 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9658 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1291 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8675104034304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165194170518528 _step‚765 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101382689 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚169136483661912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚169136380819428 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 253755392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173252304870912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165160068 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63777377 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚121 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165177248 _step‚766 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63784022 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101383556 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173270351253504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9599 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165167576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65300643 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚169153998464970 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63784022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27499111 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164822617280512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164822634319872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27499352 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8447821409792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8427145146414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160726939121792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101393226 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169154101307566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37801534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173887361591296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚121 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101383556 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 2940455936 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 2768240640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 253755392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160726956161152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165177248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164822634319872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8447821409792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8676008859648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164822617280512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8676141623808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173270455729664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165211352731648bk£$·ÞM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165211377635328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173887519259136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173270455729664 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8447733972992 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65300886 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37801532 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63784020 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8427059343178 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7858098229268858e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160726956161152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165177248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104476160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164822634319872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8447821409792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8676008859648 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 3021996032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 2768240640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87436800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚251 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚164822617280512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8676141623808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173270455729664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 2940455936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚165211352731648 _runtime‚ 4101.21904117 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8447733972992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165177248 _step‚766 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63784022 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101383556 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚85803236 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 2768240640 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 253755392 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚251 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173270351253504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104476160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚9599 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 2712961024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165167576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 157667840 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 102842596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 2645753856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65300643 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚169153998464970 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63784022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27499111 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 248886712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164822617280512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 250523136 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164822634319872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11887 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27499352 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 2871874888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11887 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9670 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8447821409792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚548 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 132764160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87436800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚121 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9670 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8427145146414 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚251 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160726939121792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 253755392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚18 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 2712961024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101393226 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚251 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 3021996032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169154101307566 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37801534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173887361591296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 3021996032 JBDeviceStatsMonitor.on_train_batch_end/segm