2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_setup.py:_flush():81] Current SDK version is 0.28.1
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_setup.py:_flush():81] Configure stats pid to 202
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_init.py:setup_run_log_directory():729] Logging user logs to /scratch/logs/wandb/run-20260723_214221-tfees60h/logs/debug.log
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_init.py:setup_run_log_directory():730] Logging internal logs to /scratch/logs/wandb/run-20260723_214221-tfees60h/logs/debug-internal.log
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_init.py:init():772] calling init triggers
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_init.py:init():777] wandb.init called with sweep_config: {}
config: {'_wandb': {}}
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_init.py:init():820] starting backend
2026-07-23 21:42:21,277 INFO    MainThread:202 [wandb_init.py:init():835] sending inform_init request
2026-07-23 21:42:21,442 INFO    MainThread:202 [wandb_init.py:init():840] backend started and connected
2026-07-23 21:42:21,447 INFO    MainThread:202 [wandb_init.py:init():910] updated telemetry
2026-07-23 21:42:21,452 INFO    MainThread:202 [wandb_init.py:init():933] communicating run to backend with 90.0 second timeout
2026-07-23 21:42:21,996 INFO    MainThread:202 [wandb_init.py:init():978] starting run threads in backend
2026-07-23 21:42:22,077 INFO    MainThread:202 [wandb_run.py:_console_start():2621] atexit reg
2026-07-23 21:42:22,077 INFO    MainThread:202 [wandb_run.py:_redirect():2471] redirect: wrap_raw
2026-07-23 21:42:22,077 INFO    MainThread:202 [wandb_run.py:_redirect():2540] Wrapping output streams.
2026-07-23 21:42:22,077 INFO    MainThread:202 [wandb_run.py:_redirect():2563] Redirects installed.
2026-07-23 21:42:22,081 INFO    MainThread:202 [wandb_init.py:init():1016] run started, returning control to user process
2026-07-23 21:42:22,418 INFO    MainThread:202 [wandb_run.py:_config_callback():1346] config_cb None None {'hidden_dim': 256, 'transformer_dim': 256, 'transformer_dim_scale': 2.0, 'initial_embedding_dim': 256, 'position_embedding_dim': 256, 'num_embedding_layers': 6, 'num_encoder_layers': 4, 'num_branch_embedding_layers': 3, 'num_branch_encoder_layers': 3, 'num_jet_embedding_layers': 0, 'num_jet_encoder_layers': 2, 'num_detector_layers': 2, 'num_regression_layers': 3, 'num_classification_layers': 3, 'split_symmetric_attention': 1, 'num_attention_heads': 8, 'transformer_activation': 'gelu', 'skip_connections': 1, 'initial_embedding_skip_connections': 1, 'linear_block_type': 'GRU', 'transformer_type': 'Gated', 'linear_activation': 'gelu', 'normalization': 'LayerNorm', 'masking': 'Filling', 'linear_prelu_activation': True, 'use_pairwise_interactions': 1, 'num_pairwise_features': 4, 'pairwise_embedding_dim': 8, 'pairwise_input_source': 'all', 'pairwise_block_embeddings': 1, 'pairwise_cross_type': True, 'use_moe': False, 'num_experts': 8, 'num_experts_per_tok': 2, 'moe_loss_scale': 0.01, 'event_info_file': '/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml', 'training_file': '/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5', 'validation_file': '', 'testing_file': '', 'normalize_features': 1, 'limit_to_num_jets': 0, 'balance_particles': 0, 'balance_jets': 0, 'balance_classifications': False, 'partial_events': 1, 'dataset_limit': 0.1, 'dataset_randomization': 0, 'train_validation_split': 0.95, 'limit_val_batches': 1.0, 'batch_size': 2048, 'num_dataloader_workers': 0, 'mask_sequence_vectors': 1, 'combine_pair_loss': 'min', 'optimizer': 'AdamW', 'learning_rate': 0.00011712902303919398, 'focal_gamma': 0.0, 'combinatorial_scale': 0.0, 'learning_rate_warmup_epochs': 1.0, 'learning_rate_cycles': 1, 'assignment_loss_scale': 1.0, 'detection_loss_scale': 1.0, 'kl_loss_scale': 0.0, 'regression_loss_scale': 0.0, 'classification_loss_scale': 0.0, 'use_mass_classification': False, 'mass_classes': [], 'balance_losses': True, 'l2_penalty': 0.0002, 'gradient_clip': 10.0, 'dropout': 0.26251405799488203, 'epochs': 50, 'num_gpu': 1, 'verbose_output': False, 'usable_gpus': '', 'trial_time': '', 'trial_output_dir': './test_output'}
2026-07-23 22:45:56,499 INFO    wandb-AsyncioManager-main:202 [service_client.py:_forward_responses():122] [no run ID] Reached EOF.
2026-07-23 22:45:56,499 INFO    wandb-AsyncioManager-main:202 [mailbox.py:close():154] [no run ID] Closing mailbox, abandoning 2 handles.
2026-07-23 22:45:56,499 ERROR   wandb-AsyncioManager-main:202 [asyncio_manager.py:fn_wrap_exceptions():197] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 195, in fn_wrap_exceptions
    await fn()
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
    await asyncio_compat.race(
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 279, in race
    async with open_task_group(race=True) as tg:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/contextlib.py", line 217, in __aexit__
    await anext(self.gen)
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 241, in open_task_group
    await task_group._wait_all(race=race, timeout=exit_timeout)
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_compat.py", line 181, in _wait_all
    raise exc
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
    result = await handle.wait_async(timeout=None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
    response = await self._handle.wait_async(timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
    raise HandleAbandonedError()
wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError
2026-07-23 22:46:03,248 ERROR   wandb-AsyncioManager-main:202 [asyncio_manager.py:fn_wrap_exceptions():197] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/asyncio_manager.py", line 195, in fn_wrap_exceptions
    await fn()
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/run_stopping.py", line 87, in loop
    should_stop = await self._check_should_stop()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/run_stopping.py", line 115, in _check_should_stop
    handle = await self._interface.deliver_async(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/interface/interface_sock.py", line 58, in deliver_async
    handle = await self._client.deliver(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/lib/service/service_client.py", line 57, in deliver
    handle = self._mailbox.require_response(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/wandb/sdk/mailbox/mailbox.py", line 89, in require_response
    raise MailboxClosedError()
wandb.sdk.mailbox.mailbox.MailboxClosedError