2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_setup.py:_flush():81] Current SDK version is 0.30.0
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_setup.py:_flush():81] Configure stats pid to 508
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_init.py:setup_run_log_directory():739] Logging user logs to /scratch/c2v-B-ghost-fixed/wandb/run-20261011_001216-z2r63i9q/logs/debug.log
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_init.py:setup_run_log_directory():740] Logging internal logs to /scratch/c2v-B-ghost-fixed/wandb/run-20261011_001216-z2r63i9q/logs/debug-internal.log
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_init.py:init():783] calling init triggers
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_init.py:init():788] wandb.init called with sweep_config: {}
config: {'_wandb': {}}
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_init.py:init():831] starting backend
2026-10-11 00:12:16,517 INFO    MainThread:508 [wandb_init.py:init():846] sending inform_init request
2026-10-11 00:12:16,557 INFO    MainThread:508 [wandb_init.py:init():851] backend started and connected
2026-10-11 00:12:16,558 INFO    MainThread:508 [wandb_init.py:init():921] updated telemetry
2026-10-11 00:12:16,561 INFO    MainThread:508 [wandb_init.py:init():944] communicating run to backend with 90.0 second timeout
2026-10-11 00:12:17,146 INFO    MainThread:508 [wandb_init.py:init():981] starting run threads in backend
2026-10-11 00:12:17,218 INFO    MainThread:508 [wandb_run.py:_console_start():2531] atexit reg
2026-10-11 00:12:17,218 INFO    MainThread:508 [wandb_run.py:_redirect():2381] redirect: wrap_raw
2026-10-11 00:12:17,218 INFO    MainThread:508 [wandb_run.py:_redirect():2450] Wrapping output streams.
2026-10-11 00:12:17,218 INFO    MainThread:508 [wandb_run.py:_redirect():2473] Redirects installed.
2026-10-11 00:12:17,220 INFO    MainThread:508 [wandb_init.py:init():1019] run started, returning control to user process
2026-10-11 00:12:17,221 INFO    MainThread:508 [wandb_run.py:_config_callback():1283] config_cb None None {'hidden_dim': 128, 'transformer_dim': 128, 'transformer_dim_scale': 2.0, 'initial_embedding_dim': 128, 'position_embedding_dim': 128, 'num_embedding_layers': 10, 'num_encoder_layers': 4, 'num_branch_embedding_layers': 3, 'num_branch_encoder_layers': 3, 'num_jet_embedding_layers': 0, 'num_jet_encoder_layers': 2, 'num_detector_layers': 2, 'num_regression_layers': 3, 'num_classification_layers': 3, 'split_symmetric_attention': 1, 'num_attention_heads': 8, 'transformer_activation': 'gelu', 'skip_connections': 1, 'initial_embedding_skip_connections': 1, 'linear_block_type': 'GRU', 'transformer_type': 'Gated', 'linear_activation': 'gelu', 'normalization': 'LayerNorm', 'masking': 'Filling', 'linear_prelu_activation': True, 'use_pairwise_interactions': False, 'num_pairwise_features': 4, 'pairwise_embedding_dim': 8, 'pairwise_input_source': '', 'pairwise_block_embeddings': False, 'pairwise_cross_type': True, 'pairwise_feature_set': '', 'pairwise_eq2to2': False, 'use_moe': False, 'num_experts': 8, 'num_experts_per_tok': 2, 'moe_loss_scale': 0.01, 'event_info_file': '/config/tt_hadronic_ghost_fixed.yaml', 'training_file': '/traindatavol/collide2v/tt_hadronic_1M_ghost_v2/tt_hadronic_training.h5', 'validation_file': '/traindatavol/collide2v/tt_hadronic_1M_ghost_v2/tt_hadronic_validation.h5', 'testing_file': '', 'normalize_features': 1, 'limit_to_num_jets': 0, 'balance_particles': 0, 'balance_jets': 0, 'balance_classifications': False, 'partial_events': 1, 'dataset_limit': 1.0, 'dataset_randomization': 1, 'train_validation_split': 1.0, 'limit_val_batches': 1.0, 'batch_size': 1024, 'num_dataloader_workers': 0, 'mask_sequence_vectors': 1, 'combine_pair_loss': 'min', 'optimizer': 'AdamW', 'learning_rate': 0.0004, 'focal_gamma': 0.0, 'combinatorial_scale': 0.0, 'learning_rate_warmup_epochs': 1.0, 'learning_rate_cycles': 1, 'assignment_loss_scale': 1.0, 'detection_loss_scale': 1.0, 'kl_loss_scale': 0.0, 'regression_loss_scale': 0.0, 'classification_loss_scale': 0.0, 'use_mass_classification': False, 'mass_classes': [], 'balance_losses': True, 'l2_penalty': 0.0002, 'gradient_clip': 10.0, 'dropout': 0.2, 'epochs': 500, 'num_gpu': 1, 'verbose_output': False, 'usable_gpus': '', 'trial_time': '', 'trial_output_dir': './test_output', 'run_dir': '/scratch/c2v-B-ghost-fixed', 'spatop_git_sha': '036d6f0', 'spanet_ref': 'aaa5635cd51bdd30926dbc91234271398cf7412a'}
2026-10-11 00:12:17,719 INFO    MainThread:508 [wandb_run.py:_config_callback():1283] config_cb None None {'hidden_dim': 128, 'transformer_dim': 128, 'transformer_dim_scale': 2.0, 'initial_embedding_dim': 128, 'position_embedding_dim': 128, 'num_embedding_layers': 10, 'num_encoder_layers': 4, 'num_branch_embedding_layers': 3, 'num_branch_encoder_layers': 3, 'num_jet_embedding_layers': 0, 'num_jet_encoder_layers': 2, 'num_detector_layers': 2, 'num_regression_layers': 3, 'num_classification_layers': 3, 'split_symmetric_attention': 1, 'num_attention_heads': 8, 'transformer_activation': 'gelu', 'skip_connections': 1, 'initial_embedding_skip_connections': 1, 'linear_block_type': 'GRU', 'transformer_type': 'Gated', 'linear_activation': 'gelu', 'normalization': 'LayerNorm', 'masking': 'Filling', 'linear_prelu_activation': True, 'use_pairwise_interactions': False, 'num_pairwise_features': 4, 'pairwise_embedding_dim': 8, 'pairwise_input_source': '', 'pairwise_block_embeddings': False, 'pairwise_cross_type': True, 'pairwise_feature_set': '', 'pairwise_eq2to2': False, 'use_moe': False, 'num_experts': 8, 'num_experts_per_tok': 2, 'moe_loss_scale': 0.01, 'event_info_file': '/config/tt_hadronic_ghost_fixed.yaml', 'training_file': '/traindatavol/collide2v/tt_hadronic_1M_ghost_v2/tt_hadronic_training.h5', 'validation_file': '/traindatavol/collide2v/tt_hadronic_1M_ghost_v2/tt_hadronic_validation.h5', 'testing_file': '', 'normalize_features': 1, 'limit_to_num_jets': 0, 'balance_particles': 0, 'balance_jets': 0, 'balance_classifications': False, 'partial_events': 1, 'dataset_limit': 1.0, 'dataset_randomization': 1, 'train_validation_split': 1.0, 'limit_val_batches': 1.0, 'batch_size': 1024, 'num_dataloader_workers': 0, 'mask_sequence_vectors': 1, 'combine_pair_loss': 'min', 'optimizer': 'AdamW', 'learning_rate': 0.0004, 'focal_gamma': 0.0, 'combinatorial_scale': 0.0, 'learning_rate_warmup_epochs': 1.0, 'learning_rate_cycles': 1, 'assignment_loss_scale': 1.0, 'detection_loss_scale': 1.0, 'kl_loss_scale': 0.0, 'regression_loss_scale': 0.0, 'classification_loss_scale': 0.0, 'use_mass_classification': False, 'mass_classes': [], 'balance_losses': True, 'l2_penalty': 0.0002, 'gradient_clip': 10.0, 'dropout': 0.2, 'epochs': 500, 'num_gpu': 1, 'verbose_output': False, 'usable_gpus': '', 'trial_time': '', 'trial_output_dir': './test_output'}