/opt/conda/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( /opt/conda/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( Configuration ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Parameter ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ assignment_loss_scale │ 1.0 │ │ balance_classifications │ False │ │ balance_jets │ 0 │ │ balance_losses │ True │ │ balance_particles │ 0 │ │ batch_size │ 1024 │ │ classification_loss_scale │ 0.0 │ │ combinatorial_scale │ 0.0 │ │ combine_pair_loss │ min │ │ dataset_limit │ 1.0 │ │ dataset_randomization │ 0 │ │ detection_loss_scale │ 1.0 │ │ dropout │ 0.2 │ │ epochs │ 500 │ │ event_info_file │ /config/tt_hadronic.yaml │ │ focal_gamma │ 0.0 │ │ gradient_clip │ 10.0 │ │ hidden_dim │ 128 │ │ initial_embedding_dim │ 128 │ │ initial_embedding_skip_connections │ 1 │ │ kl_loss_scale │ 0.0 │ │ l2_penalty │ 0.0002 │ │ learning_rate │ 0.0004 │ │ learning_rate_cycles │ 1 │ │ learning_rate_warmup_epochs │ 1.0 │ │ limit_to_num_jets │ 0 │ │ limit_val_batches │ 1.0 │ │ linear_activation │ gelu │ │ linear_block_type │ GRU │ │ linear_prelu_activation │ True │ │ mask_sequence_vectors │ 1 │ │ masking │ Filling │ │ mass_classes │ [] │ │ moe_loss_scale │ 0.01 │ │ normalization │ LayerNorm │ │ normalize_features │ 1 │ │ num_attention_heads │ 8 │ │ num_branch_embedding_layers │ 3 │ │ num_branch_encoder_layers │ 3 │ │ num_classification_layers │ 3 │ │ num_dataloader_workers │ 0 │ │ num_detector_layers │ 2 │ │ num_embedding_layers │ 10 │ │ num_encoder_layers │ 4 │ │ num_experts │ 8 │ │ num_experts_per_tok │ 2 │ │ num_gpu │ 1 │ │ num_jet_embedding_layers │ 0 │ │ num_jet_encoder_layers │ 2 │ │ num_pairwise_features │ 4 │ │ num_regression_layers │ 3 │ │ optimizer │ AdamW │ │ pairwise_embedding_dim │ 8 │ │ pairwise_input_source │ │ │ partial_events │ 1 │ │ position_embedding_dim │ 128 │ │ regression_loss_scale │ 0.0 │ │ skip_connections │ 1 │ │ split_symmetric_attention │ 1 │ │ testing_file │ │ │ train_validation_split │ 0.95 │ │ training_file │ /traindatavol/collide2v/tt_hadronic_1M… │ │ transformer_activation │ gelu │ │ transformer_dim │ 128 │ │ transformer_dim_scale │ 2.0 │ │ transformer_type │ Gated │ │ trial_output_dir │ ./test_output │ │ trial_time │ │ │ usable_gpus │ │ │ use_mass_classification │ False │ │ use_moe │ False │ │ use_pairwise_interactions │ False │ │ validation_file │ │ │ verbose_output │ False │ └────────────────────────────────────┴─────────────────────────────────────────┘ Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Index Range: 0...245991 Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Warning: no target weights in the dataset, creating ones weights Index Range: 245992...258938 Trainer already configured with model summary callbacks: []. Skipping setting a default `ModelSummary` callback. GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores 💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform. `Trainer(limit_val_batches=1.0)` was configured so 100% of the batches will be used.. Training Version c2v-C-v0main-asmain (/scratch/c2v-C-v0main-asmain) wandb: WARNING The anonymous setting has no effect and will be removed in a future version. wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY. wandb: Currently logged in as: mequinna (spatop) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin wandb: setting up run snd4yqu5 wandb: Tracking run with wandb version 0.30.0 wandb: Run data is saved locally in /scratch/c2v-C-v0main-asmain/wandb/run-20261010_213913-snd4yqu5 wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run c2v-C-v0main-asmain wandb: ⭐️ View project at https://wandb.ai/spatop/spatop-train wandb: 🚀 View run at https://wandb.ai/spatop/spatop-train/runs/snd4yqu5 You are using a CUDA device ('NVIDIA A100-SXM4-80GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ┏━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.814 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 /opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. /opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=49` in the `DataLoader` to improve performance. /opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. /opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=49` in the `DataLoader` to improve performance. `Trainer.fit` stopped: `max_epochs=500` reached. Epoch 499/499 ━━━━━━━━━━━━━━━━━━━ 240/240 0:02:06 • 0:00:00 1.90it/s v_num: yqu5 wandb: wandb: 🚀 View run c2v-C-v0main-asmain at: https://wandb.ai/spatop/spatop-train/runs/snd4yqu5 wandb: Find logs at: ../../scratch/c2v-C-v0main-asmain/wandb/run-20261010_213913-snd4yqu5/logs