Environment variables#
pimm reads these 43 variables. scripts/train.sh first sources a .env file at the repository root, so assignments there reach every training process. <NAME> stands for part of a variable name filled in at run time, such as a dataset revision. A default is the value the code falls back to when the variable is unset. Libraries pimm uses, such as PyTorch, W&B and the Hugging Face Hub, read their own variables, which aren’t listed.
Data#
Variable |
Default |
Read in |
Note in the code |
|---|---|---|---|
|
|
|
— |
|
— |
|
Revision-specific env vars keep each data root independent. |
|
— |
|
— |
Checkpoints and downloads#
Variable |
Default |
Read in |
Note in the code |
|---|---|---|---|
|
— |
|
— |
|
— |
|
— |
|
— |
|
Respect an external/.env MODEL_DIR (empty by default) and export it so the training process sees it (checkpoint redirect + HF cache location). |
|
— |
|
— |
|
— |
|
— |
Distributed runs and Slurm#
Variable |
Default |
Read in |
Note in the code |
|---|---|---|---|
|
— |
|
— |
|
|
|
— |
|
— |
|
— |
|
|
|
Get master address from SLURM_NODELIST when torchrun/env did not set it. |
|
|
|
Derive a stable port for SLURM jobs so all ranks rendezvous together. |
|
|
|
— |
|
|
|
— |
|
|
|
— |
|
|
|
— |
|
|
|
— |
|
|
|
— |
|
|
|
— |
|
— |
|
— |
|
|
|
Prefer torchrun’s explicit rank env, then fall back to SLURM variables. |
|
— |
|
— |
|
— |
|
— |
|
|
|
— |
Launcher and job scripts#
Variable |
Default |
Read in |
Note in the code |
|---|---|---|---|
|
|
|
— |
|
— |
|
— |
|
|
|
— |
|
— |
|
— |
|
|
|
— |
|
|
|
— |
|
— |
|
— |
|
— |
|
— |
|
— |
|
— |
Training#
Variable |
Default |
Read in |
Note in the code |
|---|---|---|---|
|
|
|
— |
|
— |
|
— |
|
— |
|
— |
|
|
|
— |
|
— |
|
The original user-facing |
|
— |
|
— |
Logging and traces#
Variable |
Default |
Read in |
Note in the code |
|---|---|---|---|
|
— |
|
— |
|
|
|
— |
|
— |
|
— |