Environment variables#

pimm reads these 43 variables. scripts/train.sh first sources a .env file at the repository root, so assignments there reach every training process. <NAME> stands for part of a variable name filled in at run time, such as a dataset revision. A default is the value the code falls back to when the variable is unset. Libraries pimm uses, such as PyTorch, W&B and the Hugging Face Hub, read their own variables, which aren’t listed.

Data#

Variable

Default

Read in

Note in the code

JAXTPC_DATA_ROOT

/path/to/jaxtpc/production

configs/detector/_base_/jaxtpc_seg.py

—

PILARNET_DATA_ROOT_<REVISION>

—

pimm/datasets/pilarnet/h5.py (PILArNetH5Dataset.__init__)

Revision-specific env vars keep each data root independent.

PILARNET_PARQUET_ROOT_<REVISION>

—

pimm/datasets/pilarnet/parquet.py (PILArNetParquetDataset.__init__), pimm/datasets/pilarnet/parquet.py (PILArNetParquetIterableDataset.__init__)

—

Checkpoints and downloads#

Variable

Default

Read in

Note in the code

HF_HOME

—

pimm/utils/checkpoints.py (configure_hf_cache)

—

HF_HUB_CACHE

—

pimm/utils/checkpoints.py (configure_hf_cache)

—

MODEL_DIR

—

pimm/utils/checkpoints.py (configure_hf_cache), scripts/train.sh

Respect an external/.env MODEL_DIR (empty by default) and export it so the training process sees it (checkpoint redirect + HF cache location).

WANDB_ENTITY

—

pimm/utils/checkpoints.py (configure_logger_from_checkpoint)

—

WANDB_PROJECT

—

pimm/utils/checkpoints.py (configure_logger_from_checkpoint)

—

Distributed runs and Slurm#

Variable

Default

Read in

Note in the code

GROUP_RANK

—

pimm/utils/comm.py (setup_distributed)

—

LOCAL_RANK

0

pimm/observability/structured_logger/jsonl_handler.py (TraceJsonlFormatter.__init__), pimm/utils/comm.py (setup_distributed)

—

LOCAL_WORLD_SIZE

—

pimm/utils/comm.py (setup_distributed)

—

MASTER_ADDR

127.0.0.1

pimm/utils/comm.py (setup_distributed), pimm/launch/torchrun.sh

Get master address from SLURM_NODELIST when torchrun/env did not set it.

MASTER_PORT

29500

pimm/utils/comm.py (setup_distributed), pimm/launch/torchrun.sh

Derive a stable port for SLURM jobs so all ranks rendezvous together.

RANK

0

pimm/observability/structured_logger/structured_logging.py (init_structured_logger), pimm/utils/comm.py (setup_distributed)

—

SLURM_JOB_ID

local, pimm

pimm/observability/structured_logger/structured_logging.py (init_structured_logger), pimm/utils/comm.py (get_slurm_env), scripts/train.sh, pimm/launch/torchrun.sh

—

SLURM_LOCALID

0

pimm/utils/comm.py (get_slurm_env)

—

SLURM_NNODES

1

pimm/utils/comm.py (get_slurm_env)

—

SLURM_NODEID

0

pimm/utils/comm.py (get_slurm_env), pimm/utils/comm.py (setup_distributed), pimm/launch/torchrun.sh

—

SLURM_NODELIST

127.0.0.1

pimm/utils/comm.py (get_slurm_env)

—

SLURM_NTASKS

1

pimm/utils/comm.py (get_slurm_env)

—

SLURM_NTASKS_PER_NODE

—

pimm/utils/comm.py (get_slurm_env)

—

SLURM_PROCID

0

pimm/utils/comm.py (get_slurm_env), pimm/utils/comm.py (setup_distributed), scripts/train.sh, pimm/launch/torchrun.sh

Prefer torchrun’s explicit rank env, then fall back to SLURM variables.

SLURM_STEP_TASKS_PER_NODE

—

pimm/utils/comm.py (get_slurm_env)

—

SLURM_TASKS_PER_NODE

—

pimm/utils/comm.py (get_slurm_env)

—

WORLD_SIZE

1

pimm/observability/structured_logger/structured_logging.py (init_structured_logger), pimm/utils/comm.py (setup_distributed)

—

Launcher and job scripts#

Variable

Default

Read in

Note in the code

EXP_ROOT

exp

scripts/train.sh

—

PIMM_NODE_RANK

—

pimm/launch/torchrun.sh

—

PIMM_RDZV_BACKEND

c10d

pimm/launch/torchrun.sh

—

PIMM_RDZV_ID

—

pimm/launch/torchrun.sh

—

PIMM_USE_EXEX

0

pimm/engines/_execution.py (configure_training), pimm/launch/_exex.py (enabled), pimm/launch/config.py (finalize_config)

—

PIMM_WANDB_HISTORY

new

pimm/engines/defaults.py (resolve_wandb_history), pimm/launch/config.py (finalize_config), pimm/utils/events.py (WandbSummaryWriter.__init__)

—

PIMM_WATCHDOG_CHILD

—

pimm/launch/submit.py (run_submit)

—

SLURM_JOB_START_TIME

—

scripts/train.sh

—

USER

—

pimm/launch/watchdog.py (dir_state), scripts/pilarnet/stage_lscratch.sh

—

Training#

Variable

Default

Read in

Note in the code

EXEX_INPUT_DIR

/nonexistent

pimm/engines/_execution.py (configure_training)

—

EXEX_OUTPUT_DIR

—

pimm/engines/_execution.py (configure_training)

—

EXEX_PAUSE_REQUEST

—

pimm/engines/_execution.py (configure_training._ExecutionCheckpoint._pause)

—

PIMM_DISABLE_SERIALIZE_CUDA

0

pimm/models/utils/serialization/_cuda_backend.py (_disabled)

—

PIMM_LAUNCH_COMMAND

—

pimm/engines/_train_utils.py (_save_config_artifacts), pimm/engines/defaults.py (default_config_parser)

The original user-facing pimm submit/launch ... command (set by the launcher via PIMM_LAUNCH_COMMAND) – the command to re-run this job.

PIMM_SIMULATE_CRASH_STEP

—

pimm/engines/hooks/diagnostics.py (SimulateCrash.after_step)

—

Logging and traces#

Variable

Default

Read in

Note in the code

PIMM_STRUCT_LOGGER_HANDLERS

—

pimm/observability/structured_logger/structured_logging.py (init_structured_logger)

—

PIMM_SUBMITIT_ATTEMPT

1

pimm/observability/structured_logger/structured_logging.py (init_structured_logger)

—

TORCHELASTIC_RUN_ID

—

pimm/observability/structured_logger/structured_logging.py (init_structured_logger)

—