Run and reproduce#
Operational guides for changing an experiment without losing its provenance or misstating what an interrupted run did.
Configuration
Inheritance, precedence, overrides, common fields, and resolved artifacts.
Checkpoints and resume
On-disk formats, atomic saves, warm starts, exact-resume boundary, and topology changes.
Logging and diagnostics
Console, TensorBoard, W&B, health monitors, and profiling.
Structured logging
Per-rank phase traces for distributed stalls, stragglers, and restarts.
Reproducibility
The minimum artifact and metadata bundle behind a defensible result.
Troubleshooting
Find an exact symptom, run one diagnostic, and apply the narrow fix.