Justfile Command Reference
| Status | Applies to | Owner |
|---|---|---|
| Pre-release draft | current Justfile command surface | Development maintainers |
The repository Justfile is a thin, stable entrypoint layer. It names common
workflows and delegates workflow truth to saved Hydra configs under
tooling/configs/.
Run just help for the full recipe list.
All of Us and Workbench
These recipes select saved configs for the experimental All of Us workflow. Pass paths and research-specific selections as trailing Hydra overrides.
| Recipe | Purpose |
|---|---|
data-all-of-us |
Plan or execute explicit local cohort selection and BGEN export. |
workspace-demo |
Generate a new participant-free integration bundle. |
workspace-preflight |
Check a pinned job and estimate its chunk resources. |
workspace-profile |
Inspect a bounded genotype prefix, retaining native-validation requirements. |
workspace-run |
Launch a fresh attempt, validate outputs and publish a completion marker. |
Workbench tooling uses the minimal workbench dependency group. It is copied
into the deployment image and remains outside the installed public g API.
Real controlled-data execution and validation must occur in the authorized
workspace.
Policy
Environment variables configure the machine.
Hydra configs configure the workflow.
Justfile recipes select the workflow.
Environment variables are appropriate for secrets, scheduler-provided state, local cache/toolchain roots, and explicit one-off path overrides. They must not be the primary source of truth for benchmark grids, profiler modes, dataset or chromosome identity, output format, run semantics, or named machine profiles.
Hydra-backed recipes accept trailing overrides for ad hoc work, but routine
workflow behavior belongs in a saved config. Every migrated tooling.cli.*
recipe should use --config-name.
Core Development
Use these on the login node when they are lightweight:
just doctor
just doctor-server
just dev-bootstrap
just dev-bootstrap-gpu
just dev-install
just dev-install-release
just cuda-format
just cuda-format-check
just cuda-lint
just check-local
just check-artifact-schema data/profiles/example/report.json
just check-rust-architecture
just rust-check
just workspace-check
just test-local
just docs-build
just docs-check
server-setup-tools runs in a minimal uv --no-project environment so it can
install Mold before any Linux build attempts to compile the project itself.
dev-install uses Cargo's fast dev profile. dev-install-release uses the
FatLTO, single-codegen-unit release profile used by benchmarks and final
builds. Cargo defaults to 30 build jobs through .cargo/config.toml;
CARGO_BUILD_JOBS or an explicit --jobs option can impose a different cap.
SLURM jobs use the scheduler allocation unless CARGO_BUILD_JOBS is already
set.
cuda-format applies the repository ClangFormat policy to the explicit list of
maintained .cu, .cc, and .h files, including the shared private CUDA
driver support header. cuda-format-check checks the same files without
rewriting them. cuda-lint runs the pinned ClangTidy analysis of
CUDA host/device code and native C++ through the saved
debug_check_cuda_native configuration. These commands exclude generated PTX
and vendored sources. Static analysis runs on Linux x86-64 without a GPU and
does not compile or regenerate PTX. The aggregate format, lint, check,
and check-local recipes include the corresponding native checks.
Use SLURM for CPU-heavy validation:
just slurm-cpu-check
just slurm-cpu-test
just slurm-cpu-rust-build
just slurm-cpu-rust-test
just slurm-gpu-test-parity-required
CPU and GPU correctness run in separate processes. The CPU recipe excludes data-dependent parity; the GPU recipe qualifies all blocking upstream-REGENIE workflows with required local fixtures.
Data
just data-fetch
just data-simulate
just data-prepare
just data-baseline-binary
just data-baseline-qt
just data-verify-binary-gpu-inputs
GWAS_ENGINE_DATA_DIR may point at a shared data directory, such as the main
checkout's gitignored data/ directory from a temporary worktree.
Matrices
just matrix-chr10-dry
just slurm-gpu-matrix-chr10
just matrix-chr22-dry
just slurm-gpu-matrix-chr22
The matrix-* workflows replace direct hand-written g regenie recipes. Use
dry configs for command/config validation and full matrix configs for parity or
performance runs. Bounded execution requires a prepared bounded BGEN fixture;
the production CLI has no variant-limit option.
Benchmarks
just bench-linear-startup-gpu
just bench-binary-hot-gpu
just bench-binary-hot-gpu-smoke
just bench-firth-compute-gpu
just rust-bench
just bench-torchgwas-chr22
just bench-tensorqtl-chr22
just bench-rust-build-profiles
just slurm-gpu-bench-binary-hot
just slurm-gpu-bench-firth-compute
just slurm-gpu-bench-torchgwas-chr22
just slurm-gpu-bench-tensorqtl-chr22
rust-bench runs Criterion benchmarks through cargo bench --workspace so
workspace-owned benches are discovered from their owning crates. Cargo's
bench profile inherits the maximum-performance release profile.
bench-rust-build-profiles compares the built-in dev and release profiles.
The repo Cargo configuration defaults to 30 build jobs and, on Linux, enables
target-cpu=native and links through Mold using the cc compiler driver.
doctor verifies the compiler driver can complete a tiny Mold link. SLURM jobs
isolate native Cargo artifacts under target/slurm/<node>/; an explicit
CARGO_TARGET_DIR still takes precedence.
RUSTC_WRAPPER remains an environment-specific choice.
Historical data-baseline generation remains available under legacy-*:
just legacy-baselines
just legacy-baselines-full
Use the TorchGWAS chr22 competitor benchmark through SLURM for real GPU evidence:
just slurm-gpu-bench-torchgwas-chr22
The benchmark records first-process and repeat-process runs for single-trait
quantitative chr22. It is a workflow/runtime comparison, not strict statistical
parity with REGENIE Step 2 LOCO output. g reads the chr22 BGEN plus sample
file; full TorchGWAS runs read the existing PLINK .bed/.bim/.fam triplet
because the pinned TorchGWAS BGEN path stalls in PLINK2 raw-table parsing at
chr22 scale. TorchGWAS PLINK runs do not emit a persistent genotype cache, so
their warm cases reflect repeated process/filesystem-cache behavior.
Bounded comparisons require a prepared genotype fixture shared by both tools;
the production g CLI does not expose a variant-limit option.
Use the tensorQTL chr22 competitor benchmark through SLURM for real GPU evidence:
just slurm-gpu-bench-tensorqtl-chr22
The benchmark records first-process and repeat-process runs for single-trait
quantitative chr22, but the comparison boundary is QTL-shaped: g runs REGENIE
Step 2 with LOCO predictions on BGEN input, while tensorQTL runs dense trans
nominal linear association on generated phenotype/covariate matrices and local
PLINK .bed/.bim/.fam input. Full runs expose the PLINK files through a
BED-only symlink prefix under the benchmark output directory because tensorQTL
auto-selects PGEN when PGEN and BED files share the same prefix, and its PGEN
reader fails on the local chr22 multiallelic records. Do not treat the result as
statistical parity.
Performance
just perf-smoke
just perf-gpu
just perf-compare results/perf/baseline.json results/perf/new.json
just perf-jax-runtime
perf-smoke and perf-compare are login-node-safe. perf-gpu submits through
SLURM and uses a saved benchmark config for its output location.
Profiling
just profile-deep-dry
just profile-deep-smoke
just profile-rust-criterion
just profile-app-full-dry
just profile-app-full-smoke
just profile-app-full
just profile-chr10-binary-gpu-dry
just profile-chr10-binary-gpu-full
just profile-chr22-binary-gpu-dry
just profile-chr22-binary-gpu-full
The full profile recipes submit GPU work through SLURM. The bare smoke recipes
are inner workflows and must run inside a GPU allocation; from the login node,
use just slurm-gpu-just profile-deep-smoke or
just slurm-gpu-just profile-app-full-smoke. Native Criterion profiles use the
configured CPU compute node. The focused chr10 and chr22 binary GPU recipes
keep their profiler selection and campaign budgets in saved Hydra configs
rather than in the Justfile.
SLURM Substrates
just slurm-cpu-shell
just slurm-cpu-run 'cargo build --workspace --all-targets'
just slurm-cpu-just check
just slurm-gpu-shell
just slurm-gpu-run 'nvidia-smi'
just slurm-gpu-just bench-binary-hot-gpu-smoke
The SLURM wrappers are execution substrates. Workflow choices should still live in saved configs or named Just recipes.
Guardrails
just check-rust-architecture verifies Cargo workspace dependency boundaries
for the Rust migration. just check-python-architecture verifies Python
ownership of current compute kernels, the numerical backend and its private
helpers. It rejects native CLI recursion, Python orchestration, file I/O and
worker queues in the numerical layer, and resolves imported aliases when
checking calls. Parameters and local assignments shadow enclosing imports;
ambiguous import rebinding within one scope is checked conservatively against
all imported targets. This AST check does not execute dynamic name assignments.
It also requires the current CLI, backend and compute source
roots to exist; a partial snapshot cannot silently pass.
just check-dead-code checks module reachability for all Python source under
src/g/ and tooling/, including tooling/data/. Its roots are console scripts
in pyproject.toml, module commands in the Justfile and CI workflows, maintained
test/script imports, literal native PyModule::import targets and configured
Hydra _target_ values. The supported direct tooling.data.fetch and
tooling.data.simulate interfaces are reviewed explicit roots. Registry imports,
JAX-decorated kernels, literal importlib calls and python -m argument lists
are followed as edges from reachable modules. A module's own __main__ block,
an arbitrary diagnostic string or an orphan import cycle does not establish
reachability. Empty package markers are allowed; reachable package initializers
are followed.
The gate reports source and root counts and fails on missing source anchors or
fewer than 30 production and 60 tooling Python files. These conservative floors
protect against empty or truncated snapshots and must be reviewed alongside an
intentional reduction of maintained source. The scan uses bounded source
folders, so it avoids caches and large data/results trees without excluding
tooling/data. PR CI runs the module gate and Python architecture check in the
Python lint lane.
Module reachability does not prove that every function, method or branch is live. Symbol candidates from Skylos still need callsite review for Rust/PyO3 registrations, JAX callbacks and CUDA FFI, and Rust dependencies need the maintained Cargo checks. An unfamiliar dynamic import needs an explicit, reviewable root or recognized edge rather than a blanket exclusion.
just check-justfile verifies that the command surface stays config-backed and
that maintained docs do not reference removed recipe names. These guardrails
are included in just check and just check-local; just rust-check and
just workspace-check keep a Rust-focused validation lane for multicrate
migration phases.