Development Tooling Guide
| Status | Applies to | Owner |
|---|---|---|
| Pre-release draft | current repository-local tooling | Development tooling maintainers |
This page is the operational reference for the development-only tooling/
package. The shorter design summary is in
Development Tooling Architecture.
The experimental All of Us Workbench workflow adds
tooling.workbench for input inspection, pinned-file localization, resource
planning and validated subprocess execution. It invokes the existing Step 2
CLI and preserves the engine's scientific and resume contracts. Install the
minimal workbench dependency group when using this tooling outside the
development environment. It is copied into the Workbench image, rather than
exported through the public g package.
Scope
tooling/ owns repeatable development workflows: data preparation, benchmark
campaigns, profiler orchestration, machine profiles, report generation, and
repository guardrails. It is not installed as part of the public Python
package.
Production behavior remains in src/g and crates/. Tooling invokes the same
supported boundary as users: g regenie --config <file> in a subprocess or
g._core.cli.run(["regenie", "--config", ...]) in a same-process lifecycle.
It must not recreate a second Python orchestration layer.
Generated data and profiler output belong under ignored locations such as
data/benchmarks/, data/profiles/, and results/perf/.
Repository-wide Ruff checks exclude .git/. Git metadata, including reflogs
for branch names ending in .py, is outside the Python source scope.
Layout
tooling/
benchmark/
benchmark.py
linear_startup.py
native_lifecycle.py
cli/
benchmark.py
benchmark_firth_compute.py
benchmark_regenie2_binary_hot.py
benchmark_tensorqtl_chr22.py
benchmark_torchgwas_chr22.py
data.py
debug.py
performance.py
profile_regenie2_deep.py
run_regenie2_matrix.py
rust_build_profiles.py
schema_check.py
server.py
common/
artifact_format.py
commands.py
downloads.py
g_regenie.py
hydra_arguments.py
jax_cache.py
paths.py
registry.py
reports.py
configs/
dataset/
machine/
telemetry/
workload/
*.yaml
profile_deep/
data/
debug/
performance/
server/
tooling.common.g_regenie is the single tooling-owned renderer for production
TOML and g regenie --config commands. tooling.benchmark.native_lifecycle
owns fresh-process, discarded-warm, hot same-process, cache, telemetry, and
completed-output evidence shared by the lifecycle benchmarks.
Configuration
All maintained tooling entrypoints are Hydra-driven. Each Justfile workflow
selects an explicit config with --config-name; files under
tooling/configs/ own datasets, machines, workload geometry, profiler modes,
and output policy.
Inspect a composed configuration without executing it:
uv run --no-sync python -m tooling.cli.benchmark_regenie2_binary_hot \
--config-name bench_binary_hot_gpu --cfg job
Apply narrow one-off overrides after the config name:
just slurm-gpu-just bench-binary-hot-gpu tool.hot_run_count=7
just matrix-chr22-dry tool.output_dir=data/benchmarks/chr22-plan
just profile-chr22-binary-gpu-dry tool.output_dir=data/profiles/chr22-plan
Every config keeps Hydra's working-directory change disabled:
hydra:
job:
chdir: false
There is no production variant-count cap. A bounded benchmark must use a
physically bounded BGEN fixture with matching metadata and prediction inputs.
The optional regenie_baseline_variant_limit field is limited to the external
REGENIE baseline and creates an extract list; it never changes a g run.
Lifecycle Benchmarks
Binary hot path
Run the production binary approximate-Firth geometry with:
just slurm-gpu-bench-binary-hot
The default chr22 config uses packed8 GPU delivery, 16,384-variant chunks, Firth batch size 512, candidate capacity 1,024, eight output writers, direct Parquet parts, and telemetry off.
The harness records:
- an optional fresh-process diagnostic;
- one discarded same-process compile/warm lifecycle;
- the requested number of telemetry-off same-process hot lifecycles, each with a distinct output root;
- optional profile-telemetry diagnostics kept out of headline timings.
It rejects pre-populated default cache directories, requires the warm lifecycle to populate the cache, and requires the cache tree to remain byte-identical during headline runs. Evidence includes the native-library and dependency-lock hashes, input fingerprints, runtime and machine details, per-run elapsed time, cache snapshots, manifest coverage, Parquet row counts, schemas, metadata, and part hashes.
For a quick harness check on a GPU node:
just slurm-gpu-just bench-binary-hot-gpu-smoke
Quantitative lifecycle
bench-linear-startup-gpu exercises the same current production boundary for
quantitative Step 2. It has the same fresh-process, discarded-warm,
same-process-hot, and optional diagnostic separation as the binary harness.
just slurm-gpu-just bench-linear-startup-gpu
Focused Firth compute
The focused JAX benchmark compiles one fixed-capacity approximate-Firth executable and measures synchronized hot calls at representative active candidate counts:
just slurm-gpu-bench-firth-compute
It records StableHLO and executable fingerprints, compile/cache evidence, device timing, and correctness hashes. It is a causal compute benchmark, not a whole-application speed claim.
CPU, GPU, and Cache Matrices
The matrix tool runs binary and quantitative Step 2 on CPU, GPU, and a repeated GPU process using the populated persistent cache:
just matrix-chr10-dry
just slurm-gpu-matrix-chr10
just matrix-chr22-dry
just slurm-gpu-matrix-chr22
Dry configs render and validate plans without running the workload. Full runs write independent production TOML files and output roots. A GPU matrix starts with an empty campaign-owned cache; its first GPU process populates that cache, and the repeated process must leave it byte-identical.
Previous manifests are comparable when their input/workload identity, cache policy, machine/runtime versions, Rust flags, and normalized runner protocol match. Git, native-library, source, dependency-lock, and raw-runner hashes are reported separately as implementation provenance. A provenance change is expected during development and is not itself a workload mismatch.
Competitor Benchmarks
Use the saved chr22 competitor workflows through SLURM:
just slurm-gpu-bench-torchgwas-chr22
just slurm-gpu-bench-tensorqtl-chr22
These are workflow/runtime comparisons, not statistical-parity claims.
TorchGWAS uses the local PLINK triplet for full runs. That path does not create a persistent genotype cache, so its cases are named first-process and repeat-process; the repeat may benefit from the filesystem cache but is not a declared application-cache hit.
tensorQTL runs dense nominal trans association on QTL-shaped phenotype and
covariate matrices. It also reports first-process and repeat-process cases
without claiming a persistent application cache. Its statistical model differs
from REGENIE Step 2 with LOCO predictions.
Native Boundary Benchmarks
Run focused Rust benchmarks by explicit package and bench name:
cargo bench --package g-genotype --bench bgen_read
cargo bench --package g-engine --bench scheduler
cargo bench --package g-output --bench writer
The repository profiling recipe intentionally runs the genotype target alone on the CPU node:
just profile-rust-criterion
Criterion is not embedded in the GPU deep-profiler campaign. Add a new direct recipe only when a real crate bench exists and its node requirements are clear.
Deep Profiling
Start by rendering a plan:
just profile-app-full-dry
just profile-chr10-binary-gpu-dry
just profile-chr22-binary-gpu-dry
Run retained campaigns with:
just slurm-gpu-just profile-app-full-smoke
just profile-app-full
just profile-chr10-binary-gpu-full
just profile-chr22-binary-gpu-full
The focused full configs can collect JAX trace/device memory, cProfile, py-spy, Scalene, Memray, Nsight Systems, Nsight Compute, and Linux perf when the tools and node permissions allow them. Missing or permission-blocked optional profilers are reported as skipped.
JAX trace capture warms up one complete application run in the same process
before starting the trace, using a distinct .trace_warmup.g output root.
Python call tracing is disabled so device work and host dispatch remain
inspectable without filling the trace with compilation and import calls.
The trace directory's capture_scope.json records this scope, warmup time,
output root, and JAX version. These traces describe a warm application run;
fresh-process initialization remains covered by the separate stage diagnostic
and startup-oriented profilers. External profiler success also requires valid
completed application output. An interrupted sampler's partial trace is retained
but is not reported as a successful application profile.
Headline candidate and finalist trials use production telemetry-off execution. Exact stage timing is a separate diagnostic lifecycle and is excluded from headline elapsed times. Current native profile telemetry must contain:
jax_runtime_configuration;jax_backend_initialization;native_run_preparation;native_run_execution;runner_total.
Rust-owned output timing artifacts are preserved independently and may contain stages such as enqueue, record-batch creation, Parquet writing, manifest commit, and writer finish. They must not be relabeled as Python callback or JAX compute attribution.
The main outputs include preflight.json, summary.json, summary.md,
artifact_manifest.json, per-run logs, generated trial configs, headline
outputs, and profiler-specific artifacts. Diagnostic output roots are distinct
from headline roots.
Use the public tuning dimensions exposed by the saved configs:
tool.workload_keys;tool.chunk_sizes;tool.output_writer_thread_counts;tool.rayon_thread_counts;tool.firth_batch_sizes;- profiler enable/timeout fields;
- trial and campaign budget fields.
Removed callback, staging, decode-tile, queue-depth, alternate-output, and post-run-finalization controls are not valid tuning dimensions.
Telemetry and Output Evidence
Headline production runs use telemetry = "off". Profile diagnostics require
native profile summary evidence and Rust output-stage timing evidence. Progress
mode requires progress events. A successful output is accepted only when:
- the manifest reports success and a valid integer variant count;
- committed chunks are ordered, non-overlapping, gap-free, and exactly cover
[0, variant_count); - part paths are safe and exactly match observed Parquet files;
- Parquet parts are readable and their row counts sum to the manifest count;
- schema and metadata are consistent across parts.
The tooling records content and manifest hashes in addition to counts so a timing result remains auditable.
Artifacts and Comparison
Durable tooling bundles use the repository's shared artifact-format models and include schema identification, producer and git state, command records, input fingerprints, metrics, failures, comparisons, and an artifact manifest. Check a completed bundle with:
just check-artifact-schema data/profiles/example
These internal evidence schemas retain their own existing versions for reader compatibility. They are separate from public product option, output-manifest, telemetry, and profile-summary contracts, which remain at version 0 until the first release.
Do not compare timings solely because two manifests share a filename. Confirm the compatibility identity first, then retain both implementation provenance records in the report.
SLURM and Node Policy
Dry plans, small report transformations, and docs checks are login-node-safe.
Use cantor or another exclusive CPU allocation for compilation, full tests,
Criterion, and CPU benchmarks. Serialize GPU work on landau:
just slurm-cpu-just rust-check
just slurm-gpu-just bench-binary-hot-gpu-smoke
just slurm-gpu-matrix-chr22
The SLURM wrappers configure the build job count from the allocation and keep node-specific Cargo artifacts isolated. Do not overlap independent benchmark or profiler jobs on the single GPU.
Adding or Changing Tooling
When adding a maintained workflow:
- Put workflow truth in an entrypoint-specific saved Hydra config.
- Parse it into a fully typed arguments dataclass.
- Reuse
g_regenie,native_lifecycle, command, cache, and report helpers rather than duplicating production invocation or evidence validation. - Add only a thin, named Justfile recipe with
--config-name. - Add focused tests for parsing, generated production TOML/commands, failure behavior, cache invariants, and artifact contracts.
- Run
just check-justfileandjust docs-buildwhen the command surface or this guide changes.
Avoid compatibility aliases for deleted scripts or configuration fields. A bounded workflow should get a bounded fixture, and an intrusive profiler should get a distinct diagnostic run rather than changing headline semantics.