Performance Guide
| Status | Applies to | Owner |
|---|---|---|
| Pre-release draft; guidance, not a benchmark guarantee | CPU and GPU Step 2 implementation as of 2026-10-06 | Public user docs |
Performance depends on genotype format, trait mode, phenotype count, BGEN decode cost, host-device transfer, JAX compilation, Parquet writing, storage, and cluster placement. Treat numbers from one machine as local evidence, not a portable guarantee.
What To Check First
- Confirm you are running on the intended node.
- Confirm JAX sees the intended device:
uv run python -c "import jax; print(jax.devices())"
- Confirm the statistical modes are comparable. Compare score-only to score-only, and approximate Firth to approximate Firth with the same threshold.
- Check whether the run is dominated by BGEN decode, host-device transfer, JAX compute, or output writing before changing knobs.
- Use telemetry/profile output from the same command shape you intend to tune.
Common Bottlenecks
| Symptom | Likely first checks |
|---|---|
| GPU visible but little speedup | Single phenotype, small chunks, BGEN decode, host-device transfer, or output may dominate. |
| First run slow but repeated work faster | JAX backend initialization and compilation are likely visible. |
| Approximate Firth much slower than binary score test | Candidate density from --pThresh and Firth solver work dominate. |
| Output stage slow | Storage throughput and writer threads. |
| Resume startup slow | Manifest validation and strict chunk reconciliation. |
Binary score-only runs prepare only the null-logistic and score state. They do not pay the chromosome-level null-Firth fit, full-null deviance, or Firth-state memory cost; those operations are exclusive to approximate-Firth runs.
Quantitative kernels reduce squared shifted dosages on the selected device to preserve rare-allele variance in large cohorts. This adds reduction work to the native-summary path; raw moments alone cannot safely reconstruct that variance. Unshifted variants continue to use the supplied native sums of squares.
Covariate preparation uses float64 conditioning and QR once per aligned
sample group. Quantitative preparation keeps its covariate basis and phenotype
residuals in float64 until chromosome residual projection; genotype chunks
and score operands remain float32. Compared with storing those shared arrays
in float32, persistent storage grows by approximately
4 × samples × (covariate parameters + phenotypes) bytes: about 60 MB for
500,000 samples, 20 parameters, and 10 phenotypes. Setup also needs temporary
conditioning arrays. Binary preparation stores its final conditioned design
in float32 and pays the QR cost during group preparation.
Its chromosome-level null fit uses float64 safeguarded Newton arithmetic;
per-variant score computation remains float32.
Group conditioning, QR, rank evaluation, and quantitative residual preparation
run as compiled programs. Groups with matching sample, covariate, and phenotype
counts can reuse those programs. A new shape still pays preparation compilation;
rank validation and binary intercept validation retain their existing errors.
Binary scoring also checks whether dosages vary across samples. The decoded
path carries that check through its existing sample tiles; the materialized
path performs a per-variant comparison reduction.
For packed8 binary input, approximate Firth retains the packed chunk and decodes only selected correction rows. It avoids keeping a full float32 dosage matrix alive through scoring and correction; correction capacity and sample count still determine the selected matrix size. The float32 decoding boundary and statistical formulas are preserved. On the large synthetic fixture in the profiling review, process- lifetime peak live JAX allocation fell by 27%; allocator pool reservation did not fall. This is a workload-specific memory result, not a speed guarantee.
CUDA Firth batches exclude inactive lanes and successful pseudo-stage lanes from subsequent solver work. Benefits depend on candidate density and fallback frequency. Small candidate sets can add overhead; CPU components keep their original validity handling.
Cold, Warm, And Hot Runs
Do not compare timing modes as if they measured the same thing:
| Mode | Includes |
|---|---|
| Cold process | Python process startup, JAX backend initialization, first compilation, BGEN decode, compute, and output. |
| Warm JAX cache | New process startup plus cache lookup/reuse when persistent cache entries match. |
| Hot same process | Already-live Python process, JAX backend, and compiled functions for compatible shapes. |
Use cold-process timing for batch-job wall-clock expectations. Use warm-cache or hot same-process timing only when that is the workflow being measured.
Fresh GPU compilation can choose different vendor matrix kernels, changing low float32 bits as well as runtime even with identical source and inputs. Record software versions, device details, and compilation-cache provenance when comparing results. The profiling review includes controlled replay that separates this variation from a decoding change.
The October architecture review measured 3.69–6.50% lower warmed full-scan time on chromosome 22 workloads with distinct phenotype sample masks. Those comparisons include output writing and match every output bit. Independent binary compilations still show the vendor kernel variation above; these measurements do not establish a binary scan speedup or a guarantee for other cohorts or hardware.
Explicit resumes that have no pending output still perform input and committed- output validation, then finish without initializing JAX or a device backend. In the same review's completed-run comparison, median fresh harness-process time fell from 13.49 s to 1.08 s. This measures completed resumes with warm filesystem caches; unfinished scans still need their normal compute runtime.
For several compatible production scans, use the native batch command to make the hot same-process path explicit:
uv run --no-sync g batch \
--config chromosome_21.toml \
--config chromosome_22.toml
Batch mode constructs every frontend config and checks output-root and process-policy compatibility before starting. Run-owned input, output, and resume preflight remains per entry. The process reuses JAX/CUDA state, compiled executables, and one verified BGEN index. Shape changes may compile an additional executable, so group configs with stable shapes when throughput is the priority.
Approximate-Firth chunk executables specialize on correction settings and prepared chromosome operands. Changing only the null-logistic or null-Firth fitting policy can rebuild the chromosome state without compiling another otherwise identical chunk executable. Changing correction solver settings or candidate geometry can still compile a separate executable. This reduces repeated setup work across compatible scans; it does not make an already compiled chunk kernel faster.
The process-local BGEN cache retains at most one index with an estimated allocation size of 256 MiB or less, plus one open file descriptor. It retains no mapping or genotype payload. Larger indexes are usable but are not cached. Each open still validates the source and creates its own mapping; matching device, inode, size, modification time, and change time permit index reuse, including through path aliases. An index remains reusable after the previous reader closes. This helps repeated scans within one process, not independent CLI processes. Allocator bookkeeping and retained free pages can make process RSS larger than the cache's allocation budget. Before admitting an index, the cache observes an unchanged physical identity for at least two seconds using a monotonic clock, then reparses the source. This avoids stale reuse when filesystem timestamps have only second precision. The initial opens can therefore still pay indexing cost; admission never waits or blocks a run for that interval.
In the September 2026 measurements, repeated hot binary scans took 11.8% less time on the chromosome-22 fixture. Fresh-process runs did not improve, and an uncached BGEN open cost about 3 ms more. These results describe that workload and cache state, not a general speed guarantee.
Native decode buffers submitted to JAX transfer their allocation into NumPy; there is no full genotype memcpy at the binding boundary. Phenotype, covariate, and single-use LOCO matrices also transfer their allocations directly into NumPy. Native input indexes each selected LOCO file once using byte offsets and line numbers. Files with identical headers share one loader-side identifier index and one alignment recipe per group; those large header strings are discarded after source construction. Resume planning validates only chromosome blocks that still need output. When execution reaches one of those blocks, input reads, parses, finite-validates, and aligns just that chromosome into its final trait-major matrix. File metadata snapshots and raw-row SHA-256 digests reject changes between indexing and deferred reading. Indexing also computes the exact whole-file SHA-256 used by output fingerprints, avoiding a second complete read. Output verifies and pins that indexed source identity; observable identity changes cause an error. The digest describes the bytes read during indexing. Inputs must remain unchanged during a run: metadata checks cannot detect every concurrent edit on a filesystem with coarse timestamps, while deferred row digests protect the prediction values actually read. Persisted fingerprint values and resume compatibility are unchanged for unchanged inputs. Fully committed chromosomes therefore never allocate or parse prediction values. A repeated noncontiguous chromosome block alone keeps one matrix for safe clones until its final planned use.
For each complete JAX input shape, dosage and packed8 delivery keep the variant
dimension fixed across chunks. A short chromosome-final chunk is padded only
after exact BGEN decode to min(bsize, input variant count); metadata,
statistics written to output, and result rows retain the logical variant count.
Chunk grids restart at each chromosome, avoiding the extra boundary fragment
created by a whole-file grid. This trades bounded tail transfer and compute for
reuse of that shape's executable. Different aligned sample counts or trait
counts produce different complete JAX shapes and may compile separate
executables.
The native association scheduler starts one compute thread, one host-result materialization thread, and one bounded channel set for each direct delivery. At a chromosome boundary, all results drain and the compute worker acknowledges destruction of the replaced JAX state before its successor is built, avoiding both per-chromosome worker churn and overlapping chromosome-state device memory. Group-level device state is created at first use and released after its final chromosome preparation. Fully resumed phenotype groups initialize progress but do not select BGEN samples, prepare JAX state, or start scheduler workers. Eligible linear GPU groups can also use the bounded source-sharing strategy below.
Within a run, successive groups reuse the compressed packed8 layout plan when their ordered pending chunk ranges match exactly. Different sample masks can share that geometry, while different resume subsets replace the one cached plan. This plan cache retains only chunk metadata; statistical state remains specific to each aligned sample group.
For compatible compressed packed8 linear GPU input, the engine can process two sample groups together. It decodes a shared pending chunk once with all source samples, then selects each group's exact samples and computes its own statistics. Only one source chunk and two groups' states are retained. Each active group submits one private selection, then both results drain before the next source chunk. This permits overlap between one group's association and the other group's selection and materialization. Each output keeps its original variant order and resume coverage, and the output writer-pool size is unchanged.
This internal strategy admits at most 128 MiB of retained source pairs and statuses, and 64 MiB of combined linear group, chromosome, and sample-selection array payloads. The two groups' selected inputs and association results can overlap beyond these limits. Preparation temporaries, compiled workspaces, and allocator reserves are also excluded; these are not total GPU-memory limits. Singleton groups, unsupported modes or input, disjoint pending plans, and geometries above either budget use ordinary delivery. The extra selection work and scheduling barriers make the benefit workload-dependent.
In the 2,504-source-sample chromosome-22 qualification, hot scans with eight to 32 distinct masks took 9.5–10.7% less time. Peak live JAX allocation increased by about 75–153 MiB, and the retained allocator pool grew from 522 to 1,034 MiB. Those allocator counters exclude native allocations outside JAX and do not measure total GPU memory. See the profiling review for the workloads, controls, and measurement limits.
Runtime Knobs
Use the current packaged defaults first. Override only with measurements.
| Option | Main effect |
|---|---|
--bsize |
Variants per chunk; affects memory, JAX shapes, compilation, and per-chunk overhead. |
[compute] device |
JAX execution target, cpu or gpu. |
[compute] cpu_threads |
Native Rayon worker count for Rust-owned work. |
[compute] multi_phenotype_sample_mode |
Per-phenotype or shared complete-case sample alignment. |
[output] writer_threads |
Output writer worker count. |
[compute] firth_batch_size |
Approximate-Firth candidate batch size. |
[compute] firth_candidate_capacity |
Per-trait scaling value for the aggregate hard approximate-Firth capacity; larger values increase the static executable shape. |
[compute] jax_cache_dir |
Optional location override for the always-enabled persistent JAX compilation cache. |
[diagnostics] telemetry |
off, progress, or profile; profile mode can perturb timing. |
The BGEN decode tile and GPU genotype representation are internal runtime policies, not configuration keys. The current reader uses 32-variant decode tiles. Linear and binary GPU runs use packed8 delivery when no-missing diploid compatibility validation passes, including multi-phenotype groups. CPU and otherwise-supported biallelic diploid Layout-2 inputs that are not packed8-compatible use dosage delivery. Multiallelic, non-diploid, or otherwise unsupported input fails instead.
Approximate Firth uses an aggregate hard static JAX capacity of the configured
value, capped by the static compute chunk width, multiplied by the trait count.
The packaged per-trait scaling value is 1,024. A batch with more aggregate
candidates is not truncated: it finishes the normal device-to-host result
synchronization and then fails with an instruction to increase
[compute] firth_candidate_capacity.
For compatible zlib-compressed packed8 input, GPU runs transfer aligned raw DEFLATE members and decode them on the active device with nvCOMP. Selection, exact integer summaries, and packed probability-pair construction stay on the device; association kernels consume that representation without first materializing a host dosage matrix. CPU runs, non-zlib BGEN encodings, and inputs requiring dosage retain the native host decoder. The nvCOMP library and private JAX FFI target are initialized lazily only when a compressed packed8 group is prepared, so those fallback paths do not pay its initialization cost.
Current default values are in crates/interface/src/config.default.toml.
Older pre-release configurations may contain
firth_newton_raphson_zero_start_iterations. Remove that key when upgrading;
the production scalar solver always starts at zero and the separate setting was
unreachable, so it is no longer accepted.
The V100-tuned binary path uses 512-lane Firth batches and a
1,024-candidate-per-trait middle dispatch tier. Larger candidate sets retain
the full-chunk overflow path.
Larger batches can reduce some separate-process cache-load overhead while
increasing device execution time, so compare complete workflows rather than
choosing from compile or kernel timing alone.
Project profiling recipes isolate CPU JAX caches by host and CPU feature
fingerprint under /tmp/g-jax-cpu-profile-cache by default. This avoids reusing
CPU AOT artifacts across SLURM nodes when profile outputs are stored on shared
filesystems. GPU profile caches remain node-local under /tmp/g-jax-profile-cache
or /tmp/g-jax-binary-hot-cache unless G_PROFILE_GPU_JAX_CACHE_PARENT is
overridden.
CPU Runs
CPU runs exercise native BGEN decode, sample alignment, output writing, and JAX
CPU kernels. For large scans on a cluster, submit to a compute node instead of a
login node and set [compute].cpu_threads to the scheduler CPU count.
uv run --no-sync g regenie \
--qt \
--bgen /path/to/genotypes.bgen \
--sample /path/to/genotypes.sample \
--phenoFile /path/to/phenotypes.tsv \
--phenoCol phenotype_continuous \
--pred /path/to/regenie_step1_qt_pred.list \
--out /path/to/output/g_cpu_regenie2
GPU Runs
GPU runs require a CUDA-capable JAX environment and a GPU allocation. Single phenotype scans can still be limited by BGEN decode, transfer, and output rather than accelerator compute.
Useful GPU checks:
uv run python -c "import jax; print(jax.devices())"
uv run --no-sync g regenie --config gpu.toml ...
Multi-phenotype quantitative runs can amortize BGEN decode and process startup
when sample semantics allow it. Use per-phenotype for separate-run semantics
and complete-case only when the shared complete-case intersection is intended.
As a practical rule of thumb:
per-phenotypemaximizes per-trait usable samples and is the conservative choice when missingness differs by phenotype.complete-casetends to reduce repeated alignment work because one shared mask is used across traits, but it can reduce sample size for each trait and change statistics when missingness differs.
If speed is the only concern, test both modes on a small representative subset before changing production scripts.
Parquet Output
Each phenotype run writes a parts/ Parquet dataset. writer_threads is the
public filesystem-throughput control. Queue depth, part grouping, and
compression are internal output policies.
Measuring
Production mode:
[diagnostics]
telemetry = "progress"
Profile mode:
[diagnostics]
telemetry = "profile"
Development benchmark protocols and repository-specific SLURM recipes live in Benchmarking and Server Gauss SLURM. The September 25 profiling review records the latest measured optimizations, workload scope, and limitations.