Architecture
| Status | Applies to | Owner |
|---|---|---|
| Active production architecture | Production code as of 2026-10-06 | Development maintainers |
g is a Rust host application with a Python/JAX numerical backend. Rust owns
configuration, planning, input, scheduling, lifecycle, telemetry, shutdown,
resume, and output. Python owns JAX arrays and statistical kernels.
Production Flow
g.cli (argument/output bootstrap only)
|
g._core.cli.run(arguments)
|
g-runner::run_cli
|
g-interface -> g-plan -> g-engine::execute_coordinated_run
| |
| +-> g-genotype + g-input
| +-> g-genotype-contracts
| +-> g-runtime + g-output
v
PreparedRun::execute_with_progress<AssociationBackend, RunHooks> + bounded pipeline
|
batch-oriented PyO3 adapter
|
g.jax_backend -> JAX kernels
The backend operations are prepare_group, prepare_chromosome,
transfer_batch, transfer_compressed_batch, prepare_shared_source,
select_shared_source, compute_batch, and materialize_batch. Delivery
capability selects host-decoded, compressed, or shared-source transfer. These
operations retain opaque JAX state while Rust owns scheduling and lifetimes.
Materialization performs one batched device-to-host transfer and returns typed
arrays to Rust.
Ownership
| Area | Owner |
|---|---|
| CLI parsing, TOML, defaults, validation | g-interface |
| Immutable run contracts and host policy | g-plan |
| Shared variant metadata and output-facing genotype columns | g-genotype-contracts |
| BGEN mmap/index, immutable read sessions, delivery planning, decode/preprocessing, and owned decoded or compressed batches/buffers | g-genotype |
| Optional capability-gated packed8 raw-DEFLATE delivery and typed-XLA FFI handlers | g-genotype-cuda |
| Optional capability-gated CUDA association kernels and typed-XLA FFI handlers | g-compute-cuda |
| Private header-only CUDA driver ABI, loading, and device/context validation | native/cuda-driver |
| Sample, phenotype, covariate, and prediction alignment | g-input |
| BGEN delivery orchestration, result writing, backend trait, bounded compute/materialize pipeline | g-engine |
| Logging/telemetry transport, timing, Rayon state, SIGTERM | g-runtime |
| CLI lifecycle, process/JAX policy, crate orchestration, terminal rendering | g-runner |
| Parquet writers, manifests, and resume | g-output |
| PyO3 object construction, opaque Python state, NumPy conversion, PyErr adaptation, JAX process calls | root Rust extension under src/binding |
| Device state, compressed-device decode invocation, and association mathematics | src/g/jax_backend.py, private helpers in src/g/backend/, src/g/compute/, and capability-gated kernels in g-compute-cuda |
g-runner is the root dependency through which the binding reaches CLI,
planning, runtime, and execution services. It owns CLI dispatch, resolves the
application plan into concrete logging, telemetry, and timing resources,
process-global runtime compatibility, terminal rendering, and JAX setup before
the coarse engine invocation. g-runtime consumes only that generic resolved
session policy; it does not import or interpret g-plan. RunEngine owns the shared plan
and output manager; PreparedRun owns aligned groups, BGEN state, resume state,
bounded scheduling, move-only backend buffers, delivery, interruption, abort,
and output completion. Input retains shared
LOCO row indexes and compact alignment recipes, then materializes only the next
post-resume chromosome. Group, genotype, and single-use LOCO matrices move
directly into NumPy ownership. The binding
retains only calls that attach to Python, hold opaque
JAX objects, preserve a concrete PyErr, or label telemetry with the current
Python thread. It imports canonical genotype, input, and output payload types
only to convert the private AssociationBackend boundary to and from NumPy;
all services remain behind g-engine and g-runner.
Backend errors retain their concrete host type through delivery and scheduling.
RunHooks asks the host whether such an error represents an interruption before
the engine chooses graceful flushing or failure abort. The Python adapter
recognizes signal exceptions by type, preserves their identity, and leaves
interrupted lazy backend initialization retryable. It does not classify
signals by matching exception text.
Engine resolves the prediction list once into an input-owned path catalog. Input indexing/alignment and output-manifest fingerprinting borrow that same catalog, so group preparation does not reparse the list or duplicate its path strings.
Input computes each LOCO source's complete raw-byte SHA-256 during row indexing and exposes an opaque, metadata-verified fingerprint snapshot. Engine registers those snapshots with output before building manifest fingerprints. Output pins each registered configured path to its indexed physical identity and fails on observable changes. The digest describes the indexing read; coarse filesystem timestamps can hide concurrent edits from metadata checks, while deferred row hashes still verify consumed prediction bytes. Inputs must remain unchanged during execution. The persisted manifest schema and digest recipe are unchanged.
Genotype owns a process-local, single-entry BGEN index cache. It retains shared immutable index arrays and an inode-pinning descriptor, with a 256 MiB estimated index-allocation limit. Admission requires a monotonic stability interval and a subsequent fresh parse to handle coarse filesystem timestamps. Every reader still owns a fresh validated mapping. Engine separately keeps one compressed-layout plan per prepared input, keyed by exact ordered pending chunk ranges. Neither cache retains decoded genotypes, sample masks, LOCO matrices, or device state.
Genotype, engine, and output import one canonical set of data-plane column
contracts from g-genotype-contracts. Genotype retains dictionary-coded
metadata through a shared store and output builds Arrow metadata lazily, so the
crate boundary adds no row copies, eager arrays, or adapter DTOs.
Each tiled delivery clones one immutable metadata owner for its active groups.
They share UTF-8 validation and lazy Arrow arrays; their numerical statistics
remain independently owned.
The backend advertises a genotype-delivery capability to g-engine.
g-genotype chooses either an owned host-decoded batch or a canonical
compressed packed8 batch; g-engine schedules that typed payload without
knowing Python or CUDA details. The root backend only lends the crate-owned
slab and metadata to NumPy and calls the private JAX transfer method. The CUDA
FFI handler and all raw-DEFLATE validation/finalization live in
g-genotype-cuda; binding code performs only lazy library loading and target
registration required by PyO3/JAX.
Binary approximate-Firth GPU runs independently probe g-compute-cuda and
register its private component-reduction target once. Compatible runs use the
embedded PTX through typed XLA FFI; an unavailable driver, unsupported device,
or registration failure selects the mathematically equivalent JAX reduction.
This optional compute capability is not coupled to packed8 delivery or nvCOMP.
The two CUDA crates have independent Rust APIs and PTX/kernel ownership. They
source-include only the neutral repository-private native/cuda-driver support
header so CUDA ABI declarations, dynamic loading, and device/context checks
have one maintained implementation without a linked support library.
Runner holds process runtime state across logging-compatibility validation, native session construction, subscriber installation, and topology recording. An incompatible repeated run therefore cannot open a log/telemetry file or start an asynchronous writer before it is rejected.
g-runner::run_cli is the coarse native coordinator used by the binding. It
owns the call to g-engine::execute_coordinated_run, which in turn owns
preparation, execution, delivery-report handling, completed-artifact
construction, and writer-completion telemetry. The binding does not
orchestrate domain crate calls.
Python Surface
Production Python outside the kernels is limited to:
src/g/cli.py console bootstrap and output forwarding
src/g/jax_backend.py mode-specific JAX backends
src/g/backend/ private transport, array contracts, and materialization
src/g/compute/ JAX kernel state and mathematics
Python does not parse files, align samples, schedule workers, write results, manage manifests/resume, select cleanup policy, or own telemetry lifecycle.
Invariants
- Domain crates remain free of PyO3.
- The JAX boundary is batch-oriented; there are no per-variant Python calls.
- Rust owns all bounded queues, worker joins, host buffers, and output order; backend-bound genotype allocations move into NumPy without a second matrix. Materialized result arrays are copied into owned Rust buffers before asynchronous output; JAX/NumPy allocator ownership is not transferred.
- Planning enums have one definition in
g-plan. Only domain owners interpret them; infrastructure crates such asg-runtimereceive projected generic policy. - Resume commit sets and the immutable
RunPlanuse shared ownership rather than per-group clones. - CLI validation and help complete before importing JAX-heavy modules.
- Binary score-only execution does not construct Firth policy, fit a null-Firth model, compute full-null deviance, or retain Firth-only chromosome arrays. Approximate-Firth state composes the compact score state when correction is enabled.
- First SIGINT/SIGTERM preserves resumable output; a second SIGTERM uses the operating system default action.
- Unsupported REGENIE options are rejected, not silently adapted.
- Rust host scientific buffers, association scores, and persisted public
statistics are
f32. Approximate-Firth state, reductions, objective components, and validation aref64; its inner pseudo-logistic elementwise proposal isf32. - Production exports are limited to
g._core.cli.
Detailed contracts: