Skip to content

Resume And Manifest

Status Applies to Owner
Pre-release draft Step 2 resume and manifest behavior as of 2026-09-24 Public user docs

This page is the canonical user-facing reference for resumable output runs.

For output layout and schemas, see Output Files. For config merge behavior, see Configuration.

Per-Phenotype Metadata

Every phenotype output run writes:

effective_config.toml
run_manifest.json

effective_config.toml is the final merged config after packaged defaults, the optional TOML file, and explicit CLI overrides.

run_manifest.json records execution-plan-affecting state, including:

  • manifest and output schema versions;
  • association mode;
  • BGEN, sample, phenotype, covariate, prediction-list, and selected LOCO prediction-file fingerprints;
  • phenotype name, covariate names, sample count, variant count, and chunk size;
  • multi-phenotype sample mode, phenotype compute-group identifier, sample-set fingerprint, covariate-design fingerprint, aligned phenotype-design fingerprint, and prediction-alignment fingerprint;
  • selected association backend such as jax_dosage or jax_packed8, with the resolved genotype delivery format;
  • binary correction plan and binary kernel settings when applicable;
  • quantitative variance thresholds when applicable;
  • JAX device/precision policy and dtype choices;
  • genotype-moment accumulation, covariate projection, and binary null-solver numerical policies;
  • output writer settings;
  • committed chunk identifiers and Parquet part metadata.

Pre-release manifest schema version 0 stores immutable compatibility state once under execution_plan, with its SHA-256 digest in execution_plan_hash. Top-level fields are limited to manifest/output schema versions and mutable lifecycle metadata such as status, committed chunks, command, runtime, and interruption state. The pre-release Parquet output schema is version 0; it makes INFO nullable when its expected-variance denominator is undefined.

The manifest is the resume authority. It is intentionally stricter than a file name check.

Sample identity is always the non-empty (FID, IID) pair, so there is no identity-mode field in the execution plan. Sample-file fingerprints and aligned sample-set fingerprints cover the concrete identity data used by the run. Prediction-alignment fingerprints bind the LOCO headers and chromosome rows seen during indexing plus the concrete per-trait sample-alignment recipe.

File fingerprints include resolved path, file size, and mtime_ns. Smaller control files also include a SHA-256 content hash: sample, phenotype, covariate, prediction-list, and LOCO prediction files referenced by the selected phenotype or compute group. BGEN identity comes from the exact file opened by the native reader and includes device, inode, size, modification time, and change time. This rejects replacement or in-place mutation without hashing the large genotype file during normal startup.

Starting A New Run

Without [output].resume = true, g refuses to reuse a non-empty output run directory:

Output run directory '<path>' already exists and is not empty. Enable [output].resume or choose a new output path.

Choose a new --out prefix, delete stale local output intentionally, or run with [output].resume = true when the existing manifest belongs to the same planned run.

Planning a new or resumed run only inspects the selected output paths. g does not create phenotype run directories until input preparation and validation of every selected phenotype output have succeeded. Missing paths are treated as absent; other directory or manifest inspection errors stop planning.

Enabling Resume

[output]
resume = true

Resume is always strict: after manifest compatibility passes, g reconciles committed chunk identifiers with the chunk files on disk before continuing. There is no public resume-validation mode.

If every chunk is already committed for every selected phenotype, the validated resume completes without importing or configuring JAX, discovering devices, or constructing an association backend. Input alignment, execution-plan and source fingerprints, requested device policy, and every Parquet part still pass the usual validation. Process-global logging, thread, and already configured JAX policy compatibility checks also apply. A resume with pending chunks initializes the requested backend and computes those chunks normally.

This reduces startup work for a completed resume. Fresh analyses and partial resumes still initialize their compute runtime. Profile telemetry for a completed resume has native preparation/execution timings and no JAX setup or backend initialization stage.

This deferred initialization applies only when resume = true. Fresh analyses initialize JAX before preparing output manifests, preserving the ability to fix a device setup error and retry without an initialized output directory. During a resume, an initialization failure aborts active writers while preserving existing committed output; an interruption flushes writers and records its signal.

After the writers finish successfully, completion or interruption records new chunk commits and the terminal status together in one atomic manifest update. If that update fails, the run reports an error. When new commits exist, it also attempts to record those commits while retaining the previous status and signal; this fallback does not count as successful completion. Finalized Parquet parts remain available for strict resume reconciliation if manifest persistence fails. The manifest format and compatibility checks are unchanged.

Resume requires current chunk commit metadata in every Parquet part. Parts without the native writer's g.output.chunk_commits footer metadata are rejected instead of being reconstructed from data columns. Every part must use the current production Parquet schema, and its footer commits must have unique chunk identifiers, exact row counts, and ranges matching the current BGEN chunk plan.

Part writes use hidden temporary files. Compatible resume removes recognized stale temporary part files, including visible temporary files from older writers, after all selected runs pass validation. An incompatible resume leaves these files untouched along with the other output artifacts.

Compatibility Checks

Resume first requires an existing pre-release schema-v0 run_manifest.json. It then compares the current requested run against the canonical execution_plan and its hash. A mismatch fails with a message naming the first incompatible manifest field. Earlier manifest layouts are not adapted because the application has no released legacy output contract.

Incompatible resume attempts are non-mutating: run_manifest.json remains unchanged and effective_config.toml is not newly created or overwritten until all selected phenotype output runs pass compatibility checks.

After initialization, run_manifest.json must remain present for every lifecycle update. If it is missing when an update begins, completion or interruption fails instead of treating its absence as success or synthesizing a replacement manifest.

Common mismatch causes:

  • changed BGEN, sample, phenotype, covariate, prediction-list, or selected LOCO prediction file;
  • changed sample, phenotype, covariate, prediction-list, or selected LOCO content even when path, size, and mtime_ns are preserved;
  • changed phenotype or covariate columns;
  • changed trait mode, binary correction plan, or Firth settings;
  • changed linear_minimum_variance or linear_relative_variance_tolerance;
  • changed selected association backend;
  • changed multi-phenotype sample mode, aligned sample set, aligned phenotype or covariate design, or prediction alignment;
  • changed chunk size or schema version;
  • an application upgrade that changed fixed output-writer, result-dtype, or JAX policy recorded in the execution plan.

Resume is not a way to combine different analyses into one output directory. Current manifests also fingerprint precise genotype moments, stable covariate projection, and safeguarded binary null fitting. Runs created before these policies were recorded require fresh output directories in both quantitative and binary modes; their results cannot be mixed with current parts.

Quantitative manifests include the linear kernel settings. Older pre-release quantitative manifests without those settings cannot be resumed; start a new output directory so that every part uses the same numerical policy.

Approximate-Firth manifests also identify the null solver's initial-convergence policy. Runs created before valid initial convergence was accepted require a new output directory. Binary score-only manifests omit this Firth-specific marker but still require the current genotype and null-logistic policies above.

Approximate-Firth manifests fingerprint the fixed inner proposal policy as float32_elementwise_float64_reduction. Older pre-release runs used the prior all-float64 inner policy and have a different execution plan and hash, so they cannot be resumed into or mixed with current output parts.

In particular, runs whose execution plan still contains the removed firth_newton_raphson_zero_start_iterations field have a different plan schema and hash. Start a new output directory instead of resuming those pre-release binary runs with a newer build.

The scalar approximate-Firth solver is the only supported correction path. Pre-release configurations and manifests must remove the experimental use_block_firth_math option and its block-only coefficient, likelihood, step-halving, and initial-response settings. Effective configuration metadata uses pre-release option schema version 0, and these fields are not accepted by pre-release manifest schema version 0; start a new output directory instead of resuming an older block-Firth-compatible run.

Graceful Interruption

During g regenie, the first SIGINT or SIGTERM requests graceful shutdown. The engine flushes queued chunks, saves committed output for resume, prints an interruption message, and exits with 128 + signal_number such as 130 for SIGINT.

Interruptions raised while Python initializes a backend or executes a backend callback retain their signal classification across the native boundary. If output has been initialized, the same graceful flush and interruption metadata path applies; an interrupted startup still returns the signal-derived status.

After that, rerun the same command with a config containing:

[output]
resume = true

Parquet Parts And Resume

Committed Parquet parts are both the resumable unit and the completed dataset. After interruption, resume writes only missing chunks and does not perform a separate dataset consolidation step.