Workbench job preparation and execution
The Workbench runner prepares a local, isolated REGENIE Step 2 attempt from a
versioned manifest. It verifies input bytes, checks structural sample alignment,
chooses a smaller variant chunk when declared memory requires it, launches the
installed g regenie executable, and publishes only completed, validated outputs.
It imports neither JAX nor the native extension during preflight.
This package can be tested with synthetic and public inputs on this machine. It has not been qualified against All of Us data. Run the first real-data qualification inside the authorized workspace and retain every artifact there. The runner does not select a scientific cohort, perform genotype QC, generate ancestry covariates, or establish that a study design is statistically valid.
Inputs and scientific scope
Prepare autosomal, biallelic diploid BGEN Layout 2 inputs and an Oxford .sample
file. Probability encoding, phasing, ploidy and missingness still undergo the
native reader's full validation when the engine starts. The bounded BGEN profiler
provides an encoding inventory, not a whole-file eligibility certificate.
Phenotype and covariate tables have FID IID identifiers and tab-separated
numeric columns. Missing values use empty fields, NA, NaN, nan, or -9. Binary phenotypes
use 1 for controls and 2 for cases. Every selected phenotype must have enough
aligned, nonmissing samples; binary traits must retain both classes. Sample order
in these tables need not match the BGEN order. Duplicate sample pairs and
ambiguous FID_IID serialization are rejected.
Supply a genuine, genome-wide external REGENIE Step 1 LOCO prediction file for
each selected phenotype. The runner generates a prediction list pointing to the
verified local copies. A LOCO header begins FID_IID followed by unique serialized
sample keys; autosomal chromosome rows use 1 through 22 or chr1 through chr22.
Additional genome-wide rows such as X, chrX or 23 are accepted and
normalized under the native LOCO grammar; this does not qualify sex-chromosome analysis.
Selected samples must have finite predictions. Structural checks do not prove
that predictions came from the intended study or a correctly executed Step 1.
The default sample policy is per-phenotype: each trait retains its own available
samples after phenotype and covariate alignment. The original native strict
resume identity is unchanged. A retry creates a new attempt rather than resuming
outputs copied from another VM.
Manifest version 1
The following illustrates the exact schema. Replace the example paths, checksums and byte sizes with those of your prepared files. The all-zero digests are placeholders and will fail verification unless they match actual content. No real participant identifiers or All of Us bucket paths are included.
{
"schema_version": 1,
"dataset": {
"name": "synthetic-pilot",
"release": "fixture-v1",
"reference_build": "GRCh38"
},
"inputs": {
"bgen": {
"uri": "/work/inputs/chr22.bgen",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"size_bytes": 123456
},
"sample": {
"uri": "/work/inputs/chr22.sample",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"size_bytes": 1234
},
"pheno_file": {
"uri": "/work/inputs/phenotypes.tsv",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"size_bytes": 1234
},
"covar_file": {
"uri": "/work/inputs/covariates.tsv",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"size_bytes": 1234
}
},
"predictions": [
{
"phenotype": "trait",
"uri": "/work/inputs/trait.loco",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"size_bytes": 1234
}
],
"analysis": {
"trait_type": "quantitative",
"phenotype_columns": ["trait"],
"covariate_columns": ["age", "sex"]
},
"resources": {
"device": "gpu",
"cpu_threads": 8,
"writer_threads": 2,
"gpu_memory_gib": 16,
"memory_fraction": 0.5,
"max_chunk_variants": 16384
},
"output_uri": "/work/published"
}
Unknown or duplicate JSON fields are errors. Columns must begin with an ASCII
letter and contain at most 80 letters, digits, dots, underscores or hyphens;
FID and IID are reserved. Predictions must cover selected phenotypes exactly
once. Omit covar_file when covariate_columns is empty. The source dataset name,
release and reference build are explicit provenance labels, not automatic release
or genome-build validation.
For a CPU job use "device": "cpu" and declare "memory_gib": 32 in resources.
For a binary job use "trait_type": "binary" and add this analysis setting:
"binary": {
"fallback_method": "firth_approximate",
"p_threshold": 0.05
}
The threshold must remain strictly between zero and one after native float32
rounding; the rounded value is written explicitly to the generated TOML. The other correction
choice is score_only. Optional
resources.firth_batch_size is a positive integer for binary jobs. When
approximate Firth is selected, the default uses a power-of-two batch no greater than 256 or
max_chunk_variants, reduced further when necessary to fit the declared
full-cohort preparation budget. An explicit batch that cannot fit is rejected.
Preflight, profiling and execution
Use an explicit private work directory on sufficiently large local disk. It must be owned by the current user and not writable by other users or groups. All commands create a new owner-only attempt directory and print its path. They leave existing attempts untouched. The source manifest and input files remain unchanged.
python -m tooling.cli.workbench --config-name workbench_preflight \
tool.manifest=/work/job.json tool.work_directory=/work/attempts
python -m tooling.cli.workbench --config-name workbench_profile \
tool.manifest=/work/job.json tool.work_directory=/work/attempts \
tool.profile_max_variants=128
python -m tooling.cli.workbench --config-name workbench_run \
tool.manifest=/work/job.json tool.work_directory=/work/attempts
The repository also provides just workspace-preflight, just workspace-profile
and just workspace-run with the same overrides. In an installed Workbench
container, the python -m entrypoints are sufficient.
preflight hashes local sources, inspects the BGEN header, validates local sample,
phenotype, covariate and LOCO alignment when all inputs are local, and checks free
disk. For remote sources it checks cloud metadata, but reports content verification
as pending until localization. It never downloads GCS inputs. A remote BGEN means
sample count and output disk requirements cannot yet be determined; those checks
run again after localization.
profile localizes and verifies every input before bounded encoding inspection.
run performs localization, structural alignment, capacity planning and one
fresh native execution. Change tool.runner_prefix only when the executable
needs an explicit command prefix; it is a shell-free argument list, normally
[g,regenie].
For a plan without cloud requests, input localization or engine execution:
python -m tooling.cli.workbench --config-name workbench_run \
tool.manifest=/work/job.json tool.work_directory=/work/attempts \
tool.dry_run=true
Dry-run still writes local manifest, preflight and status evidence and verifies
already-local source bytes. It does not download inputs, invoke gcloud, import
JAX, execute the engine, or publish outputs. A dry-run with remote inputs reports
those checks as pending rather than claiming success at validation.
Memory and disk planning
The chunk estimator preserves the full cohort and reduces variant width. It
reserves one GiB for backend/compiler state, float64 design and per-trait arrays,
eight float32 chunk matrices plus two per selected trait, and additional Firth
workspace. For Firth, it models all grouped candidate lanes up to the native
per-trait capacity of 1024 (also explicitly set in generated TOML), padded to
the solver batch width, including
covariate-dependent gathered projection arrays and separate solver batch state. The native engine pads final or chromosome-tail chunks to this chosen width,
so the estimate uses the full width even for a short tail. Trait-by-chunk indices
also remain within the engine's signed 32-bit index domain. It caps the estimate
at the declared memory multiplied by
memory_fraction, which must be positive and at most 0.8. A configuration unable
to accommodate even one estimated variant is rejected.
This is a conservative working-set heuristic, not an out-of-memory guarantee. Compiler temporaries, genotype encodings, allocator fragmentation, simultaneous phenotype groups, driver allocations and other processes can change actual use. The declaration is not a hardware probe. Qualify real cohort sizes and monitor both host and GPU memory before increasing widths or trait concurrency. Do not reduce the cohort implicitly to make a job fit.
GPU subprocesses request CUDA explicitly, enable float64, disable JAX's initial GPU preallocation, and use the declared memory fraction for its allocator setting. CPU/BLAS thread counts and the local trusted compilation cache are explicit. The runner does not change the engine's statistical or GPU compiler precision policy. Never share the compilation cache with untrusted writers.
The disk preflight reserves all localized input bytes, three estimated output copies at 512 bytes per variant per trait, and two GiB of compilation headroom. Actual files can be larger, especially when identifiers are long. The launch checks free local disk again after a remote BGEN has been localized. Keep enough space on a separate local publication filesystem as well; the final copy fails without replacing a previous published attempt if that filesystem fills.
GCS localization and publication
Each input URI may instead be a literal gs://bucket/object with the same
SHA-256 and byte size. An optional generation is a positive decimal string:
{
"uri": "gs://example-workspace/prepared/chr22.bgen",
"generation": "1234567890123456",
"sha256": "0000000000000000000000000000000000000000000000000000000000000000",
"size_bytes": 123456
}
Queries, wildcard patterns, traversal components and embedded generation suffixes
are rejected. GCS mode requires the external authenticated Google Cloud CLI.
Optional tool.billing_project=PROJECT supplies its requester billing project.
Authentication and workspace permissions remain platform responsibilities.
The runner describes the pinned generation, or resolves the current generation
once, and downloads gs://bucket/object#GENERATION into a .partial file. It
disables automatic content-encoding decompression and verifies SHA-256 and byte
size before making that input available to the engine.
The resolved generation is recorded in provenance. It never substitutes a newer
generation after a failed download. Partial inputs remain diagnostic artifacts.
Cloud CLI copy contract.
Set output_uri to an authorized workspace gs://bucket/prefix to publish there.
Every attempt uses a unique child prefix. Each upload uses a create-only
precondition and must pass server-observed size and MD5 verification; a missing
MD5 fails closed. The upload subprocess disables parallel composite uploads
through a process-local Cloud CLI environment setting, leaving the user's saved
CLI configuration unchanged. All data and provenance files are verified
before the COMMITTED.json marker is sent last.
Generation preconditions.
Cloud publication is not an atomic directory transaction. A failed transfer can
leave partial objects without a commit marker. A lost acknowledgement during the
last upload can leave an unconfirmed commit marker; failed attempt status records
publication_committed: null after publication has started. Treat an uncertain
attempt as requiring inspection, and start a new attempt for a retry. No cloud
success is returned unless every verification finishes.
Local publication copies and verifies the complete bundle into a hidden sibling directory and renames it on the same filesystem. The destination appears with the commit marker and all files together. Existing destinations are never reused.
Evidence and remaining qualification
Each attempt retains the input manifest, verified local inputs, generated TOML,
relative LOCO list, alignment and capacity reports, provenance, engine stdout and
stderr, native output directories and a terminal status.json. Cancellation
forwards an interrupt to the isolated engine process group, including launcher
descendants, and allows up to 30 seconds for native chunk commits to be preserved
before killing any remaining group members. A second interrupt skips the
remaining grace period and forces group shutdown immediately. Failed attempts
remain intact.
A successful exit code alone is insufficient. Publication additionally requires
exactly the expected completed phenotype manifests, contiguous chunk coverage of
all input variants, referenced Parquet parts with matching actual footer row
counts and chunk commit metadata, safe paths, and SHA-256 identities. This check
does not compare association values against REGENIE or inspect every data page.
The successful bundle contains outputs, logs and provenance, excludes localized
source inputs and cache, and carries a hashed file inventory in COMMITTED.json.
Treat all generated bundles, logs, paths, identifiers, predictions and summary statistics as restricted workspace artifacts. The commit marker explicitly does not certify external export safety. Apply the actual platform dissemination rules before any export. No automated blanket export decision is made.
On All of Us, pin the actual release, apply release-specific exclusions and QC, prepare the intended ancestry/relatedness design, execute external Step 1, and compare quantitative and binary Step 2 results with REGENIE. Then qualify full cohort memory, missing-call decode throughput, interrupted-job behavior, input localization, cloud permissions and cost. These tasks need authorized workspace access and remain separate from local synthetic validation.