Verily Workbench deployment
The Workbench runner packages a REGENIE Step 2 analysis with pinned inputs, an explicit data release, conservative memory planning and success-only result publication. It can be developed and checked using public or synthetic data. All of Us data access, release-specific preparation, reference parity and full-cohort performance must be qualified inside your authorized workspace.
Use the All of Us preparation guide for scientific preparation and the pilot checklist. Genome-wide REGENIE Step 1 remains external: supply one matching LOCO prediction file per selected phenotype. This package neither downloads a REGENIE binary nor substitutes a chromosome-only Step 1 fit.
Choose an execution route
| Route | Intended use | Remaining workspace check |
|---|---|---|
| Existing GPU Workbench app, isolated Python environment | First GPU pilot and profiling | Python installation policy, GPU visibility, disk and data permissions |
| Pinned command-line container | Reproducible task runtime | Approved registry, image access and NVIDIA container runtime |
| Portable WDL chromosome scatter | CPU orchestration and input/output localization | Workflow backend, disk sizing, service-account access and call caching |
The shipped image is a command-line task, not a web server. Do not point the Custom App devcontainer selector at it and assume it provides Jupyter. Workbench also documents a custom base image route that installs JupyterLab and its workspace dependencies during app creation; qualify that integration inside the actual workspace before relying on it. A bespoke devcontainer must satisfy Workbench's app-network and exposed-port requirements. See custom base images and custom apps.
Install in an existing Workbench app
Transfer the reviewed source revision and its qualified portable Linux
x86-64 wheel into the workspace using approved software-transfer procedures.
Keep participant data and all run artifacts in the workspace. The local
developer extension built with target-cpu=native is not a portable wheel.
Generic CPU flags do not make a wheel compatible with every Linux release:
its wheel platform tag and minimum glibc version must also match the Workbench
app. The container provides a matching pinned operating system. If an existing
app cannot load the qualified wheel, build against that app or an appropriate
manylinux build environment before running the pilot.
Create a dedicated environment from the project root; do not replace the notebook's existing environment:
uv python install 3.14.7
UV_PROJECT_ENVIRONMENT=.venv-workbench uv sync \
--python 3.14.7 --locked --no-default-groups --group workbench --no-install-project
uv pip install --python .venv-workbench/bin/python --no-deps /approved/path/g-0.1.0-PORTABLE.whl
export PYTHONPATH="$PWD"
export PATH="$PWD/.venv-workbench/bin:$PATH"
.venv-workbench/bin/g --help
.venv-workbench/bin/python -c 'import jax; print(jax.devices("gpu"))'
Replace the illustrative wheel path with the actual artifact. This installs the lockfile's CUDA 12 dependencies, JAX 0.11.0 and nvCOMP 5.3.0.16, plus Hydra, OmegaConf and PyArrow. The Workbench group excludes the profiling, testing, linting and documentation dependency groups. GPU drivers are provided by the host, not installed by this environment. Use a GPU supported by the locked JAX libraries and the embedded compute_70 PTX; actual GPU detection and decoder tests are required before accepting a new machine type.
The runner supports local paths and GCS. GCS mode uses the Workbench-provided
gcloud storage command and its workspace identity; it can accept an explicit
requester-pays billing project. Do not copy credentials into a software image.
Build the portable container
Build on an x86-64 machine allowed to run container builds, from the repository root. On Gauss, builds belong on a compute node, never the head node's Docker daemon:
docker build --platform linux/amd64 \
--build-arg SOURCE_REVISION="$(git rev-parse HEAD)" \
--file deploy/workbench/Dockerfile --tag g-workbench:reviewed .
docker run --rm g-workbench:reviewed --help
docker run --rm --entrypoint g g-workbench:reviewed --help
The multi-stage Dockerfile pins Python 3.14.7, Rust 1.97.1 and uv 0.11.14 by
verified amd64 digest, uses a dated Debian package snapshot, builds with the
Cargo and uv lockfiles and records their hashes in /opt/g/build-locks.sha256.
Rust and C++ compilation explicitly target generic x86-64. There is no mold
requirement or host CUDA toolkit dependency for the native extension: its
checked-in verified PTX is embedded at build time. JAX still compiles
shape-specific programs and uses its locked CUDA runtime components.
The runtime runs as UID/GID 1000 and contains no Rust compiler,
maturin, tests or full development environment. Tooling source is included to
support the Hydra runner, but only the Workbench dependencies are installed.
The build context uses an allowlist and excludes local environments, Git metadata, results, genomic files, credentials and generated native binaries. The Workbench image CI builds and checks the Dockerfile on a hosted CPU runner, then runs real native quantitative and binary jobs using the participant-free demo. It requires committed outputs for all 32 variants and checks exact missing-call sample counts (118 versus 128). It does not publish an image or run GPU qualification.
For a GPU pilot with already localized inputs and a local output directory:
docker run --rm --gpus all --user "$(id -u):$(id -g)" \
--mount type=bind,src=/approved/workspace/pilot,dst=/work \
g-workbench:reviewed --config-name workbench \
tool.action=run tool.manifest=/work/job.json \
tool.work_directory=/work/attempts tool.dry_run=false
All manifest paths must exist inside the container, and the mounted work
directory must be writable by the chosen user. The core image intentionally
does not include Google Cloud CLI. Localize GCS objects inside the workspace
first, use WDL file localization, or run the environment route with Workbench's
existing gcloud. Shell command substitutions above contain only local user
and revision metadata.
Before publishing the image later, choose an approved Artifact Registry repository and confirm that the app/workflow identity can pull it. Record and use the published image digest, rather than a mutable tag, for research attempts. Registry creation, image publication, GPU quota allocation and paid cloud resources are not performed by these assets. Workbench's documented image creation workflow explains workspace registry authentication and access.
Pilot and result publication
Create a version-one job manifest following the preparation guide. It records each input's SHA-256 and byte size, the release/reference build, selected phenotypes and covariates, LOCO predictions, explicit resource limits and the output location. GCS inputs can also pin an immutable generation.
.venv-workbench/bin/python -m tooling.cli.workbench --config-name workbench \
tool.action=preflight tool.manifest=/approved/workspace/job.json \
tool.work_directory=/approved/workspace/attempts
.venv-workbench/bin/python -m tooling.cli.workbench --config-name workbench \
tool.action=run tool.manifest=/approved/workspace/job.json \
tool.work_directory=/approved/workspace/attempts tool.dry_run=false
Each attempt gets its own localized inputs, native configuration, cache,
logs and status. An output bundle is accepted only after checking completed
Parquet files and native completion metadata; COMMITTED.json is published
last. A failed attempt does not publish a successful bundle. Retain failed
attempts and logs in the workspace for diagnosis.
Size persistent disk for the original/localized inputs, the runner's verified copies, temporary files, outputs and JAX cache. This first implementation can hold two copies of an already localized BGEN. The explicit memory estimate chooses a smaller chunk for large cohorts; it is a planning bound, not proof that every JAX/native allocation fits. Observe peak memory during the pilot before increasing sample counts or concurrent traits.
Portable WDL workflow
deploy/workbench/all_of_us_step2.wdl scatters independent chromosome jobs.
Every BGEN, sample, phenotype, optional covariate and prediction is declared as
a WDL File; the executor localizes them. Predictions must appear in the same
order as the manifest's predictions array. The helper changes only the URI
to the localized path, removes cloud-only generation fields and retains the
original content hashes and byte sizes. The runner verifies those bytes before
execution. Both source and localized manifests are returned for provenance.
Validate the workflow without executing it:
uv tool run --python 3.12 --from miniwdl==1.15.0 miniwdl check \
deploy/workbench/all_of_us_step2.wdl
Workflow inputs are all_of_us_step2.chromosomes (an array of ChromosomeJob
objects), all_of_us_step2.image (a registry image digest),
all_of_us_step2.cpu_threads and all_of_us_step2.memory_gib. Each job has
manifest, bgen, sample, phenotype, nullable covariates, and an ordered
predictions file array. An executor can receive workspace GCS URLs for the
File fields. Each call returns a result.tar containing one committed
bundle, together with source and localized manifests. These outputs remain
within the approved workflow storage destination.
This portable template reserves CPU only, and rejects GPU manifests. It declares only container, CPU and memory runtime settings. Actual disk sizing, GPU resource keys, retries, service-account permissions and task-cache policy are backend-specific and must be configured and tested inside the workspace. Use the direct GPU app route for the first GPU pilot. WDL syntax validation does not qualify Verily execution or All of Us data access.
Recovery and qualification limits
The native engine's resume identity still depends on local filesystem identity. Freshly downloaded copies on another VM do not preserve it. Keep the original attempt and localized files for same-VM recovery; rerun an interrupted chromosome as a new attempt after a VM change. The Workbench publication layer does not relax native input-identity checks or promise cross-VM resume.
Before research use, run reference parity against REGENIE, missing-call and allele-orientation checks, low-count and binary-imbalance cases, interrupted attempt tests, and full-cohort memory/disk/cost measurements. The existing 2,504-sample public-data measurements are not All of Us capacity evidence. WGS missingness currently sends BGEN decoding through the CPU fallback while association can still use the GPU; record the actual selected execution path.
Keep individual-level data, prediction files, manifests, logs and profiling artifacts inside the controlled workspace. Summary-statistic export requires the applicable All of Us dissemination review. See the current dissemination policy.