All of Us cohort preparation
The local preparation tool builds an explicit, auditable handoff from accessible
genotypes and research tables to g Step 2. It runs without All of Us credentials
and does not download participant data. Run it inside the approved workspace when
using controlled inputs; its tables, keep lists, logs, and manifests contain
participant information and remain workspace artifacts.
This first version accepts local PGEN/PVAR/PSAM, BED/BIM/FAM, or BGEN plus an Oxford sample file. It selects human autosomes, exactly two alleles, and upstream-confirmed diploid calls. Hail VDS filtering, allele normalization, genotype QC, and densification are upstream workspace steps. A VDS is not a directly supported preparation input. The tool applies no automatic ancestry, relatedness, or sex exclusions.
Explicit identities and research inputs
Create a tab-delimited identity map with these four columns:
source_FID source_IID FID IID
0 synthetic-001 family-a participant-a
0 synthetic-002 family-b participant-b
The source pairs must cover the complete genotype sample file exactly once;
they are not inferred from a phenotype ID or a truncated identifier. The target
pairs are the canonical identities used by phenotypes, covariates, exclusions,
Step 1 predictions, and Step 2. If a PSAM omits FID, its source FID is explicitly
0, following PLINK's convention. Nonzero SID fields require upstream
normalization. Identifiers retain leading zeroes and are never parsed as numbers.
Duplicate source or target pairs, blanks, whitespace or double quotes in identities, and
ambiguous LOCO tokens fail. For example, (a_b, c) and (a, b_c) cannot coexist
because both serialize as a_b_c. Ordinary underscores are allowed.
Phenotype and covariate TSVs must have FID, IID, and the selected numeric
columns. Every selected participant needs a physical table row; use an explicit
missing value when an observation is unavailable. Accepted missing tokens are
empty fields, NA, NaN, nan, and -9. Prepared tables normalize these tokens
to NA, preserving missingness without filling any value. Binary phenotypes
use 1 = control and 2 = case. Nonfinite values, float32 overflow, malformed
rows, and duplicate table identities fail.
Keep, remove, and release-exclusion files have tab-delimited FID and IID
headers. #FID is also accepted for reuse of a generated PLINK keep list.
An empty keep list fails; a header-only exclusion list is valid. Unknown keep
pairs fail instead of silently shrinking the cohort. Release-exclusion lists can
include participants outside the current source: matching and absent counts are
reported. Convert any provider-supplied one-column participant list into explicit
canonical pairs through the approved workspace identity mapping first.
Supply the current release's flagged-sample list using
tool.release_exclusion_files=[...], and record tool.release_label. The tool
does not embed release-specific IDs or assume that a historical exclusion list
is current. Do not check these inputs into Git.
Plan and table-only preparation
From a repository installation with the tooling dependencies available:
python -m tooling.cli.data --config-name data_all_of_us \
tool.mode=plan \
tool.source_format=tables \
tool.identity_map=/workspace/inputs/identity-map.tsv \
tool.phenotype_file=/workspace/inputs/phenotypes.tsv \
'tool.phenotype_columns=[trait]' \
tool.covariate_file=/workspace/inputs/covariates.tsv \
'tool.covariate_columns=[age,sex,PC1,PC2]' \
tool.keep_file=/workspace/inputs/cohort.tsv \
'tool.release_exclusion_files=[/workspace/inputs/release-exclusions.tsv]' \
tool.release_label=chosen-release \
tool.output_directory=/workspace/work/prepared-tables
plan reads and validates inputs and prints JSON. It does not create the output
directory, execute PLINK, or create Hydra log/config directories. tables uses
the same arguments with tool.mode=tables to materialize aligned analysis tables
and keep lists without PLINK. In table-only source mode, identity-map row order
defines cohort order. With a genotype source, physical genotype sample order
defines it.
Plan and tables modes also work with source_format=pgen, bed, or bgen to
validate accessible sample identities and render the future conversion commands.
Genotype conversion requires execute explicitly.
Genotype preparation
For an already QC-selected PGEN source:
python -m tooling.cli.data --config-name data_all_of_us \
tool.mode=execute \
tool.source_format=pgen \
tool.input_prefix=/workspace/inputs/chr22-qc \
tool.confirm_diploid=true \
tool.identity_map=/workspace/inputs/identity-map.tsv \
tool.phenotype_file=/workspace/inputs/phenotypes.tsv \
'tool.phenotype_columns=[trait]' \
tool.covariate_file=/workspace/inputs/covariates.tsv \
'tool.covariate_columns=[age,sex,PC1,PC2]' \
tool.keep_file=/workspace/inputs/cohort.tsv \
'tool.release_exclusion_files=[/workspace/inputs/release-exclusions.tsv]' \
tool.variant_extract_file=/workspace/inputs/chr22-variants.txt \
tool.minimum_allele_count=5 \
tool.minimum_variant_call_rate=0.98 \
tool.release_label=chosen-release \
tool.threads=8 \
tool.memory_megabytes=16384 \
tool.output_directory=/workspace/work/prepared-chr22
The filter values above illustrate configuration; choose thresholds for the actual research analysis. Both filters are optional. Allele counts include nonfounders and are computed in the selected preparation cohort. Call rate counts available dosages, rather than treating an uncertain but present dosage as absent. These filters are not recomputed separately for every phenotype's missingness mask. Prepare a separate, explicitly defined cohort when the study requires filters calculated on a particular trait's complete cases.
For compressed PVAR input, set tool.pvar_zstd=true. BED input uses
tool.source_format=bed and its fileset prefix. BGEN input requires
tool.source_format=bgen, tool.input_bgen, tool.input_sample, and an explicit
tool.bgen_reference=ref-first, ref-last, or ref-unknown assertion. Do not
guess the reference convention from the file extension.
For PLINK conversion, the Oxford sample file must place ID_1 and ID_2 in its
first two columns. The engine's direct-reader contract permits other positions,
but the PLINK importer does not. Normalize the sample header upstream when
necessary, preserving each sample row's identity and order.
confirm_diploid=true records an upstream input assertion; it is not a scan that
qualifies every source genotype's ploidy. Check ploidy and upstream QC in the
workspace before making that assertion.
The audited argument vectors execute directly without a shell:
- Select the source FID/IID pairs and variants into
selected.pgen/.pvar/.psam. - Apply the explicit identity update and write canonical
genotypes.pgen/.pvar/.psamplus Layout-2 BGEN 1.2 withref-first,bits=8, and a two-ID Oxford sample file.
The intermediate and canonical PGEN files remain in the attempt for inspection and genome-wide Step 1 reuse. Budget local disk for both PGEN filesets, the BGEN, temporary PLINK conversion files, and outputs. BGEN preparation can be a substantial CPU and storage workload.
No fill-missing, genotype-imputation, or mean-imputation option is used. Hard-call roundtrip tests cover 0/0, 0/1, 1/1, missing calls, and allele order. Arbitrary BGEN genotype posteriors are not losslessly preserved: PLINK converts them to dosages, then reconstructs probabilities for eight-bit export. Qualify dosage precision and INFO effects for probabilistic inputs. BED A2 and other provisional-REF sources are not proof of biological reference alleles; normalize and verify the genome/reference convention upstream. See the official PLINK input, export, and filter documentation.
Audit and failures
preparation.json records the release/build labels, input paths and fingerprints,
identity-list hashes, selected/excluded counts, columns and coding, missingness
counts, complete-case counts, filters, exact PLINK arguments, and output hashes.
The installed PLINK version and each command's stdout/stderr are retained.
Before success, physical sample identities and order must match at each stage;
retained PVAR IDs must be unique and nonmissing; BGEN dimensions must agree with
the canonical sample and variant counts. Every exported BGEN record boundary is
checked without decompressing genotype blocks, including empty compressed
payloads, truncation, and unexpected trailing bytes. This also rejects malformed
tiny-cohort exports observed with the local pinned PLINK build despite its exit
status being zero.
Large genotype inputs default to path, size, modification/change times, inode,
and device identity to avoid an additional full input read during a plan. Set
tool.hash_genotype_inputs=true for full source content hashing. Small identity,
table, and list inputs are always hashed; successful output files are always
hashed. Inputs are revalidated before and after execution. These local
fingerprints are preparation provenance, not a cross-VM g resume contract.
An existing output directory is always refused. A failed attempt retains its
partial outputs and a failed manifest; correct the cause and choose a new
directory. tables_prepared records table-only completion and is not a completed
genotype export. A complete preparation does not replace scientific validation
of association results or a full BGEN probability-validation scan.
External genome-wide REGENIE Step 1
g implements Step 2 and consumes external REGENIE LOCO predictions. Prepare
the same canonical cohort on genome-wide QC-selected common markers, using
the same identity map, exclusion policy, phenotypes, and covariates. The
chromosome-22 pilot is a Step 2 pilot; it is not an adequate Step 1 training
dataset. Marker QC, LD pruning, population covariates, and relatedness strategy
must be chosen for the study.
The preparation command also writes genotypes.pgen/.pvar/.psam with canonical
IDs. For a genome-wide prepared dataset and a reviewed marker extraction
list, an external quantitative-trait handoff is:
regenie --step 1 \
--pgen /workspace/work/prepared-genome-wide/genotypes \
--keep /workspace/work/prepared-genome-wide/regenie.keep \
--extract /workspace/inputs/step1-markers.txt \
--phenoFile /workspace/work/prepared-genome-wide/phenotypes.tsv \
--phenoColList trait \
--covarFile /workspace/work/prepared-genome-wide/covariates.tsv \
--covarColList age,sex,PC1,PC2 \
--bsize 1000 --threads 8 \
--out /workspace/work/step1/quantitative
For binary phenotypes add --bt --cc12, because the prepared table uses
1/2 coding. regenie.keep is deliberately headerless. The companion
cohort.keep.tsv has a PLINK-style header for preparation reuse. Prepared missing
observations are NA, as required by REGENIE. See
REGENIE's input and Step 1 options.
Create the Step 1 output directory before starting REGENIE. Pin its version and
record all Step 1 inputs and settings. Pass its resulting phenotype-matched
*_pred.list to g --pred, with the same selected covariates and 1/2 phenotype
coding. Keep participant-level predictions inside the workspace. Validate a
quantitative and binary Step 2 pilot against REGENIE before scaling out.