Methods
About › Methods · see also Scope, Sources & acknowledgements, Fields.
Two tiers, one rule: every value on this site points back to a record in a public archive or an open paper.
About this page
The registry classifies every shotgun-metagenome study reached in ENA (INSDC BioProjects from ENA, NCBI SRA and DDBJ) from its archive record — human host is one of the classified attributes (6,898 of 64,369 studies are human = yes or mixed). The catalog curates the human-gut studies of the registry, all ages, at sample level: four routes recover values, each committed only with a verbatim evidence quote (≤ 12 words), a labelled source, a locator, a route and a confidence; unknown stays unknown. Every release is immutable and versioned (Downloads & releases); code, configuration and vocabularies live in microbiome_repo-pipeline.
1 · Registry: enumeration and classification
Every INSDC BioProject with METAGENOMIC / WGS-type runs — deposited in ENA, NCBI SRA or DDBJ and enumerated through ENA's mirror (portal API) in dated slices — is a registry study; the enumeration reaches every shotgun-metagenome study, not only human ones, and each is then classified for host (registry_universe_audit.csv proves completeness per slice). Per study the questions are: host (human / non-human / mixed / unknown), body sites from a controlled vocabulary anchored to UBERON (17 terms), life stages, population flags, assay (shotgun / amplicon / genome / transcriptome / other) and access. Deterministic rules decide what the archive fields settle (host taxon, library fields, numeric age with unit); the rest is classified by two independent model replicates over title, description and sample attributes, disagreements adjudicated by a stronger model; every call carries its evidence rows. 63,952 of 64,369 studies are classified (99 %); 417 are pending. The sample tier harvests every BioSample of the human studies and normalises site, age, sex and country from its attributes (637,397 BioSamples).
| classification stage | meaning | studies |
|---|---|---|
| deterministic_rule | ENA-field rule (host taxon 9606, library fields, numeric age with unit) | 52,142 |
| sonnet_x2 | two replicate model classifications in agreement | 5,164 |
| deterministic_prior | prior carried over from an earlier triage (reason code / body-site call) or deterministic term match | 4,808 |
| opus_adjudicated | replicate disagreement adjudicated by the stronger model | 1,663 |
| pending | not yet classified | 417 |
| owner_decision | study-level decision by the catalog owner (config/registry_overrides.yaml) | 96 |
| curator_audit | study-level host/assay audit of the archive records (config/registry_overrides.yaml) | 79 |
Body-site vocabulary (UBERON anchors) and life-stage bins
| code | label | UBERON | terms |
|---|---|---|---|
| gut_stool | Gut / stool | UBERON:0001988 (feces), UBERON:0000160 (intestine) | |
| oral | Oral cavity | UBERON:0000167 (oral cavity), UBERON:0001836 (saliva), UBERON:0016482 (dental plaque) | |
| skin | Skin | UBERON:0002097 (skin of body) | |
| nasal_nasopharyngeal | Nasal / nasopharyngeal | UBERON:0001707 (nasal cavity), UBERON:0001728 (nasopharynx) | |
| respiratory_lower | Lower respiratory | UBERON:0002048 (lung), UBERON:0007311 (sputum) | |
| vaginal_urogenital | Vaginal / urogenital | UBERON:0000996 (vagina), UBERON:0000002 (uterine cervix), UBERON:0001088 (urine) | |
| milk | Human milk | UBERON:0001913 (milk) | |
| blood_tissue | Blood / tissue / internal fluids | UBERON:0000178 (blood) | |
| eye_ear | Eye / ear | UBERON:0001811 (conjunctiva), UBERON:0001352 (external acoustic meatus) | |
| other_site | Other human site | ||
| unknown_site | Body site unknown | ||
| multi_site | Multiple sites |
| code | label | days | terms |
|---|---|---|---|
| neonate | Neonate | — | |
| infant | Infant / toddler | — | |
| child | Child | — | |
| adolescent | Adolescent | — | |
| adult | Adult | — | |
| elderly | Elderly | — | |
| unknown_age | Age unknown | — | |
| mixed_ages | Mixed ages | — |
2 · Catalog scope
A registry study enters the catalog when its host is human, its assay is shotgun and gut / stool is among its body sites (Scope shows the funnel and what is left out). Every harvested BioSample of such a study is a catalog sample; its body_site_class says whether it is a gut sample (primary), has no site attribute (unknown) or belongs to another site of the same study (excluded / linked). Age categories (neonate, infant, child, adolescent, adult, elderly) come from the numeric age when known, else from the sample's stated life stage, a cohort-wide statement, or the study's primary life stage — the basis is recorded per sample.
3 · Curation routes and precedence
| route | source | method | confidence | values |
|---|---|---|---|---|
| R1 | the sample's own archive attributes (BioSample) | a field map of attribute keys → fields with deterministic parsers (units, enumerations, plausibility windows); a utility model triages ambiguous keys, never values; health-condition text → vocabulary codes by rule and by a model pass over distinct (key, value) pairs | 0.7–0.9 | 2,857,800 |
| R2 | supplementary tables of the linked open-access papers | Europe PMC supplementary files (xlsx / csv); a table is used only when its sample identifiers join the study's accessions, aliases or library names exactly (≥ 50 % of rows or ≥ 20 hits); header-named columns; no model. Since R2026.11 this route also carries external curated resources (evidence_source = external.<resource>, locator = link to the source record; see Sources) joined on archive accessions, at confidence 0.6–0.75 so that a study's own tables win ties | 0.6–0.85 | 398,671 |
| R3 | paper full text (Methods / Results) | the first open-access PMC paper is read twice by independent Sonnet-class replicates; a cohort-wide statement (life stage, country, single-sex cohort, a condition shared by every subject, an antibiotic exclusion window ≥ 1 month) is kept only when both replicates agree and its quote is a verbatim substring; case–control group statements are recorded but never applied to samples | 0.6–0.7 | 452,285 |
| R4 | abstract, title, ENA study description | utility model, one request per study, JSON output, verbatim-quote check; cohort-wide statements only | ≤ 0.5 | 278,201 |
Precedence per sample × field: R1 > R2 > R3 > R4; a cohort-wide statement (scope = study_all) fills only the samples that lack a sample-level value. For the 368 studies that were curated in depth for the infant extension, that curation (which also covers delivery mode, feeding, gestational age, birth weight, maternal antibiotics, probiotics, HMO, NEC) is copied verbatim and wins. Conflicting values (two attributes giving materially different ages) are committed as neither and logged.
| field | R1 | R2 | R3 | R4 |
|---|---|---|---|---|
| age_category | 0 | 0 | 0 | 0 |
| country | 475,105 | 19,082 | 16,020 | 13,398 |
| health_condition | 109,978 | 57,770 | 50,250 | 67,989 |
| subject_id | 317,593 | 26,254 | 65,086 | 18,243 |
| age_at_collection_days | 119,303 | 39,984 | 5,374 | 344 |
| sex | 133,426 | 47,217 | 11,526 | 0 |
| detailed_location | 0 | 0 | 0 | 0 |
| lifestyle | 3,545 | 696 | 8,204 | 6,427 |
| collection_date | 387,454 | 499 | 46,813 | 5,300 |
| intervention | 43,715 | 5,241 | 63 | 133 |
| antibiotic_exposure | 26,144 | 38,437 | 45,171 | 5,870 |
| bmi | 34,941 | 18,672 | 0 | 0 |
| timepoint_label | 157,154 | 22,544 | 0 | 0 |
| diet | 12,676 | 4,150 | 46 | 57 |
| smoking_status | 11,326 | 2,095 | 536 | 0 |
| medication | 7,751 | 2,193 | 570 | 38 |
| stool_consistency_bristol | 3,038 | 0 | 0 | 0 |
Curated depth of the 2,798 studies (routes that contributed ≥ 1 value): R1 719 · R1;R3 568 · R1;R4 561 · R1;R2;R3 244 · R1;R3;R4 222 · R1;R2 214 · R1;R2;R4 89 · R1;R2;R3;R4 84 · none 28 · R4 25 · R3 13 · R3;R4 9 · R2;R3 8 · R2 6 · R2;R3;R4 4 · R2;R4 4.
4 · Evidence and confidence
Every row of gut_sample_determinations.parquet carries evidence_source (e.g. sample.attr.host_age, paper.supplement.<file>, paper.fulltext.methods, paper.abstract), evidence_locator (attribute key, PMCID, PMID or accession), evidence_quote (≤ 12 words, verbatim), determined_by (parser or model role and version) and a confidence tier. Confidence is an engine tier, not a calibrated probability (Fields › Confidence). Model ids are resolved per role at run time from config/models.yaml; the exact model is recorded on every model-derived row.
5 · Cohorts, papers, authors
Papers are linked to studies from Europe PMC (accession mentions, data-availability statements) and NCBI BioProject publication records, with a relation (own_data when the paper describes the deposit, unsure when it may re-analyse it). Authors are the Europe PMC author lists of those papers, string-matched and not disambiguated. Cohorts group studies that share an own-data paper or a curated cohort record (connected components); the grouping is recomputed at every release.
6 · Releases and provenance
Releases are immutable and numbered (R2026.n); each maps to a data-package semver and to Git tags in the data and site repositories. Fact tables carry release_added / release_retired / package_added; the changes pages list what each release added. The site is generated deterministically from the package (byte-identical rebuilds); data/manifest.json lists every data file with its SHA-256.
7 · Limitations
Why some values are still missing. Every project went through the sources we can reach: the archive records (all BioSample attributes; sample names, titles and library names decoded per project with two independent model replicates that must agree), the abstract and archive description, the open full text (methods prose and article tables), the open supplementary tables, and external curated resources. Projects without a linked paper were searched again by accession and title. Supplementary files come from Europe PMC, which delivered them slowly during this pass; supplements that could not be downloaded are marked not available below and on the Contribute page, so they can be shared or retried. What remains missing is mostly not published in a form we can reach:
| source | projects with values | checked, nothing usable | not available |
|---|---|---|---|
| Sequence archive | 2,701 | 97 | 0 |
| Abstract / description | 998 | 1,530 | 270 |
| Full text | 1,152 | 792 | 854 |
| Supplementary tables | 435 | 1,403 | 960 |
| External resources | 312 | 0 | 2,486 |
| User contribution | 0 | 0 | 2,798 |
| Human verified | 0 | 0 | 2,798 |
Each project page shows the same checkmarks for that project, and the Project sheet can be filtered by source. How many values each source contributes, overall and per field, is on Where the values come from.
- Coverage is bounded by what submitters deposited and what open papers state: 29 % of samples have an age, 50 % a health-condition code, 34 % a sex, 20 % an antibiotic-exposure value.
- Case–control cohorts stay unknown at sample level unless an attribute or table names the group; cohort-wide codes are assigned only when every sequenced subject shares the condition.
- Archive ages without a unit are read as years for studies outside the infant extension; ambiguous or conflicting values are not committed.
- R2 reads machine-readable supplements (xlsx / xls / csv / tsv / docx tables, nested archives) that join on sample, run, alias or subject identifiers; PDF-only supplements are not parsed; papers that are not open access contribute only their abstract.
- Values added in the deep pass fill gaps only: they never replace a value from an equal or higher-ranked route, and where both exist they agree for sex, country and age in ≥ 98 % of compared samples.
- Registry classification of pending studies and the model-based calls carry their evidence but are not human-reviewed; use the Flag an issue link on any study or cohort page.
- Sandpiper community profiles are shown where they exist; the registry records how many runs of each study are in the Sandpiper snapshot.
Documents
- README (README.md)
- Data dictionary (DATA_DICTIONARY.md)
- Changelog (CHANGELOG.md)
- Registry report (REGISTRY_REPORT.md)
- Sandpiper report (SANDPIPER_REPORT.md)
- Authors report (infant extension) (AUTHORS_REPORT.md)
- Data-model fix report (infant extension) (DATA_MODEL_FIX_REPORT.md)
- Infant-extension catalog report (CATALOG_REPORT.md)
- Infant-field extraction report (EXTRACTION_REPORT.md)
- Collections: how membership rules work and how to propose one (COLLECTIONS.md)
- Sandpiper profiles for the whole catalog (report) (GUT_SANDPIPER_REPORT.md)