# DATAFIX_REPORT — data_package_v1.2.0 → v1.2.1

Deterministic leaf run (no LLM), environment `infantcat`, inputs: data_package_v1.2.0.zip, sandpiper_profiles.parquet (8,470,057 rank rows / 79,473 samples), organisations.parquet, bioproject_records.parquet, authors.parquet, curation_kernel_ext.py. Package zip sha256 `fe785de73c5bc06accd2c2dfa9a969cdfc18214df35944b34b80dece240b479c` (62.3 MB, 49 files).

## Headline counts (v1.2.1)
* frac_profiled: 0.5154
* n_age_scope_infant: 74,695
* n_author_rows: 28,367
* n_biosample_units: 153,685
* n_body_site_excluded: 5,475
* n_catalog_scope: 71,795
* n_distinct_authors: 10,965
* n_included_without_author: 103
* n_organisation_keys_displayable: 3,747
* n_parent_biosamples: 16
* n_profiled_samples: 79,473
* n_run_units: 521
* n_runs: 174,022
* n_runs_profiled: 87,125
* n_samples: 154,206
* n_scope_cols: 16
* n_sp_cols: 39
* n_studies: 389
* n_studies_with_author: 1,264
* n_studies_with_displayable_organisation: 9,513
* n_studies_with_own_profiled_runs: 302
* n_studies_with_panel: 181
* n_studies_with_profiles: 304
* n_study_cols: 117
* n_wide_cols: 147
* package_version: 1.2.1

## Per finding
| finding | status | before (v1.2.0) | after (v1.2.1) | checks / notes |
|---|---|---|---|---|
| F1 / R3-5 | fixed | 18,888 organisation rows, 9,886 keys; 6,042 study_authors_summary rows with a SUB id; 8 e-mail rows; 9 placeholders; 93 lowercase-username rows | 6,143 rows tagged not_an_organisation ({'submission_id': 6033, 'lowercase_username': 93, 'placeholder': 9, 'email_or_handle': 8}); 3,747 displayable keys; 0 published strings match SUB\d{6,} or '@'; PRJNA studies prefer BioProject Organization (663 ENA centre rows suppressed); studies with displayable organisation 9,513/9,579 (included 386/389; PRJNA376566, PRJNA327106, PRJEB80456 now have none) | assert no organisation string in study_authors_summary / study table / index matches SUB\d{6,} or contains '@' — passed. universe_studies_all.parquet has no organisation column (the site's universe_index.json field 'o' is built from study_authors_summary.csv v2). |
| R3-1 | fixed | exact GTDB genus match (g__Enterococcus only, etc.) | regex ^g__<Name>(_[A-Z]+)?$ for 11 genus indicators + 6 core genera; sp_ra_g_Enterococcus: 19,090 samples changed, mean 0.0234→0.0324; sp_ra_g_Veillonella: 6,076 samples changed, mean 0.0211→0.0253; sp_ra_enterobacterales_core: 3,315 samples changed, mean 0.1228→0.1243; sp_ra_g_Bacteroides: 1,786 samples changed, mean 0.0577→0.0577; sp_ra_g_Phocaeicola: 590 samples changed, mean 0.0408→0.0409 | indicators with no suffix genera unchanged (max \|Δ\| 5e-7 from stored rounding); gtdb_to_ncbi_note table added to DATA_DICTIONARY; matched taxa listed per indicator. |
| R3-2 / F3 / F13 | fixed | panels on body-site scope, no depth filter: 298 studies; PRJNA1046438 n=424 (all adult_flagged), PRJEB11419 n=888 (234 adults), PRJNA695570 n=1,614, PRJNA300541 n=179, PRJNA294605 n=138 of '169'; 38 studies with n ≠ recomputation; n_samples_x/_y artefacts; sp_median_* on all profiled | panel scope = catalog_scope ∧ sp_profiled ∧ NOT sp_low_depth: 181 studies with a panel (41,290 rows); fallback labels: {'catalog_scope_adequate_depth': 181, 'fallback:no_catalog_scope_samples': 118, 'no_profiled_samples': 85, 'fallback:all_age_evidenced_profiled_samples_low_depth': 3, 'fallback:no_age_evidenced_profiled_samples': 2}; PRJNA1046438 → fallback:no_catalog_scope_samples, PRJEB11419 → fallback:no_age_evidenced_profiled_samples, PRJNA695570 n=698, PRJNA300541 n=24, PRJNA294605 n=138 (study 158 rows); sp_median_* recomputed on the same scope (changed for 182 studies, null for 208 with label in sp_panel_scope); panels carry n_profiled_age_infant/adult/unknown for the caption | assert panel n_samples_panel == recomputation from the shipped wide table for every study — passed; assert n_samples_panel == n_profiled_catalog_scope − n_low_depth_excluded — passed. Caption text itself is site-track. |
| R3-3 | fixed | PRJEB6921: 172 Mus musculus/Gallus gallus rows + 103 isolate-genome rows with body_site_class unknown; PRJEB49206: 20 isolate rows primary; nothing in value_history | 295 rows → body_site_class = excluded with exclusion_reason_code (host_nonhuman 172; assay_isolate_genome 123); 295 value_history rows (change_stage auditor_review:R3-3), all passed curation_kernel_ext.validate_row; body-site excluded 5,180 → 5,475; study n_excluded_host_or_isolate, n_body_site_excluded added; n_infant_scope_samples PRJEB6921 772→497, PRJEB49206 438→418 | affected studies: PRJEB6921, PRJEB49206 only (rule: any run with non-human host taxon; or all runs GENOMIC ∧ named-microbe organism). PRJNA799247 (68) and PRJNA1082298 (2) have named-microbe organisms on METAGENOMIC runs — not covered by the deterministic rule, listed in review_named_microbe_metagenomic_samples.csv and flagged via sp_nonmeta_class; 3,158 GENOMIC-only samples (2,494 primary) in review_genomic_only_samples.csv; sandpiper_auditor_candidate_runs.csv (375 runs) shipped. n_samples (archive definition) unchanged by design — excluded rows stay in the table as for other body-site exclusions; the owner may want to re-triage PRJEB6921 (mixed mouse/chicken/isolate/human deposit). |
| R3-6 | fixed | stage_order by artifact date (screen_haiku after later stages in 1,155 studies); verdict_norm null 3,775; consolidated rows unmarked | stage_rank/stage_level ontology; screen_haiku first in all 3,978 studies with a screen; catalog_final last in all 9,579; verdict_norm null → 176 (ERROR 135 + 41 empty) with verdict_norm_note; yes→uncertain 351, no→exclude 3,248; is_consolidated on 20,222 rows with consolidated_kind final/triage_v2/growth_wave | site panel rebuild is site-track. |
| R3-7 | fixed | cell 6 AttributeError; scope on body_site_class; no age_scope/Sandpiper; runs join misses run units | rewritten on age_scope/catalog_scope; coverage cell uses coverage_age_scope_infant/coverage_catalog_scope; Sandpiper cell; runs joined via biosample_accession/run_accession; executed in the package directory with outputs stored, 0 errors | nbclient failed with PermissionError (kernel launch not permitted in this sandbox) → flattened exec of every code cell in one namespace (deviation declared); outputs stored in the .ipynb. |
| R3-8 | fixed | no sandpiper_run_qc section; sample_unit 'biosample_pooled' (521); 57 undocumented study columns; wrong counts | run_qc section with miss-reason (10 values incl. unknown_size) and organism-class vocabularies; sample_unit → 'run' in run_qc and study_coverage; generated complete column reference: 147 sample + 117 study columns, assert every column has a description — passed; counts corrected in CHANGELOG (study sp_* = 18; study_paper_links 973; study_all_infant studies 31) |  |
| R3-9 | fixed | sp_flag_non_metagenome True for 1,650 samples (1,617 via 'Homo sapiens') | derived from sp_nonmeta_class ∈ {named_nonhuman_host, named_microbe_or_other} over all runs: 365 samples; sp_nonmeta_class column on sample table and summary ({'not_flagged': 135255, 'named_human_host': 18586, 'named_microbe_or_other': 193, 'named_nonhuman_host': 172}); strict flag kept in run_qc |  |
| F4 | fixed | n_biosamples = nunique(biosample_accession) incl. 16 parents | n_biosamples = BioSample-unit rows (153,685); n_run_units 521; n_parent_biosamples 16; README headline reworded; build_counts.json n_biosample_units/n_run_units/n_parent_biosamples | 153,685 + 521 = 154,206 — asserted. |
| F6 | fixed | 5 studies with n_samples ≠ rows and no note; sp fractions on mixed denominators; three 'studies with profiles' counts | shared_biosample_note on PRJNA1019702, PRJNA1198101, PRJNA555020, PRJNA613032, PRJNA613054; sp_frac_samples_profiled = profiled / n_sample_rows everywhere; single definition documented: 304 studies with profiled rows, 302 with own profiled runs, 181 with a panel (README + build_counts) |  |
| F9 | fixed (owner decision pending on peri-rectal swabs) | dictionary vocabulary wrong; excluded/linked undefined; scopes never intersected | body_site_class vocabulary primary/unknown/excluded/linked defined; catalog_scope column (71,795 rows = 74,695 age-scope − 2,900 body-site excluded/linked) + study n_catalog_scope; field_coverage_summary gains coverage_catalog_scope; README headline uses catalog_scope | PRJEB90577 peri-rectal swabs left as-is (owner decision); no-panel message wording is site-track but sp_panel_scope now distinguishes the cases. |
| F10 | fixed | three vocabularies; GlobDB flags defined though unused | sandpiper_flag_table.csv (20 rows: field, level, site_label, definition, used) is the single source; sandpiper_flag_definitions.json regenerated from it; GlobDB flags used=false | assert every sp_flag*/class column in run_qc and the sample table is in the flag table — passed. |
| R1-11 | fixed | zip -qr embeds mtimes; VERSION.json 40 tables vs 41 files; no zip hash | python zipfile with fixed date_time (2026-09-26 00:00), sorted entries, external_attr 0644; two independent builds byte-identical (sha256 fe785de73c5bc06a…); VERSION.json lists all 49 files with bytes/sha256/rows and fails on unlisted TABLE_EXT files; VERSION.json written last; zip sha256 in data_package_v1.2.1.sha256 and the sidecar VERSION.json artifact (package_zip) | the in-zip VERSION.json cannot contain its own zip hash; the sidecar carries it. |
| R3-10 | fixed | NFKD drops ø/æ/ß; 25 rows with non-ASCII initials | transliteration map then NFKD, initials folded; 111 rows re-keyed; keys 10,965 → 10,964; authors_index.json regrouped (0 items lost) |  |
| R3-11 | fixed | n_samples_x/_y; class-X studies coverage_class none | n_samples_panel / n_samples_study; sp_coverage_class defined on sample rows (PRJNA1198101, PRJNA613054 → partial with sp_coverage_note) |  |
| R3-12 | fixed | sp_n_runs_total/sp_partial null for 74,733 rows; URL = first run | sp_n_runs_total filled for all rows (from n_runs), sp_partial False when unprofiled (2,457 True); sandpiper_url = RANDOM-selection run with highest root coverage (1,324 URLs changed) |  |
| R3-13 | fixed (documentation) | share denominator and placeholder genera undocumented | dictionary documents named-species-only denominator (use sp_ra_unassigned_species) and placeholder genera g__ECMA0423 / g__G047199095 | the bifidobacterium species table is off-package; no per-sample 'unassigned within genus' row added (not in package scope). |
| R3-14 | fixed | non_wgs_strategy 94; base_count −1 → small_run | strategy first (AMPLICON, Targeted-Capture, WXS = strategies with 0 profiled runs): non_wgs_strategy 511; unknown_size 2; review_amplicon_runs.csv (39 runs, PRJNA61745); study_coverage miss_* recomputed | OTHER/WGA are profiled by Sandpiper (4,104 / 6 profiled runs) so they are not miss reasons. |
| R3-15 | fixed | 'Frontiers Production Office' person; no-initials names; homonym stat mismatch | group regex + office\|production\|editorial (1 row → group); name_needs_check on 3 rows; AUTHORS_REPORT homonym count recomputed 1,075 of 8,181 |  |
| F14 | fixed | v1.1 figure (36 %, 146k) | field_coverage.png regenerated from field_coverage_summary.csv v1.2.1 (body-site vs catalog scope, 73 % age) | site must pick up the new PNG (reports_for_site) — site-track. |
| F15 | partial | 17/22 and 18/22 both stated | README statement kept: 17/22 automated cascade, 18/22 after human review — two different quantities, both defined in the same sentence | the recommendation (template one number in builder for methods.html and README) is a site/builder change; no gold-study table is in the package to compute from. |
| F19 | fixed (footnote variant) | 21 sample + several study columns undocumented; adult_flagged misleading | generated complete column reference with mandatory description (build fails on missing); adult_flagged footnoted as 'over-age flagged (> 1,100 d, includes children)' in age_scope and n_adult_flagged entries | value not renamed (would ripple through site filters); footnote option of the recommendation taken. |
| F22 | fixed | cohort COH0019 sentinel vs study '≤ 407' | cohorts.csv unique_infants_display / _source: 362 single-study cohorts take the study display value; multi-study cohorts get a rule-based string |  |
| F23 | fixed (data/docs) | version relationship unexplained | README, dictionary and run_qc section state: profiles are Sandpiper 2.0.0 outputs from Zenodo snapshot 20419175; linked live pages may show a newer version | methods.html sentence is site-track. |

## Deviations
* R3-7: notebook executed by flattened `exec` of each code cell in one namespace (nbclient could not launch a kernel in this sandbox: PermissionError); outputs are stored in the notebook.
* F3/F6/F9/F14/F15/F23/R3-6: the data-side changes are complete; caption/message/panel wording on the site and the builder templating belong to the site track and are not touched here.
* R3-5 lowercase rule: every single all-lowercase token is tagged (44 distinct incl. some genuine acronyms such as novogene, rivm, hkust) — privacy-first; owner can whitelist via organisations.parquet.
* R3-3: the isolate rule requires library_source = GENOMIC on every run; the 70 METAGENOMIC named-microbe samples (PRJNA799247, PRJNA1082298) are listed for review, not excluded.
* universe_studies_all.parquet carries no organisation column; the universe organisation strings live in study_authors_summary.csv (rebuilt).