AREE — Aquaculture Resilience Evidence Engine

AREE is an open, reproducible research resource that converts fragmented public omics datasets into harmonized, comparable resilience-biomarker evidence for Pacific oyster (Crassostrea gigas), and is deliberately extensible to other shellfish and aquaculture organisms.

It operationalizes Objective 1 of the AREE proposal:

Develop standardized open-access, user-friendly, reproducible bioinformatics pipelines for resilience biomarker discovery through systematic reanalysis, data integration, and meta-analysis.

AREE is an evidence engine, not a claims engine. It identifies associations and evidence convergence across studies and molecular layers — never confirmed mechanistic causation — and it never presents a statistically significant hit in a single study as a validated biomarker.

All datasets shipped in this repository are clearly labeled SIMULATED demo data. They exist to exercise the full pipeline end-to-end. Real public datasets can be added with minimal modification (see docs/adding_a_study.md).

The five layers

1. Study registry & intake (registry/, src/intake/) — machine-readable dataset registration with controlled vocabularies for phenotypes and stressors, plus schema validation.
2. Standardized reanalysis workflows (workflows/, modules/) — Nextflow DSL2 scaffolds for RNA-seq, methylation, proteomics, and metabolomics, in raw-reanalysis or processed-results modes.
3. Cross-study harmonization (src/harmonize/) — a shared, assay-agnostic evidence schema plus identifier mapping with explicit confidence levels.
4. Meta-analysis & candidate prioritization (src/meta_analysis/, src/prioritize/) — random-effects pooling, heterogeneity statistics, and a transparent (non-black-box) candidate score with hard tier gates.
5. User-facing outputs (app/, docs/, src/reporting/) — a Streamlit interface, a Quarto documentation site, and per-candidate evidence cards.

Installation

Requires Python ≥ 3.9.

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,app]"

# Add `intake` if you will convert published .xls/.xlsx supplementary tables:
#   pip install -e ".[dev,app,intake]"

See docs/installation.md for details.

Quick start (runs entirely on bundled demo data)

# 1. Validate a study registration file
aree validate-study registry/studies/GIGAS_HEAT01.yaml

# 2. Register every demo study
for f in registry/studies/GIGAS_*.yaml; do aree register-study "$f"; done
aree list-studies

# 3. Harmonize each study into the shared evidence table
for sid in GIGAS_HEAT01 GIGAS_OA02 GIGAS_PATH03 GIGAS_SAL04 GIGAS_LARV05 GIGAS_GROW06; do
  aree harmonize --study "$sid"
done

# (or harmonize a single processed results table)
aree harmonize --study GIGAS_SAL04 \
  --input data/demo/proteomics/GIGAS_SAL04_low_salinity_vs_control_protein_abundance_demo.tsv

# 4. Run a meta-analysis
aree meta-analyze --phenotype thermal_tolerance --feature-type gene
aree meta-analyze --feature-type gene

# 5. Generate biomarker evidence cards
aree build-evidence-cards --phenotype thermal_tolerance

# 6. Build the docs site / launch the interface
quarto render docs
streamlit run app/main.py

Or run the whole demo in one step:

make demo

Repository structure

AREE/
├── README.md, LICENSE, CITATION.cff, CONTRIBUTING.md, CODE_OF_CONDUCT.md
├── pyproject.toml, Makefile
├── docs/                      # Quarto site + narrative + how-to documentation
├── schemas/                   # JSON Schema: study.schema.json, evidence.schema.json
├── registry/
│   ├── studies/               # per-study YAML registrations (+ templates)
│   ├── controlled_vocabularies/  # phenotype, stressor, tissue, life-stage, etc.
│   └── study_registry.csv     # flat index (generated by `aree register-study`)
├── workflows/                 # Nextflow DSL2 scaffolds (rnaseq/methylation/proteomics/metabolomics)
├── modules/                   # reusable Nextflow process modules per assay
├── containers/                # container image strategy (documentation)
├── config/                    # shared Nextflow config (base + demo)
├── data/
│   ├── demo/                  # SIMULATED demo result tables per assay
│   ├── reference/             # genome/annotation metadata
│   │   └── crosswalk/         # REAL identifier crosswalk + provenance sidecar
│   └── mappings/              # SYNTHETIC demo crosswalk + ambiguous-symbol map
├── src/
│   ├── common/                # shared paths, IO, vocabulary loaders
│   ├── intake/                # schema validation + registry ingestion
│   ├── harmonize/             # per-assay harmonizers -> shared evidence schema
│   ├── meta_analysis/         # random-effects pooling + heterogeneity
│   ├── prioritize/            # transparent scoring + tier gating
│   ├── reporting/             # evidence cards + provenance manifests
│   ├── mappings/              # builds real crosswalks from NCBI Gene + UniProtKB
│   ├── validation/            # reusable validation checks
│   └── aree/                  # the `aree` CLI
├── app/                       # Streamlit interface
├── reports/                   # generated outputs (gitignored)
├── tests/                     # pytest suite
└── .github/workflows/         # CI

Command reference

Command	Purpose
aree validate-study <file>	Validate a study YAML against schema + controlled vocabularies
aree register-study <file> [--update]	Add (or update) a study in the registry
aree fetch-samplesheet --bioproject <acc> --study <id>	Build a sample sheet + checksummed FASTQ manifest from a BioProject's deposited metadata
aree intake-supplementary <config> [--check]	Convert a published supplementary table into AREE result files; --check verifies committed files still reproduce
aree list-studies	List registered studies and their pipeline status
aree harmonize --study <id> [--input <file>]	Harmonize a study (or one processed table) into the evidence table
aree meta-analyze [--phenotype <p>] [--feature-type <t>]	Random-effects meta-analysis over the evidence table
aree build-evidence-cards [--phenotype <p>] [--feature-type <t>]	Generate per-candidate evidence cards
aree build-crosswalk [--taxid <n>]	Build a real identifier crosswalk from NCBI Gene + UniProtKB

Working with real data

The quick start above runs on simulated studies and a synthetic crosswalk. AREE also ships one real registered study — HESSER2024_VCOR, curated from the open-access supplementary tables of Hesser et al. 2024 (Vibrio coralliilyticus challenge of C. gigas larvae). Harmonizing it requires the real crosswalk:

export AREE_CROSSWALK=data/reference/crosswalk/mgigas_gene_id_crosswalk.tsv
aree harmonize --study HESSER2024_VCOR

The result tables it harmonizes are themselves derived from the published supplementary spreadsheet by a committed, re-runnable intake config — no manual copy-paste step sits between the publication and the evidence:

aree intake-supplementary data/studies/HESSER2024_VCOR/intake.yaml --check

--check regenerates the tables into a temporary directory and compares checksums against both the committed files and the committed provenance, so a hand-edited result table or a swapped source artifact fails loudly. CI runs this on every push. Drop --check to actually rewrite the files.

87.2% of its published identifiers resolve (274 exact, 32 inferred via NCBI's retired-GeneID remapping); every unresolved identifier is a gene NCBI discontinued without a replacement. Read docs/first_real_study.md before curating your own — it records what broke the first time real data hit the pipeline.

Real and simulated evidence are kept strictly apart: simulated is a column in the evidence schema and part of the meta-analysis grouping key, and each study selects its crosswalk from its own simulated flag, so a real study will refuse to run against the demo crosswalk.

Rebuild it only when the NCBI annotation changes (streams ~230 MB, 15-20 min):

aree build-crosswalk

The demo and real crosswalks are deliberately never merged — the demo's LOC numbers collide with real NCBI GeneIDs that denote different genes. See docs/identifier_mapping.md for the coverage figures and their consequences, in particular that UniProt links only 8.4% of M. gigas genes, so proteomics evidence carries a much higher unresolved rate than transcriptomics evidence. Identifiers retired by NCBI re-annotation (9,057 of them) resolve to their current replacement as inferred; the 13,601 discontinued with no replacement stay unresolved by design.

Documentation

Start with docs/index.qmd or render the site with quarto render docs. Key pages:

• Why this resource matters
• Design document · Technical architecture
• Adding a study · Identifier mapping
• Interpreting meta-analysis · Interpreting candidate scores
• Governance & provenance · Roadmap

Implementation status

This is a functioning MVP. See docs/roadmap.md for a candid breakdown of what is complete and runnable (schemas, registry, harmonization for all four assay types, meta-analysis, prioritization, evidence cards, CLI, Streamlit app, Quarto docs, tests, CI) versus scaffolded but not yet production-ready (the Nextflow raw-data workflows, which are structurally complete but have not been executed against real sequencing data in this build) versus planned (real ortholog mapping, additional species, hosted deployment).

License

MIT — see LICENSE.

Citation

See CITATION.cff.