feat(elt-common): Add transform functionality

[WHTaylor]

Closes #421

• Makes 'ingest' more distinct, renaming ELTJobManifest -> ELTIngestManifest and runner.py -> ingest.py
• Implements the ability to run transform steps with dbt via elt run

I've tested this locally with the electricity_sharepoint pipeline (using #403), with the entirety of the warehouses/facility_ops/transform directory copied into elt-pipelines, because the DB schema in the landing warehouse hasn't changed. elt run facility_ops electricity_sharepoint or elt run facility_ops electricity_sharepoint --step transform to only run the new step.

Couple of notes:

• I went with the source: selector to avoid relying on the names of the models
  • I think this works well because the names of the sources are determined by the folder structure
• Not a fan of how many dependencies dbt brings in, but I don't think there's anything to do about it
• I've just copied over the existing 'common excludes', though we may want to make these configurable at some point
• I considered making the cli three distinct commands - ingest, transform, and run (which is ingest then transform) - rather than just run with the --step option. No real opinion on which is better.

Follow up work:

• Port facility_ops/transform from warehouses to elt-pipelines
  • Because of schema changes between the ingest pipelines, most of these need to be tweaked
  • Will include docs about the required structure of the transform directory/dbt project
• Add an option to the CLI which gets passed through as the remote argument, to allow this to actually run against prod. This'll require docs about how the profiles in the dbt project need to be set up.
• Potentially add some amount of additional selector options for the models (e.g. the name of the target table, rather than the source), though we may want to just use dbt directly for this

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

The PR adds dbt transformation execution to elt-common, separates ingest and transform runners, updates CLI step selection, and replaces ELTJobManifest with ELTIngestManifest.

ELT orchestration

Layer / File(s)	Summary
Manifest and pipeline model elt-common/src/elt_common/pipeline_types.py, elt-common/src/elt_common/pipeline.py, elt-common/src/elt_common/extract.py, elt-common/tests/unit_tests/test_extract.py, elt-common/tests/unit_tests/test_pipeline.py	The ingest manifest now uses job_dir. PipelinesProject discovers ELTIngestManifest objects and exposes transform_dir. Extraction uses the updated manifest fields.
Ingest and transform runners elt-common/src/elt_common/ingest.py, elt-common/src/elt_common/transform.py, elt-common/tests/unit_tests/test_ingest.py, elt-common/pyproject.toml, elt-common/README.md	Ingest connects to the warehouse named by warehouse_name. The new transform runner invokes dbt with destination selection and optional remote profile arguments. Runtime dependencies and package documentation describe the updated framework.
CLI step orchestration elt-common/src/elt_common/cli.py	The CLI resolves one ingest manifest, runs ingest and transform steps according to StepOption, logs timing, and reports dbt failures.

Possibly related issues

• #421 — The PR implements the requested transform step in the elt command, including dbt argument construction.
• #321 — The PR supports the unified ingest and transform command described by this issue.

Possibly related PRs

• ISISNeutronMuon/analytics-data-platform#334 — Introduced the CLI and pipeline manifest APIs that this PR extends.
• ISISNeutronMuon/analytics-data-platform#352 — Added runner functionality that this PR moves into the ingest module and integrates with transforms.
• ISISNeutronMuon/analytics-data-platform#420 — Added related dbt transformation support in elt-common.

Suggested reviewers: martyngigg

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: adding transform functionality to elt-common.
Description check	✅ Passed	The description explains the transform step, manifest renaming, CLI usage, testing, and planned follow-up work.
Linked Issues check	✅ Passed	The changes implement dbt transform execution, source-based model selection, a transform directory, and full or transform-only CLI runs for issue #421.
Out of Scope Changes check	✅ Passed	The manifest renaming and ingest module changes support the stated transform functionality and do not show unrelated scope.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

elt-common/src/elt_common/transform.py (1)

11-27: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add pytest coverage for the transform execution flow.

Mock dbtRunner, run_ingest, and run_transform. Assert the dbt selector, remote-profile arguments, step order for all, transform-only execution, and non-success exit handling.

• elt-common/src/elt_common/transform.py#L11-L27: Test command construction and dbtRunner.invoke() arguments.
• elt-common/src/elt_common/cli.py#L58-L86: Test all, ingest, and transform step selection.
• elt-common/src/elt_common/cli.py#L110-L131: Test transform failure reporting and exit status.

As per coding guidelines, write unit and e2e tests for the elt-common package using pytest and place them under elt-common/tests/.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@elt-common/src/elt_common/transform.py` around lines 11 - 27, Add pytest unit
and end-to-end coverage under elt-common/tests/: for
elt-common/src/elt_common/transform.py lines 11-27, mock dbtRunner and verify
dbt selector construction, remote profile arguments, and dbtRunner.invoke()
inputs; for elt-common/src/elt_common/cli.py lines 58-86, mock run_ingest and
run_transform and verify step selection and ordering for all, ingest-only, and
transform-only flows; for elt-common/src/elt_common/cli.py lines 110-131, verify
non-success transform results are reported and produce the expected exit status.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@elt-common/src/elt_common/transform.py`:
- Around line 11-27: Add pytest unit and end-to-end coverage under
elt-common/tests/: for elt-common/src/elt_common/transform.py lines 11-27, mock
dbtRunner and verify dbt selector construction, remote profile arguments, and
dbtRunner.invoke() inputs; for elt-common/src/elt_common/cli.py lines 58-86,
mock run_ingest and run_transform and verify step selection and ordering for
all, ingest-only, and transform-only flows; for elt-common/src/elt_common/cli.py
lines 110-131, verify non-success transform results are reported and produce the
expected exit status.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d676ad60-0504-473a-8049-121d84c1fcab

📥 Commits

Reviewing files that changed from the base of the PR and between af04e9e and 107a019.

⛔ Files ignored due to path filters (2)

• elt-common/uv.lock is excluded by !**/*.lock
• elt-pipelines/uv.lock is excluded by !**/*.lock

📒 Files selected for processing (11)

• elt-common/README.md
• elt-common/pyproject.toml
• elt-common/src/elt_common/cli.py
• elt-common/src/elt_common/extract.py
• elt-common/src/elt_common/ingest.py
• elt-common/src/elt_common/pipeline.py
• elt-common/src/elt_common/pipeline_types.py
• elt-common/src/elt_common/transform.py
• elt-common/tests/unit_tests/test_extract.py
• elt-common/tests/unit_tests/test_ingest.py
• elt-common/tests/unit_tests/test_pipeline.py