diff --git a/contrib/dbt_factory/CONTRIBUTING.md b/contrib/dbt_factory/CONTRIBUTING.md new file mode 100644 index 0000000..d43e12d --- /dev/null +++ b/contrib/dbt_factory/CONTRIBUTING.md @@ -0,0 +1,110 @@ +# Contributing to dbt-factory + +Thanks for contributing! This guide covers the **dbt-factory** template +([`contrib/templates/dbt-factory`](../templates/dbt-factory)) and this ready-to-run example +([`contrib/dbt_factory`](.)). Most development happens from this example directory — it has the +`Makefile`, the unit tests, and the end-to-end test. + +## First principles + +- Keep a generated project **self-contained and dependency-light** — it should run with just `uv` + and the Databricks CLI, with no per-model YAML to maintain. +- The factory core under `src/databricks_dbt_factory/` is **vendored**, not developed here — see + [Vendored factory](#vendored-factory). +- **Serverless is the default** compute and the **job cluster** path is opt-in; keep both working. + +## Local setup + +Requirements: + +- Python (`>=3.10,<3.13`, per `pyproject.toml`) +- The [`uv`](https://docs.astral.sh/uv/) package manager +- The [Databricks CLI](https://docs.databricks.com/dev-tools/cli/databricks-cli.html) + +Install the development environment: + +``` +$ make setup # == uv sync --dev (into .venv, from the frozen uv.lock) +``` + +Prefer the `make` targets over calling `uv`/`dbt`/`databricks` directly where one exists: +`make manifest`, `make validate`, `make deploy`, `make run`, `make test`, `make test-e2e`. + +## Before you open a PR (Definition of Done) + +Run these locally — they mirror the CI gates plus the manual checks: + +1. **Format** — `uv run ruff format .` The `fmt` CI runs `ruff format --check`, so unformatted code + fails the build. +2. **Unit tests** — `make test` (offline; no workspace needed). This is the fast gate. +3. **End-to-end** — `make test-e2e` deploys, runs, verifies, and tears down a real job for **both** + compute modes (serverless and job cluster). It is **not** a CI gate, so run it locally before + merging any change to the factory or template. It needs `DATABRICKS_*` / `DBT_FACTORY_*` + environment variables — see [`tests/e2e/README.md`](tests/e2e/README.md). +4. **Snapshot** — if you intentionally changed the generated task output, refresh the saved snapshot + with `make test-update-expected-tasks` and commit the result. +5. **Docs** — update the relevant README(s) for any user-facing change. + +## Testing + +- **`make test`** — runs the vendored factory's unit tests plus an offline test of the PyDABs + integration against the committed manifest; no workspace is required. One test compares the + generated tasks with `tests/test_data/expected_tasks.json`, so unintended changes to the generated + job fail the suite. +- **`make test-e2e`** — the real end-to-end check (generate → deploy → run → verify → destroy) for + both compute modes. See [`tests/e2e/README.md`](tests/e2e/README.md) for the required environment + and details. + +### Scaffolding check + +The unit tests run against the committed (pre-rendered) example, so they do **not** exercise the +template's prompts or rendering. When you change the template, scaffold a throwaway project with +`databricks bundle init` to confirm it still scaffolds correctly — do it once per compute mode +(`use_serverless = yes` and `= no`): + +``` +# from a neutral directory (not inside a bundle), pointing at the template: +$ databricks bundle init /contrib/templates/dbt-factory --output-dir /tmp/dbt-factory-check +$ cd /tmp/dbt-factory-check/ +$ make setup && make manifest && make test +``` + +This needs a configured Databricks CLI profile — the template resolves workspace values (such as +`smallest_node_type` and `workspace_host`) at init time. `make test-e2e` already runs `bundle init` +for both modes end-to-end, but this quick check catches rendering/prompt errors without a full +deploy. + +## Keeping the template and example in sync + +The template ([`contrib/templates/dbt-factory`](../templates/dbt-factory)) and this example are two +copies of the same project (the template's files are `*.tmpl`; this example is a rendered, serverless +copy). When you change a shared file — the factory core, `resources/__init__.py`, the tests, the +`Makefile`, `dbt_profiles/profiles.yml` — update **both** so the template's serverless rendering +still matches this example. + +### Vendored factory + +`src/databricks_dbt_factory/` is vendored from +[databricks-dbt-factory](https://github.com/mwojtyczka/databricks-dbt-factory) and pinned to the +commit recorded in [`NOTICE`](NOTICE). The `dbt-factory vendor-sync` CI check enforces that: + +- each vendored file is byte-identical to that upstream commit, and +- the example and template copies are byte-identical to each other. + +So **don't edit the factory here** — make the change upstream (see the factory's +[contributing guide](https://github.com/mwojtyczka/databricks-dbt-factory/blob/main/CONTRIBUTING.md), +which uses its own Hatch-based `make dev`/`fmt`/`lint`/`test`/`integration` workflow), then re-vendor +into both copies and update the commit/version in both `NOTICE` files. + +## Opening the PR + +Unless you have write access to +[`databricks/bundle-examples`](https://github.com/databricks/bundle-examples), work from a **fork**: + +1. Fork the repo and clone your fork (keep its `main` in sync with upstream). If you have write + access, you can branch directly in the upstream repo instead. +2. Create a feature branch from `main`, make your change, and run the Definition-of-Done checks + above. +3. Push the branch (to your fork, or upstream if you have access) and open a PR against + `databricks/bundle-examples` `main`, with a clear description and `Resolves #NNN` if it fixes an + issue. diff --git a/contrib/dbt_factory/README.md b/contrib/dbt_factory/README.md index 45b193a..37eaae0 100644 --- a/contrib/dbt_factory/README.md +++ b/contrib/dbt_factory/README.md @@ -1,7 +1,7 @@ # dbt_factory This example runs a [dbt](https://docs.getdbt.com/) project on Databricks as a -**Databricks Workflow with one task per dbt object** (model, seed, snapshot, test) instead of +**Databricks Lakeflow Job with one task per dbt object** (model, seed, snapshot, test) instead of running the whole project as a single opaque task. It does this by combining two pieces: @@ -20,13 +20,13 @@ the dbt manifest each time you deploy. ## Why one task per dbt object? -By default dbt's integration with Databricks Workflows treats the whole project as a single +By default dbt's integration with Databricks Lakeflow Jobs treats the whole project as a single task — a black box. Expanding it into one task per object gives: * **Faster execution** — independent models run in parallel, and the notebook task type runs dbt from a pre-built serverless base environment, avoiding a dependency install on every task. * **Visibility & simplified troubleshooting** — pinpoint and fix issues at the model level right - in the Databricks Workflows UI. + in the Databricks Lakeflow Jobs UI. * **Enhanced logging & notifications** — per-task logs and precise, model-level error alerts. * **Improved retriability** — retry only the failed model tasks without rerunning the whole project. * **Seamless testing** — dbt data tests run as their own tasks right after each model finishes, @@ -40,7 +40,7 @@ for more. ## How it works The [`dbt-factory` template](../templates/dbt-factory) scaffolds a self-contained project. -From then on, each `databricks bundle deploy` regenerates the Workflow from your current dbt +From then on, each `databricks bundle deploy` regenerates the job from your current dbt manifest — add or remove a model and the task graph follows on the next deploy, with no per-model YAML to maintain. @@ -56,7 +56,7 @@ flowchart TD E --> F["PyDABs load_resources reads the
manifest and generates the job"] end subgraph runtime["At run time — serverless"] - G["Databricks Workflow:
one task per model / seed / snapshot / test"] --> H["Each task triggers dbt
via the runner notebook"] + G["Databricks Lakeflow Job:
one task per model / seed / snapshot / test"] --> H["Each task triggers dbt
via the runner notebook"] H --> I[("SQL warehouse")] end B --> C @@ -169,55 +169,6 @@ The version is shipped as a small `dbt_serverless_env.yaml` [base environment](h that the bundle generates and syncs on every deploy (git-ignored), so Databricks pre-builds the environment once instead of installing dbt on every task. -## Migrating an existing dbt project - -Bring your own dbt project by **generating a fresh project from the template and moving your dbt -files into it.** You don't touch dependencies, the vendored factory, or any paths — the generated -project already ships all of that. - -1. Generate a new project (or copy this `dbt_factory` example): - - ``` - $ databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/dbt-factory - ``` - -2. Remove the starter models and copy your dbt sources into the matching `src/` subdirectories: - - ``` - $ rm -r src/models/example - # Copy whichever of these your project has (skip the ones you don't use): - $ cp -R /path/to/your/dbt/models/* src/models/ - $ cp -R /path/to/your/dbt/seeds/* src/seeds/ - $ cp -R /path/to/your/dbt/snapshots/* src/snapshots/ - $ cp -R /path/to/your/dbt/macros/* src/macros/ - $ cp -R /path/to/your/dbt/tests/* src/tests/ - ``` - - The generated `dbt_project.yml` already points `model-paths`, `seed-paths`, etc. at these - `src/` folders, so your files are picked up as-is. Merge any model/seed configuration from your - own `dbt_project.yml` into the generated one (keep the generated `name`/`profile`), and remove - the leftover `models: dbt_factory: example:` block that referenced the deleted starter models — - otherwise `dbt parse` warns that those config paths don't apply to any resource. If you use dbt - packages, copy your `packages.yml` to the project root too: `make manifest` installs them - (`dbt deps`) and the bundle syncs the resulting `dbt_packages/` to the workspace, so the job - never installs packages at runtime. - -3. Point `dbt_profiles/profiles.yml` at your warehouse (`http_path`, `catalog`, `schema`). Leave - the `host`/`token` lines as they are — the runner notebook sets those at runtime. - -4. Generate the manifest and deploy: - - ``` - $ make setup - $ make manifest # dbt parse -> target/manifest.json - $ databricks bundle deploy --target dev - ``` - -That's the whole migration: no dependency wrangling and no path edits, because your project keeps -the generated layout (dbt project at the bundle root, factory under `src/`). If you'd rather keep -your project's existing directory structure instead of `src/`, edit the `*-paths` in -`dbt_project.yml` to point at your folders — nothing else changes. - ## Tests ``` diff --git a/contrib/dbt_factory/tests/e2e/README.md b/contrib/dbt_factory/tests/e2e/README.md index e134a3b..63fbeba 100644 --- a/contrib/dbt_factory/tests/e2e/README.md +++ b/contrib/dbt_factory/tests/e2e/README.md @@ -1,11 +1,12 @@ # End-to-end test `make test-e2e` (from the example root) is the check that a change to the factory won't break real dbt -execution. It generates a fresh project from the [`dbt-factory` template](../../../templates/dbt-factory) -with the factory's default options, points it at **your** Databricks workspace, drops in the fixture -dbt project in this directory, then **deploys the factory-generated job, runs it, verifies the -output, and tears everything down**. Every run destroys its bundle and drops its schema, pass or -fail, so nothing is left behind. +execution. It generates a fresh project from the [`dbt-factory` template](../../../templates/dbt-factory), +points it at **your** Databricks workspace, drops in the fixture dbt project in this directory, then +**deploys the factory-generated job, runs it, verifies the output, and tears everything down**. It +does this once for **each compute mode** — serverless and a job cluster — so both renderings of the +template are deployed and executed for real. Every run destroys its bundle and drops its schema, pass +or fail, so nothing is left behind. Unlike `make test` (fast, offline unit tests), this one deploys and runs on a real workspace — so it's not a CI gate; run it locally before merging a change to the factory. @@ -14,6 +15,10 @@ it's not a CI gate; run it locally before merging a change to the factory. - A Databricks CLI profile for your workspace: `databricks auth login --host `. - A SQL warehouse, and a catalog you can create schemas/tables in. +- Permission to **create clusters** (the cluster-create entitlement) — the job-cluster mode + provisions a single-node job cluster. The serverless mode doesn't need this; without the + entitlement the serverless half passes and the job-cluster half fails with a `PERMISSION_DENIED` + cluster-creation error. ## Run it diff --git a/contrib/dbt_factory/tests/e2e/run_e2e.py b/contrib/dbt_factory/tests/e2e/run_e2e.py index ceeb560..7da9a33 100644 --- a/contrib/dbt_factory/tests/e2e/run_e2e.py +++ b/contrib/dbt_factory/tests/e2e/run_e2e.py @@ -1,10 +1,11 @@ """ End-to-end test for the dbt-factory example. -It generates a fresh project from the ``dbt-factory`` template with the factory's default options, -points it at YOUR workspace, drops in the fixture dbt project next to this file, then **deploys the -factory-generated job, runs it, and verifies it succeeded** — before tearing everything down. This -is the check that a change to the factory won't break real dbt execution. +It generates a fresh project from the ``dbt-factory`` template, points it at YOUR workspace, drops +in the fixture dbt project next to this file, then **deploys the factory-generated job, runs it, and +verifies it succeeded** — before tearing everything down. It runs the whole flow once per **compute +mode** (serverless and a job cluster), so a change to the factory is exercised end-to-end on both. +This is the check that a change to the factory won't break real dbt execution. Run it with ``make test-e2e`` from the example root. Required environment: @@ -12,6 +13,10 @@ DBT_FACTORY_HTTP_PATH SQL warehouse HTTP path, e.g. /sql/1.0/warehouses/. DBT_FACTORY_CATALOG Catalog to create the throwaway schema + tables in (write access). +The job-cluster mode additionally needs permission to create clusters (the cluster-create +entitlement) — the serverless mode does not. Without it, the serverless half passes and the +job-cluster half fails at deploy/run with a PERMISSION_DENIED cluster-creation error. + Optional: DBT_FACTORY_SCHEMA_PREFIX Schema-name prefix (default: dbt_factory_e2e). A unique _ schema is created and dropped per run. @@ -108,18 +113,19 @@ def _sql(cfg: Config, statement: str, schema: str | None = None) -> list[list]: return payload.get("result", {}).get("data_array", []) or [] -def _init_project(cfg: Config, out_dir: Path, schema: str) -> Path: - # Set only the values that point the project at this workspace. Everything else (test bundling, - # environment key, extra dbt options) is left at the template's defaults — the factory's - # out-of-the-box behavior, which is exactly what we want the e2e to exercise. +def _init_project(cfg: Config, out_dir: Path, schema: str, use_serverless: str, project_name: str) -> Path: + # Set the values that point the project at this workspace, plus the compute mode under test + # (use_serverless). Everything else (test bundling, environment key, extra dbt options) is left + # at the template's defaults — the factory's out-of-the-box behavior, which is what we exercise. config_file = out_dir / "init-config.json" config_file.write_text( json.dumps( { - "project_name": PROJECT_NAME, + "project_name": project_name, "default_catalog": cfg.catalog, "dev_schema": schema, "http_path": cfg.http_path, + "use_serverless": use_serverless, } ) ) @@ -138,7 +144,7 @@ def _init_project(cfg: Config, out_dir: Path, schema: str) -> Path: ], cwd=out_dir, ) - return out_dir / PROJECT_NAME + return out_dir / project_name def _fill_fixture(project: Path) -> None: @@ -200,17 +206,20 @@ def _verify_output(cfg: Config, schema: str) -> list[str]: return failures -def run(cfg: Config) -> bool: - schema = f"{cfg.prefix}_{time.strftime('%Y%m%d_%H%M%S')}" - job = f"{PROJECT_NAME}_job" +def run(cfg: Config, label: str, use_serverless: str) -> bool: + schema = f"{cfg.prefix}_{label}_{time.strftime('%Y%m%d_%H%M%S')}" + # Each mode gets its own project (and therefore its own bundle path + job name), so the two runs + # are fully isolated — a silently-failed teardown of one can't collide with the other's deploy. + project_name = f"{PROJECT_NAME}_{label}" + job = f"{project_name}_job" work = Path(tempfile.mkdtemp(prefix="dbtfactory_e2e_")) project = None - print(f"\n===== dbt-factory e2e — schema {cfg.catalog}.{schema} =====") + print(f"\n===== dbt-factory e2e [{label}] — schema {cfg.catalog}.{schema} =====") try: print(" [1/6] create throwaway schema") _sql(cfg, f"CREATE SCHEMA IF NOT EXISTS {cfg.catalog}.{schema}") print(" [2/6] init project from template + drop in fixture") - project = _init_project(cfg, work, schema) + project = _init_project(cfg, work, schema, use_serverless, project_name) _fill_fixture(project) print(" [3/6] install deps + generate dbt manifest") _run(["uv", "sync", "--dev"], cwd=project) @@ -248,9 +257,17 @@ def run(cfg: Config) -> bool: def main() -> None: - passed = run(Config()) - print(f"\n===== e2e {'PASS' if passed else 'FAIL'} =====") - if not passed: + cfg = Config() + # Run the whole flow once per compute mode so both the serverless and the job-cluster + # rendering of the template are deployed and executed for real. + results = { + label: run(cfg, label, use_serverless) + for label, use_serverless in (("serverless", "yes"), ("job_cluster", "no")) + } + print("\n===== e2e summary =====") + for label, ok in results.items(): + print(f" {label:12} {'PASS' if ok else 'FAIL'}") + if not all(results.values()): sys.exit(1) diff --git a/contrib/dbt_factory/tests/test_load_resources.py b/contrib/dbt_factory/tests/test_load_resources.py index aa10276..648a421 100644 --- a/contrib/dbt_factory/tests/test_load_resources.py +++ b/contrib/dbt_factory/tests/test_load_resources.py @@ -94,8 +94,8 @@ def test_load_resources_registers_the_job(): def test_dependency_pin_rejects_non_pypi_versions(monkeypatch): - # A local or dev build cannot be pip-installed from PyPI when Databricks builds the - # serverless environment, so the deploy must fail early with a clear message. + # A local or dev build cannot be pip-installed from PyPI, so the deploy must fail early + # with a clear message. for installed in ("1.9.0+custom", "1.13.0.dev0", "not-a-version"): monkeypatch.setattr(resources, "version", lambda name, v=installed: v) with pytest.raises(RuntimeError, match="PyPI"): diff --git a/contrib/templates/dbt-factory/README.md b/contrib/templates/dbt-factory/README.md index 1632778..639df99 100644 --- a/contrib/templates/dbt-factory/README.md +++ b/contrib/templates/dbt-factory/README.md @@ -1,9 +1,9 @@ # dbt-factory template A [Declarative Automation Bundles](https://docs.databricks.com/dev-tools/bundles/index.html) template -that generates a [dbt](https://docs.getdbt.com/) project whose Databricks Workflow is built +that generates a [dbt](https://docs.getdbt.com/) project whose Databricks Lakeflow Job is built **from the dbt manifest at deploy time** — one Databricks task per dbt object (model, seed, -snapshot, test), running on serverless compute. +snapshot, test), running on serverless compute by default (or on a job cluster). It wires together two pieces: @@ -30,7 +30,7 @@ For a pre-initialized, ready-to-read version of what this template produces, see ## How it works `databricks bundle init` scaffolds a self-contained project; each `databricks bundle deploy` then -regenerates the Workflow from your current dbt manifest, so adding or removing a model just works +regenerates the job from your current dbt manifest, so adding or removing a model just works on the next deploy — no per-model YAML to maintain. ```mermaid @@ -45,7 +45,7 @@ flowchart TD E --> F["PyDABs load_resources reads the
manifest and generates the job"] end subgraph runtime["At run time — serverless"] - G["Databricks Workflow:
one task per model / seed / snapshot / test"] --> H["Each task triggers dbt
via the runner notebook"] + G["Databricks Lakeflow Job:
one task per model / seed / snapshot / test"] --> H["Each task triggers dbt
via the runner notebook"] H --> I[("SQL warehouse")] end B --> C @@ -57,6 +57,10 @@ flowchart TD ## Usage +This scaffolds a **new** project preloaded with the template's default example — a couple of starter +models you replace with your own. (Already have a dbt project? See +[Already have a dbt project?](#already-have-a-dbt-project) below.) + ``` $ databricks bundle init https://github.com/databricks/bundle-examples --template-dir contrib/templates/dbt-factory ``` @@ -81,15 +85,82 @@ $ databricks bundle run _job | `dev_schema` | Schema for the `dev` target (`prod` uses `default`). | | `http_path` | HTTP path of the SQL warehouse dbt connects to. | | `bundle_tests` | Bundle single-model tests per resource into one task (performance boost). | -| `environment_key` | Key of the serverless environment used by the generated job. | +| `use_serverless` | Run the job on serverless compute (default). Choose `no` to run on a job cluster (classic compute) defined in `resources/__init__.py`, which you can size to your workload. | +| `environment_key` | Key of the serverless environment used by the generated job (serverless only). | | `extra_dbt_command_options` | Extra options appended to every generated dbt command. | +### Compute + +The generated job supports two compute types, chosen by the `use_serverless` prompt: + +- **Serverless (default)** — no cluster to manage. +- **Job cluster (classic)** — a single job cluster, created once per run and **shared by all + tasks**, then torn down when the run finishes. It only orchestrates (queries run on your SQL + warehouse), so it defaults to a **single node** (extra workers don't speed dbt up). You can size + it by editing the `job_clusters` block in the generated `resources/__init__.py` before deploying + (see the generated project's README, "Sizing the job cluster"). + +Running on an existing (all-purpose) cluster is **not supported**. + ## Already have a dbt project? -This template scaffolds a new project. To reuse an **existing** dbt project, generate a project -from this template and move your dbt files into it — see the -["Migrating an existing dbt project"](../../dbt_factory/README.md#migrating-an-existing-dbt-project) -guide in the example. You won't need to bring any dependencies or edit paths. +Bring your own dbt project by generating a fresh project (as in [Usage](#usage) above) and moving +your dbt files into it. You don't touch dependencies, the vendored factory, or any paths — the +generated project already ships all of that. + +1. Remove the starter models and copy your dbt sources into the matching `src/` subdirectories: + + ``` + $ rm -r src/models/example + # Copy whichever of these your project has (skip the ones you don't use): + $ cp -R /path/to/your/dbt/models/* src/models/ + $ cp -R /path/to/your/dbt/seeds/* src/seeds/ + $ cp -R /path/to/your/dbt/snapshots/* src/snapshots/ + $ cp -R /path/to/your/dbt/macros/* src/macros/ + $ cp -R /path/to/your/dbt/tests/* src/tests/ + ``` + + The generated `dbt_project.yml` already points `model-paths`, `seed-paths`, etc. at these `src/` + folders, so your files are picked up as-is. Merge any model/seed configuration from your own + `dbt_project.yml` into the generated one (keep the generated `name`/`profile`), and remove the + leftover `models:` block that referenced the deleted starter models — otherwise `dbt parse` warns + that those config paths don't apply to any resource. If you use dbt packages, copy your + `packages.yml` to the project root too: `make manifest` installs them (`dbt deps`) and the bundle + syncs the resulting `dbt_packages/` to the workspace, so the job never installs packages at + runtime. + +2. Point `dbt_profiles/profiles.yml` at your warehouse (`http_path`, `catalog`, `schema`). Leave the + `host`/`token` lines as they are — the runner notebook sets those at runtime. + +3. Generate the manifest, deploy, and run: + + ``` + $ make setup + $ make manifest # dbt parse -> target/manifest.json + $ databricks bundle deploy --target dev + $ databricks bundle run _job + ``` + +That's the whole migration: no dependency wrangling and no path edits, because your project keeps the +generated layout (dbt project at the bundle root, factory under `src/`). To keep your existing +directory structure instead of `src/`, edit the `*-paths` in `dbt_project.yml` to point at your +folders — nothing else changes. + +## Do developers switch to Jobs, or stay in dbt? + +Developers continue working natively in dbt — the factory operates on the deployment side. Lakeflow +Jobs still runs the dbt project directly; instead of executing it as an opaque black box, the +factory decomposes the run into discrete, observable task nodes with per-model retries and logs. + +## Are dbt Jinja and templating converted to dynamic parameters? + +No — there are no dynamic Databricks parameters. The manifest is pre-compiled (`make manifest`), so +dbt Jinja (`ref()`, `source()`, `var()`) is resolved before deploy into static commands +(`dbt run --select my_model`). To vary behavior, use dbt's own mechanisms: + +- **`vars:` in `dbt_project.yml`** — baked into the manifest ahead of time, for deterministic runs. +- **`profiles.yml` targets** — selected per deploy target (dev/prod) for per-environment differences. +- **`env_var()`** — for runtime values that do not alter the graph topology. See https://github.com/databricks/bundle-examples/blob/main/contrib/README.md for more about community contributions. diff --git a/contrib/templates/dbt-factory/databricks_template_schema.json b/contrib/templates/dbt-factory/databricks_template_schema.json index 53d615f..fc70254 100644 --- a/contrib/templates/dbt-factory/databricks_template_schema.json +++ b/contrib/templates/dbt-factory/databricks_template_schema.json @@ -1,5 +1,5 @@ { - "welcome_message": "\nWelcome to the dbt-factory template for Declarative Automation Bundles!\nThis generates a dbt project whose Databricks Workflow is built from the dbt manifest at deploy time — one task per dbt object (model, seed, snapshot, test), on serverless.", + "welcome_message": "\nWelcome to the dbt-factory template for Declarative Automation Bundles!\nThis generates a dbt project whose Databricks Lakeflow Job is built from the dbt manifest at deploy time — one task per dbt object (model, seed, snapshot, test), on serverless compute (the default) or a job cluster.", "properties": { "project_name": { "type": "string", @@ -38,19 +38,33 @@ "description": "\nBundle single-model tests per resource into one dbt test task? This is a performance boost for projects with many tests (fewer Databricks task startups), at the cost of per-test visibility in the UI.\nbundle_tests", "order": 5 }, + "use_serverless": { + "type": "string", + "default": "yes", + "enum": ["yes", "no"], + "description": "\nRun the generated job on serverless compute? Choose 'no' to run on a job cluster (classic compute) defined in resources/__init__.py, which you can then size to your workload.\nuse_serverless", + "order": 6 + }, "environment_key": { "type": "string", "default": "Default", "pattern": "^\\w+$", "pattern_match_failure_message": "Invalid environment key.", + "skip_prompt_if": { + "properties": { + "use_serverless": { + "const": "no" + } + } + }, "description": "\nKey of the serverless environment used by the generated job.\nenvironment_key", - "order": 6 + "order": 7 }, "extra_dbt_command_options": { "type": "string", "default": "", "description": "\nAdditional options appended to every generated dbt command (e.g. \"--full-refresh\"). Selection and parse-context options (--select, --vars, --target, ...) are rejected. Leave blank if unsure.\nextra_dbt_command_options", - "order": 7 + "order": 8 } }, "success_message": "\n\nYour new project has been created in the '{{.project_name}}' directory!\n\nNext steps:\n\n $ cd {{.project_name}}\n $ make setup # install dependencies into .venv\n $ make manifest # generate the dbt manifest (dbt parse) — required before the first deploy\n $ databricks bundle deploy --target dev\n\nRefer to the README.md file for full instructions." diff --git a/contrib/templates/dbt-factory/template/{{.project_name}}/README.md.tmpl b/contrib/templates/dbt-factory/template/{{.project_name}}/README.md.tmpl index bfbed35..077baaa 100644 --- a/contrib/templates/dbt-factory/template/{{.project_name}}/README.md.tmpl +++ b/contrib/templates/dbt-factory/template/{{.project_name}}/README.md.tmpl @@ -2,7 +2,7 @@ The '{{.project_name}}' project was generated using the **dbt-factory** template for Declarative Automation Bundles. It runs a [dbt](https://docs.getdbt.com/) project on Databricks as a -Databricks Workflow with **one task per dbt object** (model, seed, snapshot, test), instead of +Databricks Lakeflow Job with **one task per dbt object** (model, seed, snapshot, test), instead of running the whole project as a single opaque task. It combines two pieces: @@ -15,8 +15,15 @@ It combines two pieces: which runs the factory against the manifest and returns the generated job. **No per-model job YAML is checked in** — the task graph is generated on the fly from the manifest. +{{ if eq .use_serverless "no" -}} +This project runs on a **job cluster** (classic compute) using the **notebook task type**. A single +job cluster is created once per run and **shared by every task**, then torn down when the run +finishes. It is defined in `resources/__init__.py` (the `job_clusters` block in `build_job`) — see +[Sizing the job cluster](#sizing-the-job-cluster) below. +{{- else -}} This project uses **serverless compute** and the **notebook task type** for the fastest task start times. +{{- end}} ## Setup @@ -68,6 +75,47 @@ Open the run URL the CLI prints to watch the generated per-model task graph exec ## Configuring the generated job +{{ if eq .use_serverless "no" -}} +The knobs you chose at init are constants at the top of `resources/__init__.py` +(`BUNDLE_TESTS`, `JOB_CLUSTER_KEY`, `EXTRA_DBT_COMMAND_OPTIONS`); edit them there to change +behavior. The dbt target, warehouse, catalog, and schema live in `dbt_profiles/profiles.yml` and +are selected per bundle target via `--target ${bundle.target}`. + +### Sizing the job cluster + +**You must size the cluster yourself by editing a file — there is no `bundle init` prompt for it.** +The job cluster ships with default settings; to change them, open +[`resources/__init__.py`](resources/__init__.py) and edit the `job_clusters` block inside +`build_job`: + +```python +"new_cluster": { + "spark_version": "18.x-scala2.13", # Databricks Runtime version + "node_type_id": "...", # instance type + "num_workers": 0, # single node (see below) + "spark_conf": {"spark.databricks.cluster.profile": "singleNode", "spark.master": "local[*]"}, + "custom_tags": {"ResourceClass": "SingleNode"}, +}, +``` + +This job cluster only **orchestrates**: each task runs the runner notebook, which submits your +models as SQL to the **SQL warehouse** set by `http_path` in `dbt_profiles/profiles.yml`. That +warehouse — not this cluster — executes the queries. Because the tasks are I/O-bound (submit SQL, +wait), the default is a **single-node** cluster: adding workers does **not** speed dbt up. Bump +`node_type_id` only if orchestration itself needs more headroom, and size the **SQL warehouse** for +your query load. All tasks share this one cluster for the whole run. + +**When to do this:** the cluster is created from this block at `databricks bundle deploy` time (not +at `bundle init`). So edit it **before your first deploy** (`make deploy` / `databricks bundle +deploy`), and after any later change **redeploy** for the new size to take effect — nothing changes +the running job until you redeploy. + +The runtime dbt-databricks version is not a knob: a job cluster has no serverless base environment, +so at deploy time `resources/__init__.py` installs the exact `dbt-databricks` from your `.venv` (the +version that generated the manifest) on the cluster as a task library. `pyproject.toml` is the +single source of truth: change the version there, re-run `make setup`, and the next deploy uses +it — so what runs in Databricks matches what you tested with. +{{- else -}} The knobs you chose at init are constants at the top of `resources/__init__.py` (`BUNDLE_TESTS`, `ENVIRONMENT_KEY`, `EXTRA_DBT_COMMAND_OPTIONS`); edit them there to change behavior. The dbt target, warehouse, catalog, and schema live in `dbt_profiles/profiles.yml` and @@ -79,6 +127,7 @@ generated the manifest), shipped as a `dbt_serverless_env.yaml` [base environmen the bundle generates and syncs each deploy. `pyproject.toml` is the single source of truth: change the version there, re-run `make setup`, and the next deploy uses it — so what runs in Databricks matches what you tested with. +{{- end}} ## Tests @@ -98,6 +147,12 @@ folders (`src/models`, `src/seeds`, …) in place of the starter models, then ru If you use dbt packages, copy your `packages.yml` to the project root too: `make manifest` installs them (`dbt deps`) and the bundle syncs the resulting `dbt_packages/` to the workspace, so the job never installs packages at runtime. -See the -[Migrating an existing dbt project](https://github.com/databricks/bundle-examples/blob/main/contrib/dbt_factory/README.md#migrating-an-existing-dbt-project) -guide for the full steps. + +## Local development with dbt + +`dbt parse` (via `make manifest`) makes no warehouse connection, so it works as-is. To run models +locally (`dbt run`, `dbt test`), dbt needs `DBT_HOST` (workspace URL) and `DBT_ACCESS_TOKEN`: the +deployed job's runner notebook sets both at runtime — that's what the `env_var(...)` host/token +lines in `dbt_profiles/profiles.yml` read, so you configure nothing there. For local runs, export +those two variables yourself, or create your own profile with `dbt init` (see +[`profile_template.yml`](profile_template.yml)). diff --git a/contrib/templates/dbt-factory/template/{{.project_name}}/databricks.yml.tmpl b/contrib/templates/dbt-factory/template/{{.project_name}}/databricks.yml.tmpl index 51a11a6..3c4717d 100644 --- a/contrib/templates/dbt-factory/template/{{.project_name}}/databricks.yml.tmpl +++ b/contrib/templates/dbt-factory/template/{{.project_name}}/databricks.yml.tmpl @@ -16,7 +16,9 @@ python: # These files are generated (by resources/__init__.py / `make manifest`) and git-ignored; include # them explicitly so `bundle deploy` still syncs them to the workspace: +{{- if eq .use_serverless "yes"}} # - dbt_serverless_env.yaml the base_environment the generated job points at. +{{- end}} # - target/partial_parse.msgpack dbt's parse cache; each task injects it to skip parsing (see # README). Regenerated by `make manifest`; usable at runtime # because the runtime dbt is pinned to your local version. @@ -24,7 +26,9 @@ python: # never install packages at runtime. sync: include: +{{- if eq .use_serverless "yes"}} - dbt_serverless_env.yaml +{{- end}} - target/partial_parse.msgpack - dbt_packages/** diff --git a/contrib/templates/dbt-factory/template/{{.project_name}}/resources/__init__.py.tmpl b/contrib/templates/dbt-factory/template/{{.project_name}}/resources/__init__.py.tmpl index 419ecf2..90a9c71 100644 --- a/contrib/templates/dbt-factory/template/{{.project_name}}/resources/__init__.py.tmpl +++ b/contrib/templates/dbt-factory/template/{{.project_name}}/resources/__init__.py.tmpl @@ -7,14 +7,16 @@ reads the dbt manifest and expands it into a Databricks job with one task per db per-model YAML is generated or checked in — the job graph is built on the fly from the manifest. This module is the only integration glue: it configures the vendored ``databricks_dbt_factory`` -core (under ``src/databricks_dbt_factory``) for serverless notebook tasks and returns a +core (under ``src/databricks_dbt_factory``) for {{if eq .use_serverless "no"}}notebook tasks on a job cluster{{else}}serverless notebook tasks{{end}} and returns a ``Resources`` object for PyDABs. """ import os from importlib.metadata import PackageNotFoundError, version +{{ if eq .use_serverless "yes" -}} import yaml +{{ end -}} from databricks.bundles.core import Bundle, Resources from databricks.bundles.jobs import Job @@ -38,8 +40,13 @@ MANIFEST_PATH = os.environ.get("DBT_MANIFEST_PATH", "target/manifest.json") # Name of the generated Databricks job. JOB_NAME = "{{.project_name}}_job" +{{ if eq .use_serverless "no" -}} +# Key of the job cluster (defined in build_job below) that every task runs on. +JOB_CLUSTER_KEY = "dbt_cluster" +{{- else -}} # Key of the serverless environment (defined on the job below). ENVIRONMENT_KEY = "{{.environment_key}}" +{{- end}} # Bundle single-model tests per resource into one `dbt test` task (fewer task startups, # faster end-to-end runtime for projects with many tests). See the databricks-dbt-factory @@ -53,6 +60,7 @@ BUNDLE_TESTS = {{if eq .bundle_tests "yes"}}True{{else}}False{{end}} # `printf "%q"` renders a safely-quoted string literal so values containing quotes stay valid Python. EXTRA_DBT_COMMAND_OPTIONS = {{ printf "%q" .extra_dbt_command_options }} +{{ if eq .use_serverless "yes" -}} # Serverless base-environment file, generated at deploy time from the pinned # dbt-databricks version and synced with the bundle. The generated job points its # serverless environment at this file (see build_job) so Databricks pre-builds the @@ -61,6 +69,7 @@ EXTRA_DBT_COMMAND_OPTIONS = {{ printf "%q" .extra_dbt_command_options }} SERVERLESS_ENV_FILE = "dbt_serverless_env.yaml" SERVERLESS_ENV_PATH = "${workspace.file_path}/" + SERVERLESS_ENV_FILE +{{ end -}} # The runner notebook shipped with the core, referenced in place (path relative to the bundle # root). `PROJECT_DIRECTORY` is the path from the notebook's own directory back up to the dbt # project root (== bundle root), where the runner changes directory to before running dbt. It is @@ -74,7 +83,11 @@ def _build_tasks(target: str) -> list[dict]: """Reads the dbt manifest and returns the list of Databricks task dicts (one per dbt node).""" resolver = DbtDependencyResolver() task_options = DbtTaskOptions( + {{- if eq .use_serverless "no"}} + job_cluster_key=JOB_CLUSTER_KEY, + {{- else}} environment_key=ENVIRONMENT_KEY, + {{- end}} notebook_path=RUNNER_NOTEBOOK_PATH, project_directory=PROJECT_DIRECTORY, profiles_directory=PROFILES_DIRECTORY, @@ -91,7 +104,19 @@ def _build_tasks(target: str) -> list[dict]: factory = DbtFactory(task_factories, bundle_tests=BUNDLE_TESTS) manifest = read_dbt_manifest(MANIFEST_PATH) +{{- if eq .use_serverless "no"}} + tasks = factory.create_tasks(manifest) + # A job cluster does not get the serverless base environment, so dbt-databricks must be + # installed on the cluster. Listing it as a library on every task installs it once on the + # shared job cluster (idempotent) and guarantees it is present no matter which task the + # scheduler starts first. + library = {"pypi": {"package": _dbt_databricks_dependency()}} + for task in tasks: + task["libraries"] = [library] + return tasks +{{- else}} return factory.create_tasks(manifest) +{{- end}} def _dbt_databricks_dependency() -> str: @@ -100,7 +125,7 @@ def _dbt_databricks_dependency() -> str: is therefore the single source of truth for the dbt version, and the version that runs in Databricks is guaranteed to match the one you tested with (no separate range that can drift). Local or dev builds (e.g. `1.9.0+custom`) are rejected: pip cannot resolve them from PyPI - when Databricks builds the serverless environment, which would fail every task at runtime. + when Databricks builds the {{if eq .use_serverless "no"}}task library{{else}}serverless environment{{end}}, which would fail every task at runtime. """ try: installed = version("dbt-databricks") @@ -118,12 +143,13 @@ def _dbt_databricks_dependency() -> str: if parsed is None or parsed.local or parsed.is_devrelease: raise RuntimeError( f"The installed dbt-databricks version ({installed}) is not a plain PyPI release, so the " - "serverless environment cannot install it. Set a released version in pyproject.toml and " + "{{if eq .use_serverless "no"}}cluster cannot install it{{else}}serverless environment cannot install it{{end}}. Set a released version in pyproject.toml and " "re-run `make setup` before deploying." ) return f"dbt-databricks=={installed}" +{{ if eq .use_serverless "yes" -}} def _serverless_environment_spec() -> dict: """Contents of the base-environment file: the environment version plus the pinned dbt-databricks dependency. @@ -158,7 +184,49 @@ def _write_serverless_environment_file() -> None: ) from exc +{{ end -}} def build_job(target: str) -> Job: +{{- if eq .use_serverless "no"}} + """Builds the Databricks job that runs the dbt project on a job cluster via notebook tasks.""" + return Job.from_dict( + { + "name": JOB_NAME, + "queue": {"enabled": True}, + "trigger": { + # Run this job every day, exactly one day from the last run; + # see https://docs.databricks.com/api/workspace/jobs/create#trigger + "periodic": {"interval": 1, "unit": "DAYS"}, + }, + "tasks": _build_tasks(target), + # The job cluster every task runs on. It only orchestrates — each task runs the runner + # notebook, which submits SQL to the warehouse in dbt_profiles/profiles.yml and waits — + # so it is a single-node cluster by default: extra workers do not speed dbt up (the + # warehouse runs the queries). Bump node_type_id if orchestration itself needs more + # headroom, and scale the SQL warehouse for query load. Changes take effect on the next + # `databricks bundle deploy`. + "job_clusters": [ + { + "job_cluster_key": JOB_CLUSTER_KEY, + "new_cluster": { + # `spark_version` holds the Databricks Runtime version. "18.x-scala2.13" is + # DBR 18 LTS; its ".x" auto-updates to the latest maintenance release. Use a + # specific release (e.g. "18.2.x-scala2.13") only if you want to pin it. + "spark_version": "18.x-scala2.13", + "node_type_id": "{{smallest_node_type}}", + # Single node (driver only): no workers, since the tasks are I/O-bound + # orchestration, not Spark compute. + "num_workers": 0, + "spark_conf": { + "spark.databricks.cluster.profile": "singleNode", + "spark.master": "local[*]", + }, + "custom_tags": {"ResourceClass": "SingleNode"}, + }, + } + ], + } + ) +{{- else}} """Builds the Databricks job that runs the dbt project on serverless via notebook tasks.""" return Job.from_dict( { @@ -180,16 +248,19 @@ def build_job(target: str) -> Job: ], } ) +{{- end}} def load_resources(bundle: Bundle) -> Resources: """ Called by the Databricks CLI during `bundle deploy` to load resources defined in Python. - Reads the dbt manifest, writes the serverless base-environment file (synced with the bundle), - and registers the generated job. After deployment this function is not used. + Reads the dbt manifest{{if eq .use_serverless "no"}} and registers the generated job{{else}}, writes the serverless base-environment file (synced with the bundle), + and registers the generated job{{end}}. After deployment this function is not used. """ +{{- if eq .use_serverless "yes"}} _write_serverless_environment_file() +{{- end}} resources = Resources() resources.add_job(JOB_NAME, build_job(bundle.target)) return resources diff --git a/contrib/templates/dbt-factory/template/{{.project_name}}/tests/test_load_resources.py b/contrib/templates/dbt-factory/template/{{.project_name}}/tests/test_load_resources.py.tmpl similarity index 74% rename from contrib/templates/dbt-factory/template/{{.project_name}}/tests/test_load_resources.py rename to contrib/templates/dbt-factory/template/{{.project_name}}/tests/test_load_resources.py.tmpl index aa10276..ad4a726 100644 --- a/contrib/templates/dbt-factory/template/{{.project_name}}/tests/test_load_resources.py +++ b/contrib/templates/dbt-factory/template/{{.project_name}}/tests/test_load_resources.py.tmpl @@ -11,7 +11,9 @@ from pathlib import Path import pytest +{{- if eq .use_serverless "yes"}} import yaml +{{- end}} from databricks.bundles.core import Bundle import resources @@ -51,6 +53,42 @@ def test_build_tasks_covers_every_buildable_node(): assert expected_keys <= task_keys +{{ if eq .use_serverless "no" -}} +def test_generated_tasks_are_job_cluster_notebook_tasks(): + tasks = resources._build_tasks("dev") + + for task in tasks: + assert "notebook_task" in task, task + assert task["job_cluster_key"] == resources.JOB_CLUSTER_KEY + assert "environment_key" not in task # job cluster, not serverless + notebook_task = task["notebook_task"] + assert notebook_task["notebook_path"] == resources.RUNNER_NOTEBOOK_PATH + # The dbt target is injected into every command. + assert "--target dev" in notebook_task["base_parameters"]["dbt_commands"] + + +def test_tasks_pin_dbt_databricks_as_a_library(): + # A job cluster has no serverless base environment, so every task installs dbt-databricks + # (pinned to the version in the bundle venv) as a cluster library. + pin = f"dbt-databricks=={version('dbt-databricks')}" + tasks = resources._build_tasks("dev") + + for task in tasks: + assert task["libraries"] == [{"pypi": {"package": pin}}] + + +def test_job_cluster_is_defined_on_the_job(): + job = resources.build_job("dev") + assert len(job.job_clusters) == 1 + assert job.job_clusters[0].job_cluster_key == resources.JOB_CLUSTER_KEY + + +def test_load_resources_registers_the_job(): + result = resources.load_resources(Bundle(target="dev")) + assert resources.JOB_NAME in result.jobs + job = result.jobs[resources.JOB_NAME] + assert job.job_clusters[0].job_cluster_key == resources.JOB_CLUSTER_KEY +{{- else -}} def test_generated_tasks_are_serverless_notebook_tasks(): tasks = resources._build_tasks("dev") @@ -91,15 +129,17 @@ def test_load_resources_registers_the_job(): assert content["dependencies"] == [pin] finally: (PROJECT_ROOT / resources.SERVERLESS_ENV_FILE).unlink(missing_ok=True) +{{- end}} def test_dependency_pin_rejects_non_pypi_versions(monkeypatch): - # A local or dev build cannot be pip-installed from PyPI when Databricks builds the - # serverless environment, so the deploy must fail early with a clear message. + # A local or dev build cannot be pip-installed from PyPI, so the deploy must fail early + # with a clear message. for installed in ("1.9.0+custom", "1.13.0.dev0", "not-a-version"): monkeypatch.setattr(resources, "version", lambda name, v=installed: v) with pytest.raises(RuntimeError, match="PyPI"): resources._dbt_databricks_dependency() +{{- if eq .use_serverless "yes"}} def test_environment_file_write_is_idempotent(): @@ -119,3 +159,4 @@ def test_environment_file_write_is_idempotent(): if env_file.exists(): env_file.chmod(0o644) env_file.unlink() +{{- end}}