docs: propose native DuckLake layered storage (SQLite-catalog)

[db-tycoon-stephen]

What

Adds docs/proposals/ducklake-layered-storage.md — a storage-engine proposal for giving each warehouse layer its own DuckLake catalog (SQLite metadata + Parquet, S3-able), composed via ATTACH; dbt reads raw read-only and materializes into mart.

Docs-only. Not part of v0.1.10 (the security pass) — proposal for a future minor.

Why

Came out of a 4-spike investigation (all green; DuckDB 1.5.3, ducklake e6a3bd0a, real Rill 0.86, dbt-duckdb 1.10.1). Headline finding corrects shipped docs (docs/commands/data/analyze.md:68): the read-while-ingest lock that justified the Parquet bridge for Rill is a property of the catalog metadata backend (DuckDB-file locks per-process), not of DuckLake or SQLite. A SQLite-backed catalog supports live Rill reads during ingest — confirmed with the real Rill binary, 0 lock conflicts.

Validated

• Quack and DuckLake do not compose (fronting DuckLake with Quack ruled out).
• DuckDB-file catalog backend locks; SQLite is multi-process safe.
• Real Rill 0.86 reads a ducklake:sqlite: catalog live during ingest, 0 conflicts.
• dbt-duckdb materializes raw-catalog → mart-catalog via profiles.yml attach:.

Relationship

• Builds on the shipped layer-aware data model (Layer-aware data model: surface sources / staging / intermediate / marts as first-class #30) — logical layers; this adds an optional physical backing. Respects "no layers: block."
• Unblocks layer-granular backup (v0.1.8+: layer-granular backup config (backup.layers replaces skip_raw) #31).
• Tracks: Native DuckLake layered storage (SQLite-catalog): optional physical per-layer catalogs #73 (this design), Docs: analyze.md:68 mislabels the DuckLake lock as SQLite-backed (it's the DuckDB-file backend) #71 (analyze.md fix), ducklake_config.py claims DuckLake but only dumps plain Parquet (misnamed) #72 (ducklake_config.py is misnamed).

Open risk

data_path: s3://… is unproven (all spikes used local paths) — gated as Phase 1.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

[gemini-code-assist]

Code Review

This pull request introduces a proposal for native DuckLake layered storage using a SQLite-catalog to enable a local-first lakehouse on object storage. The review feedback highlights a validation issue with the proposed dbt-duckdb profile configuration, which does not support custom fields like is_ducklake or options out-of-the-box, and identifies a missing markdown reference link for [no-legacy stance].

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[db-tycoon-stephen]

Review triage — commit ae56ca6

Addressed the Gemini Code Assist review:

• ✅ Undefined [no-legacy stance] markdown link → replaced with plain text.
• ↩️ "dbt-duckdb attach rejects is_ducklake/options" → accurate for older dbt-duckdb, but not for 1.10.1 (the version this proposal targets): the Attachment model declares options/is_ducklake as first-class fields, and spike 4 ran a real dbt run with this exact profile (materialized into the mart DuckLake catalog, 3 snapshots). Rather than dismiss it, added an explicit dbt-duckdb ≥ 1.10 version-floor note with the plugin fallback for older versions.

For reviewers: this PR is docs-only (a proposal) and not part of v0.1.10 (the security pass). Tracked as #73 / PTC-41. Open risk noted in the doc: data_path: s3://… is unproven (Phase 1).

[db-tycoon-stephen]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a proposal for a native DuckLake layered storage architecture using a SQLite-backed catalog to enable multi-process, local-first lakehouses. The feedback identifies two key improvements in the proposal's examples: ensuring the SQL ATTACH statement uses READ_ONLY TRUE for better DuckDB version compatibility, and adding the missing data_path option to the raw attachment in the dbt profile configuration.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[db-tycoon-stephen]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds a new proposal document (docs/proposals/ducklake-layered-storage.md) outlining a native DuckLake layered storage architecture using a SQLite-catalog. The feedback points out that the dbt profile YAML example in the proposal is missing the required schema field, which would cause dbt validation to fail, and suggests adding a default schema to make the example valid.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.