Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ cat query.sql | sqlchisel --stdin --format
- Formatting heuristics (non-contract): [`docs/style-guide.md`](docs/style-guide.md)
- Dremio support notes: [`docs/dremio.md`](docs/dremio.md)
- Dremio reference coverage tracker: [`docs/dremio-support-matrix.md`](docs/dremio-support-matrix.md)
- dbt/Jinja support tracker: [`docs/dbt-support-plan.md`](docs/dbt-support-plan.md)
- Editor integrations: [`docs/editor-integrations.md`](docs/editor-integrations.md)

## Contributing
Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This directory separates stable behavior from formatting heuristics and integrat
- [`style-guide.md`](style-guide.md): current formatting heuristics and examples (non-contract)
- [`dremio.md`](dremio.md): Dremio-specific supported syntax and formatting expectations
- [`dremio-support-matrix.md`](dremio-support-matrix.md): command-by-command Dremio SQL reference coverage tracker
- [`dbt-support-plan.md`](dbt-support-plan.md): dbt/Jinja templating support tracker

## Integrations and Development

Expand Down
3 changes: 3 additions & 0 deletions docs/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Constraints:
## Common Flags

- `--dialect <ansi|dremio>`
- `--templating <passthrough|dbt>`
- `--keyword-case <upper|lower|capitalize>`
- `--line-length <N>`
- `--indent-width <N>`
Expand All @@ -40,6 +41,8 @@ Constraints:
When a provided path is a directory, `sqlchisel` recursively collects matching files.

- Default include glob: `**/*.sql`
- In `--templating dbt`, default include globs are `**/*.sql`, `**/*.sql.j2`,
`**/*.sql.jinja`, and `**/*.sql.jinja2`
- `--include` overrides the default include list
- `--exclude` filters paths out
- Ignored directories during recursion: `.git`, `target`, `.cargo`
Expand Down
3 changes: 3 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ line_length = 100
indent_width = 2
keyword_case = "upper"
dialect = "ansi"
templating = "passthrough"
select_list_style = "auto"
strict = false
```
Expand All @@ -33,6 +34,7 @@ strict = false
- `indent_width` (`usize`): spaces per indent level. Default: `2`.
- `keyword_case` (`"upper" | "lower" | "capitalize"`): keyword rendering. Default: `"upper"`.
- `dialect` (`"ansi" | "dremio"`): parser/formatter dialect. Default: `"ansi"`.
- `templating` (`"passthrough" | "dbt"`): template handling. Default: `"passthrough"`.
- `select_list_style` (`"auto" | "per_line"`): select-list layout strategy. Default: `"auto"`.
- `strict` (`bool`): fail on parse errors instead of raw fallback. Default: `false`.

Expand All @@ -44,5 +46,6 @@ These flags override config values for the current run:
- `--indent-width`
- `--keyword-case`
- `--dialect`
- `--templating`
- `--select-list-style`
- `--strict` (enables strict mode)
96 changes: 96 additions & 0 deletions docs/dbt-support-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# dbt Syntax Support Plan

Track dbt templating support here so parser, formatter, fixtures, and docs progress stays visible in-repo.

## References

- dbt SQL models: <https://docs.getdbt.com/docs/build/sql-models>
- dbt Jinja and macros: <https://docs.getdbt.com/docs/build/jinja-macros>
- dbt Jinja functions: <https://docs.getdbt.com/reference/dbt-jinja-functions>
- dbt data tests: <https://docs.getdbt.com/docs/build/data-tests>
- Jinja template syntax: <https://jinja.palletsprojects.com/en/stable/templates/>
- Dremio dbt guide: <https://docs.dremio.com/current/data-products/deploy-with-dbt/>
- dbt-dremio walkthrough: <https://github.com/dremio/dbt-dremio/blob/main/docs/walkthrough.md>

## Status Legend

- `PASSTHROUGH`: Intentionally returned unchanged.
- `PLACEHOLDER`: Jinja/dbt syntax is protected and restored exactly while surrounding SQL can format.
- `FORMATTED`: SQL inside the syntax family is formatted where safe.
- `PARTIAL`: Some representative forms work, but known syntax remains.
- `BLOCKED`: Not supported by the current formatter architecture.

## Milestone 1: Tracker and Public Interface

- [x] Add this tracker file.
- [x] Link this tracker from `docs/README.md`.
- [x] Add `templating = "passthrough" | "dbt"` config support.
- [x] Add `--templating <passthrough|dbt>` CLI support.
- [x] Keep `templating = "passthrough"` as the default.
- [x] Include dbt SQL template extensions during directory traversal in dbt mode.
- [x] Document canonical Dremio invocation:
`sqlchisel --dialect dremio --templating dbt --format models/my_model.sql`.

## Milestone 2: dbt/Jinja Scanner Foundation

- [x] Preserve exact contents of `{{ ... }}`, `{% ... %}`, and `{# ... #}`.
- [x] Preserve whitespace-trim forms such as `{{- ... -}}` and `{%- ... -%}`.
- [x] Support quoted strings inside template tags.
- [x] Treat `{% raw %}...{% endraw %}` as an opaque preserved block.
- [x] Protect Jinja markers inside SQL comments and string literals.
- [x] Avoid splitting SQL statements on semicolons inside dbt/Jinja tags.
- [ ] Add strict-mode diagnostics for malformed template delimiters.

## Milestone 3: Core dbt Model Syntax

- [x] Format model SQL around standalone `{{ config(...) }}` blocks.
- [x] Preserve relation-like placeholders: `ref`, versioned/two-argument `ref`, `source`, and `this`.
- [x] Preserve expression placeholders: `var`, `env_var`, `target`, `is_incremental`, `adapter`, `dispatch`, and custom macro calls.
- [x] Preserve dbt dependency comments such as `-- depends_on: {{ ref(...) }}`.
- [ ] Add broader package macro fixtures for common packages such as `dbt_utils`.

## Milestone 4: Control Flow and Generated SQL

- [x] Preserve standalone `{% if %}`, `{% elif %}`, `{% else %}`, `{% endif %}` lines.
- [x] Preserve standalone `{% for %}` and `{% endfor %}` lines.
- [x] Preserve standalone `{% set %}`, `{% endset %}`, `{% do %}`, `{% call %}`, and `{% endcall %}` lines.
- [x] Format SQL inside branch bodies when the protected SQL remains parseable.
- [ ] Add targeted handling for generated select-list comma patterns that are not parseable after placeholder preservation.

## Milestone 5: dbt Resource Coverage

- [x] Add dbt model fixtures.
- [x] Add Dremio dbt fixtures.
- [x] Add reference syntax fixtures for tests, snapshots, macros, materializations, statement blocks, hooks, operations, and package macros.
- [ ] Expand each reference syntax fixture from representative syntax to broader dbt docs coverage.
- [ ] Add optional manual smoke project for `dbt parse`/`dbt compile`.

## Milestone 6: Dremio + dbt Coverage

- [x] Add Dremio dbt fixtures for `object_storage_source`, `object_storage_path`, `dremio_space`, and `dremio_space_folder`.
- [x] Add Dremio incremental fixture with `is_incremental()` and `{{ this }}`.
- [x] Cover Dremio SQL mixed with dbt placeholders and versioned refs.
- [x] Cross-link this tracker with `docs/dremio-support-matrix.md`.
- [ ] Expand adapter-style DDL/DML fixtures as dbt-dremio coverage grows.

## Syntax Coverage Matrix

| Syntax Family | Status | Fixture Area | Notes |
| --- | --- | --- | --- |
| `{{ config(...) }}` | `PLACEHOLDER` | `fixtures/dbt/ansi`, `fixtures/dbt/dremio` | Standalone config blocks are restored exactly. |
| `ref`, versioned `ref`, `source`, `this` | `PLACEHOLDER` | `fixtures/dbt/ansi`, `fixtures/dbt/dremio` | Relation placeholders format as SQL-safe identifiers internally. |
| `var`, `env_var`, `target`, custom macros | `PLACEHOLDER` | `fixtures/dbt/ansi` | Expression placeholders are restored exactly. |
| `is_incremental()` branches | `FORMATTED` | `fixtures/dbt/dremio` | Branch SQL formats when the protected SQL remains parseable. |
| `{% if %}` / `{% for %}` control flow | `PARTIAL` | `fixtures/dbt/reference-syntax` | Standalone blocks are preserved; complex generated SQL may raw-fallback. |
| `{% macro %}` / `{% materialization %}` | `PARTIAL` | `fixtures/dbt/reference-syntax` | SQL inside parseable bodies formats; non-SQL Jinja stays opaque. |
| `{% test %}` / snapshots | `PARTIAL` | `fixtures/dbt/reference-syntax` | Representative SQL-bearing forms covered. |
| statement blocks / `run_query` | `PARTIAL` | `fixtures/dbt/reference-syntax` | Preserved and restored; SQL strings are not independently formatted. |
| dbt YAML SQL strings | `BLOCKED` | none | Out of scope for this SQL formatter. |

## Acceptance Criteria

- [x] `cargo fmt --check`
- [x] `cargo clippy --all-targets --all-features -- -D warnings`
- [x] `cargo test`
- [x] Every dbt fixture formats idempotently.
- [x] Default Jinja passthrough behavior remains unchanged without `--templating dbt`.
1 change: 1 addition & 0 deletions docs/dremio-support-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Reference export timestamp: 2026-02-28T14:20:54.462923+00:00
- SQL statement formatter dispatch: [`src/format/sql/mod.rs`](../src/format/sql/mod.rs) (`format_statement`)
- Dremio command formatter: [`src/format/sql/dremio.rs`](../src/format/sql/dremio.rs)
- Regression fixtures: `fixtures/dremio/{in,expected,out}`
- dbt/Jinja templating on Dremio: [`docs/dbt-support-plan.md`](dbt-support-plan.md)

## Coverage Matrix (57 Dremio Command Pages)

Expand Down
3 changes: 2 additions & 1 deletion docs/format-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,9 @@ Strict behavior (`strict = true` or `--strict`):

## Jinja Passthrough

- If the input contains Jinja markers (`{{`, `{%`, or `{#`), `sqlchisel` currently returns the input unchanged.
- By default, if the input contains Jinja markers (`{{`, `{%`, or `{#`), `sqlchisel` returns the input unchanged.
- This passthrough behavior is part of the current stable contract to avoid corrupting templated SQL.
- When `templating = "dbt"` or `--templating dbt` is enabled, dbt/Jinja tags are protected and restored while surrounding SQL is formatted.

## CLI Mode Contract

Expand Down
6 changes: 6 additions & 0 deletions docs/style-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,12 @@ SELECT * FROM other;
- Preserve string literal casing and contents
- Preserve inline and block comments relative to nearby SQL where possible

## dbt/Jinja Templates

- Default templated SQL behavior is passthrough.
- With `--templating dbt`, dbt/Jinja tags are preserved exactly while surrounding SQL is formatted where safe.
- dbt templating support progress is tracked in [`dbt-support-plan.md`](dbt-support-plan.md).

## Future Tuning Areas

- Numeric thresholds for select-list layout tiers
Expand Down
4 changes: 4 additions & 0 deletions fixtures/dbt/ansi/expected/01_model.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{{ config(materialized="table") }}
SELECT id, {{ var("country_expression", "country") }} AS country, amount FROM {{ ref("orders") }}
WHERE created_at >= {{ var("start_date") }}
-- depends_on: {{ ref("dim_dates") }}
4 changes: 4 additions & 0 deletions fixtures/dbt/ansi/in/01_model.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{{ config(materialized="table") }}

select id, {{ var("country_expression", "country") }} as country, amount from {{ ref("orders") }} where created_at >= {{ var("start_date") }}
-- depends_on: {{ ref("dim_dates") }}
4 changes: 4 additions & 0 deletions fixtures/dbt/ansi/out/01_model.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{{ config(materialized="table") }}
SELECT id, {{ var("country_expression", "country") }} AS country, amount FROM {{ ref("orders") }}
WHERE created_at >= {{ var("start_date") }}
-- depends_on: {{ ref("dim_dates") }}
9 changes: 9 additions & 0 deletions fixtures/dbt/dremio/expected/01_incremental.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{{ config(materialized="incremental", object_storage_source="lake", object_storage_path="/dbt", dremio_space="analytics", dremio_space_folder="marts") }}
SELECT *
FROM {{ source("raw", "orders") }}
AT BRANCH {{ var("nessie_branch") }}
{% if is_incremental() %}
WHERE updated_at > (
SELECT MAX(updated_at) FROM {{ this }}
)
{% endif %}
6 changes: 6 additions & 0 deletions fixtures/dbt/dremio/in/01_incremental.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{{ config(materialized="incremental", object_storage_source="lake", object_storage_path="/dbt", dremio_space="analytics", dremio_space_folder="marts") }}

select * from {{ source("raw", "orders") }} at branch {{ var("nessie_branch") }}
{% if is_incremental() %}
where updated_at > (select max(updated_at) from {{ this }})
{% endif %}
9 changes: 9 additions & 0 deletions fixtures/dbt/dremio/out/01_incremental.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{{ config(materialized="incremental", object_storage_source="lake", object_storage_path="/dbt", dremio_space="analytics", dremio_space_folder="marts") }}
SELECT *
FROM {{ source("raw", "orders") }}
AT BRANCH {{ var("nessie_branch") }}
{% if is_incremental() %}
WHERE updated_at > (
SELECT MAX(updated_at) FROM {{ this }}
)
{% endif %}
12 changes: 12 additions & 0 deletions fixtures/dbt/reference-syntax/01_resources.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{% snapshot orders_snapshot %}
{{ config(target_schema="snapshots", unique_key="id", strategy="timestamp", updated_at="updated_at") }}
select id, updated_at from {{ source("raw", "orders") }}
{% endsnapshot %}

{% test accepted_status(model, column_name) %}
select * from {{ model }} where {{ column_name }} not in ("placed", "shipped")
{% endtest %}

{% macro cents_to_dollars(column_name, scale=2) %}
({{ column_name }} / 100)::numeric(16, {{ scale }})
{% endmacro %}
7 changes: 7 additions & 0 deletions fixtures/dbt/reference-syntax/02_materialization.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{% materialization table, adapter="dremio" %}
{% set target_relation = this %}
{% call statement("main") %}
create table {{ target_relation }} as select * from {{ ref("orders") }}
{% endcall %}
{% do adapter.commit() %}
{% endmaterialization %}
9 changes: 9 additions & 0 deletions src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,13 @@ pub enum DialectKind {
Dremio,
}

#[derive(Debug, Clone, Copy, ValueEnum, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum TemplatingMode {
Passthrough,
Dbt,
}

#[derive(Debug, Clone, Copy, ValueEnum, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum SelectListStyle {
Expand All @@ -30,6 +37,7 @@ pub struct FormatterConfig {
pub indent_width: usize,
pub keyword_case: KeywordCase,
pub dialect: DialectKind,
pub templating: TemplatingMode,
pub select_list_style: SelectListStyle,
pub strict: bool,
}
Expand All @@ -41,6 +49,7 @@ impl Default for FormatterConfig {
indent_width: 2,
keyword_case: KeywordCase::Upper,
dialect: DialectKind::Ansi,
templating: TemplatingMode::Passthrough,
select_list_style: SelectListStyle::Auto,
strict: false,
}
Expand Down
6 changes: 6 additions & 0 deletions src/format/sql/comments.rs
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ pub(super) fn reattach_comments(

while let Some(c) = comment_iter.peek() {
if !c.inline_with_prev && c.index == non_ws_idx {
if !out.is_empty() && !out.ends_with('\n') {
out.push('\n');
}
if !indent.is_empty() {
out.push_str(&indent);
}
Expand Down Expand Up @@ -175,6 +178,9 @@ pub(super) fn reattach_comments(
out.push_str(spacer);
out.push_str(&c.content);
} else {
if !out.is_empty() && !out.ends_with('\n') {
out.push('\n');
}
if !indent.is_empty() {
out.push_str(&indent);
}
Expand Down
Loading
Loading