Skip to content

feat!: add typed SQL macros and require @py./@sql. macro calls - #1224

Merged
kvlonge merged 19 commits into
mainfrom
feat/sql-macros
Oct 23, 2026
Merged

kvlonge merged 19 commits into
mainfrom
feat/sql-macros

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Why

Python macros were the only way to reuse SQL, so logic-free fragments were opaque to the compiler, formatter, rules and lineage. SQL macros are typed, checked fragments; every call names its kind.

Changes

  • SQL macros in macros/sql/ (and scoped _macros/sql/), one per file, called @sql.name(...): expression, CTE-list and column-list forms; commas around list calls are optional.
  • Typed parameters: SQL types, CONST, LIST/SET, ENUM, RELATION (...), COLUMN OF, DEFAULT, named arguments.
  • Arguments are checked against the caller's inferred column types; mismatches fail with P017 at the call, noting the parameter.
  • New codes P014–P023 and S026; bodies resolve constants, enums and nested macros in their own file's scope.
  • sqb format, rules, lineage, macro tests, __macro__ mocks and sqb mv support SQL macros.
  • BREAKING: Python macros and the _ helper modules they import move to macros/python/ (imports gain .python) and are called @py.name(...); files directly under macros/ fail with D013, bare @name(...) with P014.
  • --since compares SQL macros by their parsed definition (formatting, comments and description ignored) and follows nested SQL macro calls.
  • Known limits: calls in subqueries or set-operation parts, and arguments read through untyped sources such as unnest(), skip the argument check; rule findings inside an expansion point into the expanded query; a backslash-escaped string that also reads as balanced without escapes can be misread.

Verification

  • Rust unit tests; DuckDB E2E for every form, argument check, format, error and --since case.
  • Refactor, scope, rules and differential tests pass; goldens regenerated for the @py. prefix and python/ paths.
  • 10,000-model fresh compile without SQL macros: +0.02% instructions vs main.
  • 300 macro-calling models vs base: cold +6.3%, cache-writing +5.8%, warm 0% (instructions:u); checks cost ~2% cold.
  • make check-ci passes.

@kvlonge
kvlonge enabled auto-merge (squash) October 22, 2026 21:19
Add SQL macros: logic-free SQL fragments in macros/sql/ with a MACRO
header and typed arguments, in three forms decided by the body (an
expression, a WITH list of CTEs, or a SELECT list of columns). Calls are
written @sql.name(...), expand in place in the declaration pass, and are
checked on their own and again in every caller. Parameters are written
@name in bodies; types follow schema declaration types, with CONST for
compile-time scalars, LIST/SET, ENUM, RELATION and COLUMN OF arguments.
Runtime parameters in literal-only positions are rejected. Formatting,
rules and column lineage see through calls, and macro-mode unit tests
and __macro__ mocks work for SQL macros.

BREAKING CHANGE: Python macros now live in macros/python/ (and the
python/ folder of scoped macro roots) and are called as @py.name(...).
A macro file directly under a macros/ folder fails with D013, and a bare
@name(...) call fails with P014; move files into macros/python/ and
replace @name( with @py.name( in project SQL.
…lacement

Report RELATION columns, COLUMN OF names and typed arguments that do not fit
at the call, with the parameter's declaration. Accept every comma spelling
around CTE-list and column-list calls, reject aliases on column-list calls,
resolve macro bodies in the macro file's own scope, keep valid SQL macros
when another fails inventory, use real call spans, read backslash-escaped
strings, parenthesise before AT, OVER and FILTER, keep separators in mocks,
record hook-only SQL macro dependencies, and format expression bodies and
MACRO headers.
Check cache-hit models' SQL macro arguments too, resolve macro body
declarations through the native per-file declaration context, build related
locations as native records, and keep the SQL macro helpers flat.
Treat typed arguments in subqueries or set-operation scopes as unknown,
resolve a FROM alias before a same-named CTE, keep DISTINCT ON calls
rejected as before, and keep string literal lines when unwrapping
formatted expression bodies.
Merge a scope's typed arguments into one probe, infer probe output types
directly instead of running full analysis per probe, check a wave's models
together, and locate calls without re-scanning the whole file.
Apply group-folder layout rules within a macro's python/ or sql/ folder,
migrate the layout and move tests to kind folders and @py. calls, and teach
the agent skill SQL macros and kind-prefixed calls.
Treat a probe as unknown when any FROM or JOIN source in it cannot be
typed, so stars over unnest() or table functions no longer report missing
columns. Teach kind-prefixed macro calls in the remaining docs.
The render-stage view filtered unknown-column findings inside macro calls
by redeclaring B002, which the render code inventory reads as a code it
emits. Name the analysis-owned constant instead.
The byte-by-byte scan cost about 4 instructions per byte of every authored
SQL text, 0.15% of a 10,000-model compile without SQL macros.
Fingerprint a SQL macro declaration by its form, typed arguments, result
type and normalised body, ignoring layout, comments and its description,
and keep the edges to the SQL macros it calls.
@kvlonge
kvlonge disabled auto-merge October 23, 2026 05:13
@kvlonge
kvlonge enabled auto-merge (squash) October 23, 2026 05:57
@kvlonge
kvlonge merged commit 21a802b into main Oct 23, 2026
44 of 45 checks passed
@kvlonge
kvlonge deleted the feat/sql-macros branch October 23, 2026 06:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant