Repository navigation
feat!: make configuration the only Rule exception mechanism, speed up sqb query and fix exit and cost failures - #1175
Merged
Merged
Conversation
…hed capture A write that reached a capture already marked complete built its rejection while still holding the storage lock. Building it imports Python, which can release the GIL; the main thread then pruning the logs waited for that lock while holding the GIL, so neither could continue and the command never exited. The lock is now released before the rejection is raised.
…alling the job run_sqb now gives each command ten minutes. A command still running then has its process group aborted with Python's fault handler enabled, so the failing test reports every thread's stack, and the group is killed. Standard input ends at once when a test passes none.
… in sqb without Python
Shared* MergeTree engines now pass model validation like their MergeTree counterparts. Replicated* engines stay rejected. Support is best effort and untested on ClickHouse Cloud.
…' into feat/combined-batch
…feat/combined-batch
…very sqb query only needs the adapter and connection settings. Load the project and local configuration, validate target warehouses and connections, and connect, instead of parsing every declaration and importing project Python. Ad hoc SQL now also runs when an unrelated model or Python file is broken.
…ices Large runs always timed out collecting cost: one query-history read covered the whole run and embedded every run query ID in the SQL. Read history in slices of at most 15 minutes, select CREATE text by the run's query tag so the SQL no longer grows with the run, split a slice that reaches its statement timeout, and record cost as pending, naming the timeout, when the collection budget runs out. Other failures now name the Snowflake error.
The integration-result contract rejected the seed attachment kind, so every seed audit degraded its record and printed a warning. Accept it, and map singular audits, whose results name where they ran, to the DAG check that lists every resource they read, so end-of-run audits reach Dagster instead of being dropped.
A target can set concurrency in [targets.<name>]. Commands use --concurrency, then the selected target's value, then the local and project [settings] value. Invalid values fail with D001, and sqb build --verbose shows the effective limit and where it came from.
… interop The planner's concurrent-microbatch warning and the SQLBuild work in sqb dbt read the project setting directly, ignoring a target's concurrency. Resolve both through the same precedence as execution: --concurrency, the target, then the local and project settings.
SQBRSQL038 no longer reports a cross join when either side is provably at most one row. The proof is inferred from the query: a seed whose CSV has one data row, an aggregate without GROUP BY, a one-row VALUES, LIMIT 1, or QUALIFY ROW_NUMBER() = 1 without a partition, carried through CTEs and subqueries that only filter or project. Anything else is still reported. Findings now name the joined relation and its CTE. Rule exceptions now live only in sqlbuild_project.toml: - an inline `-- sqb: ignore` directive is a SQBRSQL000 error that suppresses nothing and prints the exact [[rules.rule_exceptions]] entry to add; - `[rules] allow_exceptions` is removed and is a configuration error explaining the removal; - [[rules.rule_exceptions]] accepts `relation` and an optional `cte` for SQBRSQL038, so one exception covers exactly one join; relation-level exceptions are stale-checked like file-wide ones. BREAKING CHANGE: inline `-- sqb: ignore` comments are errors and `[rules] allow_exceptions` is rejected. Move each reviewed deviation into a [[rules.rule_exceptions]] entry and delete the setting.
SQBRGRAPH103 rejects a published model reading a published model of its own domain, except a <domain>__mart_v__<entity> view reading <domain>__mart__<entity>. SQBRPROJECT107 requires one name shape per domain's published layer, with literal allowed_qualifiers that each carry a reason. Both are ordinary built-in errors that skip names outside the <domain>__<layer>__<entity> grammar, take configurable published_layers (default mart, mart_v), and print the sqb mv or sqb rename command where one applies.
Rewrite findings and exceptions for relation-level exceptions, one-row cross join inference and the removed inline suppressions and allow_exceptions setting, add the published-layer Rules page with the orders and finance example, and regenerate the docs skill.
A sink call still running when the shutdown budget ended deferred provider teardown to whichever daemon delivery thread finished last. That thread then ran the provider's teardown inside native frames; when interpreter finalization began during the teardown, the thread was ended by forced unwinding through those frames and the process aborted with SIGABRT after the command had already succeeded. Otherwise the teardown was silently lost. finalize_when_idle now waits, bounded by the sink call timeout (at most one second), for delivery to become idle and then runs the finalizer on the calling thread. Delivery threads no longer run finalizers.
…ways deliver outcomes Each lifecycle sink now has one long-lived delivery thread that calls the sink for every event in order and reports outcomes through a Python queue, instead of a new thread per event that the dispatcher waited on. The dispatcher keeps the bounded priority queue, hands each sink up to 64 attempts (never more than the queue capacity), and keeps the per-call timeout, blocking and shutdown-budget semantics. Sinks no longer wait on each other. Run and invocation outcome events are always admitted to the queue, displacing other events when it is full, and are never displaced. The final quarter of the shutdown budget is reserved for them: when it begins, other queued and waiting events are dropped and counted.
…xport warning The incomplete lifecycle export warning could be the last line a successful command printed. It is now followed by "sqb <command> completed successfully; only its lifecycle event export is affected.", built by the same closing-status helper the Recall notice uses, and a failed stderr write no longer turns into a command error.
…ger budget Slicing every run into 15-minute reads made long runs cost dozens of reads per attempt, so an eight-hour run could never finish within the 30-second budget and stayed pending. Read the whole window in one query again and split only a window that saturates or times out. The at-exit collection keeps its 30-second budget; sqb cost retries a pending run with a 5-minute budget.
…tion A named window is never treated as unpartitioned, so QUALIFY ROW_NUMBER() OVER w = 1 no longer hides a cross join. Findings name the top-level CTE, including WITH clauses owned by set operations, so an exception copied from a finding matches it. Joins that read no named relation say only a file-wide exception can cover them and print that entry with the file's path. Inline directives are detected only inside SQL comments, so string literals never trigger SQBRSQL000.
…never run Reports are collected once per dispatch round, so a lane thread could already have finished its first call and started the next one when the dispatcher cancelled it for the shutdown reserve. The sink still received the event, but it was counted as dropped and its report was ignored, so the incomplete-export warning could claim losses that did not happen. Each delivery thread now claims a call in a dict shared with the dispatcher before running it, and the dispatcher checks and marks the same entry, both atomically under the GIL. A cancellation counts as dropped only when the call was not claimed; a claimed call stays pending and its report is accepted. Abandoned lanes count claimed calls as timed out instead of dropped. Reports are now read after taking the dispatcher condition.
Opening several connections to a new file at once could fail with A702 at once: SQLite reports the WAL switch as busy without waiting while another connection switches the same file. The switch now goes through the per-file writer gate. Also give wheel builds 75 minutes, since the Intel macOS build reached the 45-minute limit.
The Intel runner is about twice as slow as the others and its wheel job reached 28-44 minutes. Cross-compile for x86_64 on macos-15 instead and smoke test the wheel with x64 Python under Rosetta.
…ions The native rules commands no longer read the removed exception switch, and render string-list option defaults and table-list options exactly as the Python CLI does, so SQBRGRAPH103 and SQBRPROJECT107 print their published_layers and allowed_qualifiers options. A project still setting allow_exceptions, or configuring rule options, is answered by Python as before.
kvlonge
enabled auto-merge (squash)
October 20, 2026 06:43
CI sets SQLBUILD_CONCURRENCY for E2E runs, which counts as --concurrency and overrides every setting these tests check.
The cached x64 Python is a universal binary that starts as arm64 on the Apple Silicon runner and rejects the x86_64 wheel; run the smoke commands with arch -x86_64.
…/combined-batch-2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Five ready changes, combined so they share one CI run: Rule exceptions move to configuration only (breaking),
sqb querystops paying for project discovery, two production failures are fixed (nightly Snowflake cost collection, an abort at exit with lost outcome events), and the Intel macOS wheel build moves off the slowest runner.Changes
-- sqb: ignoreis now errorSQBRSQL000, which prints the[[rules.rule_exceptions]]entry to add;allow_exceptionsis removed; exceptions forSQBRSQL038can name a relation and CTE and fail when stale;SQBRSQL038skips joins proven to be one row. New rules, on as errors:SQBRGRAPH103(published models don't read their own domain's published models) andSQBRPROJECT107(one published name shape per domain).sqb queryandsqb recall query --backend warehouseread only the two config files: CPU on 10,000 models 2.86 s -> 0.27 s.sqb cost <run_id>.concurrencyunder[targets.<name>].Verification
sqbnative/Python parity tests;make check-ci, compiler differential, fensu.failure/target-unknown-keynow listsconcurrencyamong the allowed target keys, because targets acceptconcurrency.