Repository navigation
feat: store Recall history in ClickHouse and add replacement sinks - #1163
Merged
Merged
Conversation
A [recall] or targets.<name>.recall connection on the clickhouse adapter now sends history to MergeTree tables in an existing ClickHouse database through the adapter's connection. Recall creates and additively upgrades its tables and views there, prints the DDL when the user may not, and batches RowBinary inserts with deterministic deduplication tokens. Lookups before each insert keep the first copy of a key, record conflicting reuse and refresh artifacts older than a week. Table TTLs follow the configured retention; a store with another retention is reported once a day with the statements that would change it. sqb recall query, schema and output read the ClickHouse store. Also fix the ClickHouse view definitions so aggregate aliases no longer shadow the columns they filter on, and take queued records when a heartbeat falls due before a batch fills.
…bles Every Recall table the views read is now a ReplacingMergeTree ordered by its key, and the views read it with FINAL instead of deduplicating with LIMIT BY. Filters on invocation_id now reach the primary key: on a store of about 3,000 invocations and 2.2 million events, a per-invocation artifact lookup reads 185 thousand rows instead of 507 thousand, and sqb recall output drops from about 2.8 s to 0.5 s.
…rded yet A reader that ran while the first writer was still creating the views failed with an unknown-table query error; it now gets the coded store error that says no history is recorded yet.
A @lifecycle_event_sink or @command_output_sink with a parameter annotated sqlbuild.recall.RecallWriter replaces Recall's default for its stream: Recall stores only what it publishes. Discovery allows one replacement per stream and reports D015 naming every path:function otherwise, rejects a sink with two writers or with providers, and reserves the provider name recall_writer (D012). Replacements are not run as ordinary sinks; other sinks still see every record. Lifecycle events reach the replacement as they are published, filtered by its event kinds and minimum severity. Output chunks reach it on the delivery thread as schema version 2 CommandOutputRecords carrying the exact bytes in data; record.replace() keeps the ID and sequence. An unpublished record is counted as filtered, a raising sink as failed, never falling back to the unfiltered record and never changing the exit code or output. The invocation row now waits for discovery so filter_active records whether a replacement ran.
…iew until its TTL
…for sqb recall readers
…uring discovery is listed
… within the shutdown budget
…ersion and write only stored columns
…w optional columns
kvlonge
enabled auto-merge (squash)
October 19, 2026 17:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Teams that run many commands across machines want one shared Recall history, and some need to decide what Recall stores (for example, redacting secrets from output) without losing other sinks.
Changes
[recall]/targets.*.recallconnectionmay name a ClickHouse connection. MergeTree tables with TTLs and FINAL-read views are created on first use; the DDL (with its schema-version row) is printed when the user may not create them, and a SELECT+INSERT-only user can write a store created from it. Batched idempotent inserts with conflict detection, per-table failure accounting, retention-conflict notices (TTLs are never altered), andsqb recall query/schema/outputsupport. Output is not capped by default.RecallWriterreplaces Recall's default for its stream. Both replacements run on Recall's delivery thread withinshutdown_timeout; output is held until discovery decides; filtered/failed counts andfilter_activeare recorded; exit codes and stdout are unaffected.Verification
sqlbuild-recallcargo tests.make check-ci(fmt, clippy, ruff, ty, fensu, layering, boundary).