Add REFRESH DDL command#238
Open
ryannedolan wants to merge 5 commits into
Open
Conversation
ryannedolan
force-pushed
the
ryannedolan/refresh-command
branch
2 times, most recently
from
July 15, 2026 22:17
6ebb94c to
5edfb6b
Compare
ryannedolan
changed the title
ryannedolan
marked this pull request as ready for review
July 15, 2026 22:28
…m triggers REFRESH backfills a materialized view or logical table by firing every trigger immediately upstream of it, reusing the FIRE TRIGGER machinery (including the optional FROM ... TO ... backfill window). Rather than naming a trigger, you name the object and Hoptimator discovers its upstream trigger(s). Grammar (regenerated via the Calcite codegen flow): REFRESH now takes an optional MATERIALIZED VIEW / TABLE keyword and an optional FROM/TO window, collapsed into a single SqlRefreshObject node (replacing the parse-only SqlRefreshMaterializedView). The object-type keyword is optional; when given it is validated against the resolved kind, so REFRESH TABLE on an MV (or vice versa) is rejected. REFRESH also errors when the object is unknown or has no upstream triggers — a refresh that silently fires nothing is a footgun. Discovery is decoupled from Kubernetes via a new RefreshProvider SPI (resolved by RefreshService in hoptimator-jdbc): given an object path it reports the object kind and the names of upstream triggers. K8sRefreshProvider implements it by treating the TableTriggers owned by the object's CRD (LogicalTable / materialized View) as its upstream triggers. The DDL layer never touches the k8s API. Tested: parser unit tests, executor unit tests (mocked RefreshService), K8s provider unit tests, and a live logical-refresh.id integration script. Docs updated (ddl-reference REFRESH section, extending/index SPI table). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…shProvider The graph layer already resolves an identifier to its kind (GraphService.resolve returns View for a materialized view, LogicalTable for a logical table) and builds the dependency graph with the object's owned triggers attached (OWNER_OF edges to Trigger nodes). REFRESH's discovery is exactly that data, so drop the dedicated RefreshProvider SPI + K8sRefreshProvider and resolve via GraphService instead. RefreshService now lives entirely in hoptimator-jdbc: it maps the GraphTarget to a kind and collects the root's OWNER_OF trigger nodes as the immediately-upstream triggers. RefreshTarget moves into the jdbc package (no longer a cross-module SPI type). No behavior change — the K8s graph builder already discovered owned triggers by CRD ownership, the same signal the old provider used. Tested: RefreshService unit tests (kind mapping + trigger extraction over synthetic graphs), unchanged executor tests, and the live logical-refresh.id script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
REFRESH <logical table> FROM a TO b backfills the table's offline-tier trigger over the given window. Add LogicalTableBackfillTest: it seeds the trigger's watermark to simulate processed history (a backfill is capped at the watermark), refreshes a window behind it, and asserts a one-off backfill Job is created over exactly [from, to] (WINDOW_START/WINDOW_END) and runs immediately (not suspended). Adds testCompileOnly findbugs.annotations to the module for the (test-only) SuppressFBWarnings on the intentionally-dynamic backfill-window DDL. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix stale references left after paring back — the bespoke RefreshProvider SPI was removed in favor of GraphService/GraphProvider-based discovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pivot from refreshing logical objects to refreshing PHYSICAL tables. Hoptimator doesn't distinguish logical from physical, and a consumer always reads a specific physical tier -- so 'refresh the tier I read' is what customers actually want, and it's unambiguous. This collapses the earlier complexity: no kind resolution, no MATERIALIZED VIEW/TABLE keyword, no tiers-in-DDL, no recursion. Grammar: REFRESH <table> [FROM a TO b] (regenerated; dropped the kind keyword). Discovery: RefreshService resolves the identifier and reads the producing trigger(s) off the one-hop dependency graph (trigger -> table edges). A logical table is rejected with a hint to refresh a tier; a physical table with no producer errors. Reuses the graph SPI end to end; the DDL layer never touches Kubernetes. Also fixes a real blocker: backfill Job names for real trigger names exceeded the 63-char Kubernetes limit. backfillJobName now shortens an over-long base to a readable prefix + a hash of the full base (deterministic, unique, valid). Grounded with PhysicalTableRefreshTest (integration): a reverse-ETL logical table (logical-retl) whose offline trigger writes the physical table ADS.CAMPAIGNS; the test seeds a watermark, REFRESHes ADS.CAMPAIGNS FROM a TO b, and asserts a backfill Job over exactly [from, to] -- plus the logical-table-rejected and unknown-table error paths. Unit tests: parser, executor, RefreshService.producingTriggers, and backfillJobName length capping. Removed the obsolete logical-object REFRESH script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ryannedolan
force-pushed
the
ryannedolan/refresh-command
branch
from
July 15, 2026 22:37
5edfb6b to
ba83ee4
Compare
srnand
reviewed
Jul 16, 2026
| throws SQLException { | ||
| String identifier = String.join(".", path); | ||
| GraphTarget target = GraphService.resolve(identifier, connection); | ||
| if (target instanceof GraphTarget.LogicalTable) { |
Collaborator
There was a problem hiding this comment.
do we want to allow REFRESH on view objects ? since GraphTarget could be a MV as well it seems (GraphTarget.View). need to probably add another condition here i guess since we only want to refresh physical tables.
Collaborator
Author
There was a problem hiding this comment.
Good call, will fix.
ryannedolan
commented
Jul 16, 2026
| * has zero or one producing trigger. The DDL layer never touches Kubernetes — the graph is built by | ||
| * a pluggable {@code GraphProvider}. | ||
| */ | ||
| final class RefreshService { |
Collaborator
Author
There was a problem hiding this comment.
We can probably just add these methods to GraphService. Not much here otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
REFRESHDDL commandDetails
Adds a
REFRESHDDL command that fires the trigger(s) which produce a physical table, with optional windowFROM...TO...:It's the table-level counterpart to
FIRE TRIGGERand reuses the same machinery.Testing
Create a logical table with a rETL trigger:
The resulting backfill trigger starts in
PAUSEDstate, as specified by the logical database's table templates:Refreshing a physical table associated with the logical table causes the backfill trigger to fire and a job to run:
The job: