IGNITE-27822 Support recursive query for Calcite engine - #13479
Open
vldpyatkov wants to merge 9 commits into
Open
IGNITE-27822 Support recursive query for Calcite engine#13479vldpyatkov wants to merge 9 commits into
vldpyatkov wants to merge 9 commits into
Conversation
…hin planning context.
There was a problem hiding this comment.
Pull request overview
Adds support for recursive common table expressions (CTEs) in Ignite’s Calcite-based engine by introducing dedicated planner rules and coordinator-side execution nodes, along with SQL and Java integration tests to validate planning, execution, and quota behavior.
Changes:
- Introduce new physical rels and execution nodes for recursive UNION ALL evaluation and query-local recursive “delta” state management.
- Add planner converter rules and utilities to recognize/validate Calcite recursive CTE structures and optimize invariant subtrees.
- Add SQL and integration tests covering hierarchy traversal, sequence generation, subquery/correlation usage, plan shape, and memory quota enforcement.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| modules/calcite/src/test/sql/hierarchy/test_recursive_subquery.test | Adds SQL-script tests for recursive CTEs used inside subqueries/correlated EXISTS. |
| modules/calcite/src/test/sql/hierarchy/test_recursive_sequence.test | Adds SQL-script tests for recursive sequence generation patterns. |
| modules/calcite/src/test/sql/hierarchy/test_recursive_hierarchy.test | Adds SQL-script tests for hierarchical traversal and empty-seed behavior. |
| modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite/integration/RecursiveCteIntegrationTest.java | Expands integration coverage for recursive CTE execution, explain-plan shape, validation errors, and quotas. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rule/RecursiveTableSpoolConverterRule.java | Converts Calcite transient table spools to recursive delta spools. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rule/RecursiveTableScanConverterRule.java | Converts transient recursive table scans to query-local delta scans. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rule/RecursiveStaticSpoolConverterRule.java | Materializes invariant recursive-term subtrees for reuse across iterations. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rule/RecursiveCteUtils.java | Adds utilities for identifying recursive scans, counting references, invariance checks, and state IDs. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rule/RecursiveCteConverterRule.java | Converts Calcite LogicalRepeatUnion into Ignite coordinator-side recursive union execution. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/logical/IgniteLogicalRecursiveStaticSpool.java | Adds a logical marker node for static recursive-term inputs that should be spooled. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteRepeatUnion.java | Introduces a physical rel representing coordinator-side iterative UNION ALL. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteRelVisitor.java | Extends the visitor interface to include the new recursive rels. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteRecursiveTableSpool.java | Adds a physical rel to commit/replace the current recursive delta at end-of-input. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/rel/IgniteRecursiveTableScan.java | Adds a physical rel to scan the current query-local recursive delta. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/prepare/PlannerPhase.java | Registers the new recursive planning rules into the planning pipeline. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/prepare/IgniteRelShuttle.java | Updates rel shuttle to traverse/handle new recursive rels. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/prepare/Cloner.java | Updates cloning logic to support new recursive rels. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/prepare/BaseQueryContext.java | Adds per-planning-context query-local IDs for recursive transient tables. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/metadata/IgniteMdFragmentMapping.java | Adds fragment mapping for recursive table scan as coordinator-local. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/RepeatUnionNode.java | Adds coordinator-side executor implementing iterative recursive UNION ALL control flow. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/RecursiveTableSpoolNode.java | Adds executor node that atomically commits the next delta at end-of-input. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/rel/RecursiveCteState.java | Adds query-local recursive state storage with memory tracking. |
| modules/calcite/src/main/java/org/apache/ignite/internal/processors/query/calcite/exec/LogicalRelImplementor.java | Wires new recursive rels into execution node construction and shared state management. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+20
to
+36
| import java.util.List; | ||
| import org.apache.calcite.plan.RelTraitSet; | ||
| import org.apache.calcite.rel.RelNode; | ||
| import org.apache.calcite.rel.SingleRel; | ||
|
|
||
| /** Logical marker for a static recursive-term input that must be materialized on the coordinator. */ | ||
| public class IgniteLogicalRecursiveStaticSpool extends SingleRel { | ||
| /** */ | ||
| public IgniteLogicalRecursiveStaticSpool(RelNode input) { | ||
| super(input.getCluster(), input.getCluster().traitSet(), input); | ||
| } | ||
|
|
||
| /** {@inheritDoc} */ | ||
| @Override public RelNode copy(RelTraitSet traitSet, List<RelNode> inputs) { | ||
| return new IgniteLogicalRecursiveStaticSpool(sole(inputs)); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
The same comment from me - seems you can merge RecursiveTableSpoolNode and existing TableSpoolNode, also i can`t find different implementations in calcite thus i think - this is correct approach
Contributor
Author
There was a problem hiding this comment.
- IgniteLogicalRecursiveStaticSpool#copy is definitely ignoring traitSet, which formally is a contract violation. (fixed)
- I believe that the traitSet has not to be inherited for input. Look at the Calcite class LogicalTableSpool.
- RecursiveTableSpoolNode and TableSpoolNode are different classes. TableSpoolNode accumulates rows one time and replays them on rewind(), but RecursiveTableSpoolNode makes a new row set in each iteration. We can use TableSpoolNode for only static datasets. Look at RecursiveStaticSpoolConverterRule. It creates IgniteTableSpool, which uses TableSpoolNode.
Contributor
|
ok, looks good for me, let`s wait additional review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/IGNITE-27822