GEOMESA-3582 Removes BBOX predicates from datastore SQL, adds as feature flag to the connector#3589
Open
cwdobbins wants to merge 3 commits into
Open
GEOMESA-3582 Removes BBOX predicates from datastore SQL, adds as feature flag to the connector#3589cwdobbins wants to merge 3 commits into
cwdobbins wants to merge 3 commits into
Conversation
…ng datastore SQL as limited to ST_* predicates.
…uiting logic to the plugin connector, which shortcuts the more expensive WKB decode during row-filtering, provides additional performance improvement for coarsely-pruned datasets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GEOMESA-3582: consolidate spatial pushdown in the connector
Splits spatial-filter handling cleanly between the two
geomesa-trinomodules and makes query plans deterministic regardless of how a query arrives.Background
Previously the datastore rewrote CQL spatial predicates into composite SQL —
__X_bbox__overlap prefilters plusCASE WHENrow-level shortcuts — and thespatial_icebergconnector independently derived the same pruning from eitherST_*calls or the recognized bbox-comparison pattern. The same logical query could therefore produce different plans depending on whether it came through the datastore or as hand-written Trino SQL, and near-identical logic was maintained in two places.1. Datastore emits pure
ST_*(commit 1)TrinoFilterToSQLnow emits only row-levelST_*function calls (plus temporal / attribute predicates). Swapped operand forms are normalized column-first via the OGC complement (within(a,b) ⇔ contains(b,a)) so the connector'sST_*(ST_GeomFromBinary(col), literal)recognizer always matches. The connector is now the single place bbox/Z2/XZ2 pushdown is derived, so datastore queries and hand-written Trino SQL get identical treatment and deterministic plans.2. Connector-side page-source bbox reject (commit 2, configuration-enabled via feature flag)
Emitting pure
ST_*removes the datastore's inlineCASE WHENrow-level shortcut, which had let a row whose bbox was inside the query rectangle skip the WKB decode. This commit optionally recovers a sound version of that saving entirely inside the connector, with no datastore coupling.The connector already injects the query envelope as REAL domains on a geometry's
__X_bbox__sub-fields for stat pruning; those domains ride to the worker in the table handle'sunenforcedPredicate.SpatialPageSourceProviderwraps the Iceberg page source, reads only those cheap bbox columns, and drops any row whose bbox cannot overlap theenvelope before the geometry column is materialized (via
SourcePage.selectPositions), so the engine's exactST_Intersects/ST_Withinresidual never decodes those rows' WKB.FilterParityITand by cross-checking counts against the stockicebergcatalog (no pushdown) across rectangle, triangle,ST_Within, and non-spatial filters.geomesa.spatial.bbox-page-filter): its value is inversely proportional to spatial partition quality — on a well-Z2/XZ2-partitioned table, file/row-group pruning already removes most non-matching rows, so the extra bbox-column read can outweigh the saved decodes. Enable it for coarsely-partitioned tables, where row-level rejection catches what file pruning misses.