Normalizer-based SQL pattern recognition (planner + query engine + ingest) - #561
Draft
hshafqat-art wants to merge 10 commits into
Draft
Normalizer-based SQL pattern recognition (planner + query engine + ingest)#561hshafqat-art wants to merge 10 commits into
hshafqat-art wants to merge 10 commits into
Conversation
detects ClickHouse-specific complex SQL shapes (CTE/window functions, tokenized subqueries, MOAS, multi-aggregate) and rewrites them into simplified queries. Both the planner and the query engine call into this implementation. Also adds ComputedLabelConfig/StatefulTransitionConfig as shared asap_types, wires ingest-time computed-label and stateful-transition execution, and adds the 200-query bgp_jan2024_rrc00 analyst workload.
Contributor
|
FYI @zzylol |
Contributor
|
@hshafqat-art This is my understanding of what's in here. Can you confirm if this is correct and complete? 1. SQL pattern-rewrite feature — detecting ClickHouse-specific complex shapes (CTE/window funcs, tokenized subqueries, MOAS, multi-aggregate, bucketed countIf) and rewriting/planning/serving them. Spans 2. CSV ingest: accept ClickHouse 3. Finite CSV-ingest lifecycle fixes — explicit 4. Ingest-routing now enforces 5. BGP query workload. Already taken care of by ProjectASAP/ASAPPlanner#219 |
…, derived values, and stateful transitions Adds a derived_value module for representing ingest-time-derived metric streams (raw-column re-exposure, ArgMax comparison-key tracking, gap/ transition state) as their own virtual tables. Extends stateful_transition's config to carry gap-mode fields alongside its existing boolean-predicate mode. Widens capability_matching's compatible-type and label-union logic so Sum/Count can be served from a CountMinSketch-backed candidate and ArgMax/ SetAggregator-backed aggregations route through grouping_labels/ aggregated_labels consistently, with a deterministic priority tie-break when multiple candidate window sizes are equal. Adds the exact-cardinality (SetAggregator) branch to map_statistic_to_precompute_operator and the matching subpopulation-support entry, alongside the existing approximate (HLL) path.
…pe-recognition vocabulary Splits the per-row ingest-time predicate parser (equality, ranges, IN-lists, match(), has(splitByChar(...)), day-of-week, etc.) into its own spatial_filter module, shared by both the planner and the query engine so a WHERE-clause predicate is judged ingest-filter-enforceable identically by whichever one asks. Adds detectors and surrogate builders for a large batch of query shapes to pattern_rewrites: AVG-by-composition (sum+count split), lag-gap (stateful inter-arrival derivation), correlated IN/NOT-IN subqueries (with table-alias stripping so an aliased outer query still matches every downstream bare- identifier parser), grouped/scalar exact-cardinality (uniqExact/uniq), group-by-having-count, hidden-countif-having, two-stage histograms, weekly MOAS histograms, grand-total percentage, running total, bucketed top-N, and derived ratios - plus the from_table_name fix that strips a database qualifier (`bgp.bgp_updates` -> `bgp`) so schema lookups for these shapes resolve against the table name the schema actually registers.
…nt and dual-label routing Wires the planner side of every shape added in sql_utilities: dedicated registration functions for AVG (two independent Sum/Count aggregations sharing a derived table), lag-gap (a new virtual TableDefinition for the derived "gap" metric), ArgMax over a raw/computed column, computed-GROUP-BY (plain and combined with a raw/computed value), token-select/explode, arrayZip edge-explode, bucketed countIf and top-N, and the correlated- subquery split (each half planned as if submitted standalone, arg-agg tried first). Fixes map_statistic_to_precompute_operator's Cardinality arm, which previously ignored QueryTreatmentType entirely and always produced the approximate HLL sketch even when the SQL said uniqExact - the exactness is now recovered from the surrogate text (sqlpattern_parser normalizes uniqExact/uniq/uniqCombined to the same name upstream) and, for the exact per-group case, both grouping_labels (the GROUP BY column) and aggregated_labels (the counted column) are set together, matching MOAS's existing dual-label pattern - the generic single-label routing has no way to express "partition by one column, count distinct values of another" on its own.
…eful transitions into every ingest source Adds direct MRT-file ingest (mrt_ingest.rs: bgpkit-parser field mapping to the fixed physical BGP row schema) and MrtDirectoryIngestSource / MrtBatchDirectoryIngestSource for watching or batch-replaying a directory of MRT dumps, including a paced mode that replays real per-file arrival cadence at an acceleration factor rather than ingesting at ceiling throughput. row_expansion.rs generalizes the row -> sample(s) mapping to support fanout (one row producing multiple derived observations, e.g. every ASN in an AS path) alongside the existing 1:1 case, and derived_value.rs represents a raw/computed column re-exposed as its own virtual metric stream. Fixes StatefulTransitionOperator (lag-gap/lag-transition state tracking) being wired into CSV ingest only - MRT-sourced ingest silently dropped every stateful-transition config despite the planner correctly registering it, so any lag-gap/lag-transition query was unfulfillable end to end whenever data arrived via MRT rather than CSV. Now threaded through every MRT ingest variant (single-file, directory-watch, and batch-directory, including its bounded-concurrency and paced code paths), each building a fresh operator per file since files are ingested with bounded concurrency rather than one well-ordered stream.
…or, fix SetAggregator's numeric query path Adds MultipleArg, tracking the value of one column at the row where a comparison column was extremal, per key - backs the argMax/argMin shape. Fixes SetAggregatorAccumulator::query_statistic, previously an unconditional stub returning "does not support query operation" for every call. It had only ever been read through get_keys() (SELECT DISTINCT/MOAS rendering the raw member list), so the stub was never exercised until exact per-group cardinality started registering SetAggregator for a numeric Statistic:: Cardinality request - now returns the set's own size directly for Cardinality, matching HLL's single-instance-per-key model (the key parameter is unused: an instance already IS one key's own member set, no lookup needed).
…correlated-subquery dispatch
Adds the query-time handler for each shape registered on the planner side:
AVG (execute sum/count surrogates independently, divide per key), lag-gap/
lag-transition (served over their derived virtual metric), ArgMax, MOAS and
weekly-MOAS-histogram, SELECT DISTINCT and groupArray(DISTINCT), token-
select/explode and arrayZip edge-explode, computed-GROUP-BY, bucketed
countIf/top-N, group-by-having-count, hidden-countif-having, two-stage
histograms, grand-total percentage, running total, derived ratios, and
multi-aggregate (including the ArgMax/groupArray(DISTINCT)-aware all-labels
combiner for a multi-aggregate row that isn't reducible to one plain numeric
value). Threads a new stateful_transitions field through MrtBatchDirectory/
MrtDirectory/Mrt engine-config variants and merges the planner-detected set
into each at startup, matching the merge the Csv arm already had.
Fixes two correctness gaps:
- An empty precomputed-output set was always treated as an error ("No
precomputed outputs found"), even when it's the store's only way to
represent a legitimately empty/zero answer - a Count/Sum/Cardinality/
Increase/Rate query whose filter matches nothing has a well-defined
answer (0) for an ungrouped request, or zero output rows for a grouped
one; only a statistic with no defined value over an empty population
(Min/Max/Quantile/Topk/ArgMax/ArgMin) still needs to error. "Grouped"
is read from the query's own parsed output labels, not the matched
aggregation config's internal grouping (the two can disagree - a real
GROUP BY query can resolve to a self-keyed config whose own grouping
is empty). collect_results_same_aggregation is also fixed to route a
keyed SetAggregator answering a numeric Cardinality request through its
single-value path rather than iterating its member set as if each member
were its own output key.
- resolve_correlated_half (the correlated-subquery split's per-half
resolver) didn't apply the same rewrite_recognized_pattern pass the
top-level query handler always runs first, so an ArgMax half's dedicated
fast path (which needs that rewrite to find the derived-value table the
planner registered it under) silently fell through to a generic path that
can't represent ArgMax's result shape, breaking the join.
Test-utility and fixture updates plus new coverage for exact-cardinality registration, capability matching's Sum/Count/ArgMax candidate handling, and the store-correctness/e2e suites exercising the new ingest and serving paths.
Cargo.lock reflects the dependency changes from the preceding commits.
Documents the (rho, tau, gamma, eval, pi) decomposition the SQL path uses to decide which analyst queries can be answered from a continuously maintained summary instead of a full scan: how a query is recognized and split into a maintainable piece plus a residual computation (rho), how raw records become the observations a shape needs (tau - stateless projection, fanout, or stateful comparison), what gets continuously maintained (gamma), and what's deferred to query time (pi). Also covers spatial-filter enforceability as a precondition inside rho's own "can this be maintained" decision, not a peer stage - a query can be an unambiguous shape match and still fail to register if one of its WHERE predicates isn't one the ingest-time filter recognizes. Complements CAPABILITY_MATCHING_DESIGN.md, which covers the serve-time matching layer beneath this one. The per-shape catalog (recognizer, maintained structure, and serving function for each recognized shape) is deferred to a follow-up pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Also adds ComputedLabelConfig/StatefulTransitionConfig as shared asap_types,
wires ingest-time computed-label and stateful-transition execution, and adds
the 200-query bgp_jan2024_rrc00 analyst workload.