sql: fix variance and stddev aggregates with DISTINCT#37647
Open
ggevay wants to merge 1 commit into
Open
Conversation
var_samp, var_pop, variance, stddev_samp, stddev_pop, and stddev are
planned by decomposing into sum(x*x), sum(x), and count(x). The DISTINCT
flag was pushed into each component aggregate, but sum(DISTINCT x*x)
deduplicates on x*x rather than on x, wrongly collapsing input values
that differ only in sign. For example, over {-2.0, 2.0, 2.0} var_pop
(DISTINCT x) returned 2 instead of 4.
Fix by splitting the sum of squares by the sign of x when DISTINCT is
specified. Squaring is injective on the non-negative values and,
separately, on the negative values, so
sum(DISTINCT CASE WHEN x >= 0 THEN x*x END)
+ sum(DISTINCT CASE WHEN x < 0 THEN x*x END)
deduplicates exactly like sum(DISTINCT x) and count(DISTINCT x) do. The
non-DISTINCT expansion is unchanged.
Fixes SQL-362.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes SQL-362
Motivation
var_samp,var_pop,variance,stddev_samp,stddev_pop, andstddevwithDISTINCTreturned wrong results whenever the input contained values differing only in sign. For example, over{-2.0, 2.0, 2.0},var_pop(DISTINCT x)returned 2 instead of 4. They now match PostgreSQL.Description
These aggregates are planned in
transform_ast.rsby decomposing intosum(x*x),sum(x), andcount(x). TheDISTINCTflag was pushed into each component aggregate. That is correct forsum(DISTINCT x)andcount(DISTINCT x), which deduplicate on x, butsum(DISTINCT x*x)deduplicates on x², collapsing values that differ only in sign (e.g. -2 and 2 both square to 4, which was then counted once instead of twice).The fix computes the sum of squares with deduplication on x by splitting it by sign. Squaring is injective on the non-negative values and, separately, on the negative values, so
deduplicates exactly like the other two component aggregates.
COALESCEhandles an empty sign class. For empty or all-NULL input the overall result is still NULL becausesum(DISTINCT x)is NULL. The non-DISTINCTexpansion is unchanged. All six function names and all input types are covered, since they all funnel through the sameplan_varianceexpansion.Verification
New regression tests in
test/sqllogictest/aggregates.slt: the exact repro from the issue, all six functions, float/numeric/int/smallint/bigint/real inputs, NULLs (mixed and all-NULL), empty input, a single distinct value,DISTINCTcombined withFILTERand withGROUP BY, special float values (NaN, -Infinity, -0.0), the window-function error case, and unchanged non-DISTINCTresults. The new tests fail without the fix (9 failures) and pass with it. Expected values were verified against PostgreSQL.Additionally ran a locally generated differential test of 720 randomized checks (60 random datasets x 6 functions x DISTINCT/plain, biased toward sign-mirrored values, duplicates, and NULLs) with expected values taken from a live PostgreSQL instance: all pass. Also ran
test/sqllogictest/window_funcs.sltandtest/sqllogictest/github-5797.sltto cover the unchanged non-DISTINCTpaths.🤖 Generated with Claude Code