CSHARP-5495: Support $concatArrays and $setUnion aggregation accumulators by papafe · Pull Request #2014 · mongodb/mongo-csharp-driver

papafe · 2026-05-27T17:49:35Z

No description provided.

…tors Map g.SelectMany(x => x.Arr) and g.SelectMany(x => x.Arr).Distinct() to the new $concatArrays and $setUnion accumulators inside $group, $bucket and $bucketAuto via two new pattern matches in AstGroupingPipelineOptimizer. Add ConcatArrays and SetUnion extension methods on ISetWindowFieldsPartitionExtensions for use inside $setWindowFields.

…accumulators

…etunion-accumulators # Conflicts: # src/MongoDB.Driver/Core/Misc/Feature.cs

Copilot

Pull request overview

Adds LINQ3 translation support for MongoDB’s $concatArrays and $setUnion accumulators/window operators, including new window-partition extension methods, AST operator wiring, serializer inference, and integration tests for both $setWindowFields and grouping/bucket scenarios.

Changes:

Added ConcatArrays / SetUnion window-partition extension methods and reflection/translation plumbing to render $concatArrays and $setUnion.
Updated grouping pipeline optimizer to recognize SelectMany-shaped ASTs and rewrite them into $concatArrays / $setUnion accumulators.
Expanded test coverage for new operators in window and grouping/bucket contexts, plus serializer-finder expectations.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/MongoDB.Driver.Tests/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MethodTranslators/WindowMethodToAggregationExpressionTranslatorTests.cs	Adds window translation tests for `$concatArrays` and `$setUnion`.
tests/MongoDB.Driver.Tests/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MethodTranslators/SelectManyMethodToAggregationExpressionTranslatorTests.cs	Adds grouping/bucket tests expecting `$concatArrays`/`$setUnion` accumulators; adjusts fixture data and existing tests.
tests/MongoDB.Driver.Tests/Linq/Linq3Implementation/SerializerFinders/WindowTests.cs	Adds serializer resolution coverage for new window methods.
src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MethodTranslators/WindowMethodToAggregationExpressionTranslator.cs	Registers `ConcatArrays`/`SetUnion` as unary window methods and maps them to AST operators.
src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MethodTranslators/ConcatArraysMethodToAggregationExpressionTranslator.cs	New method translator entry point delegating to window translation when applicable.
src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MethodTranslators/SetUnionMethodToAggregationExpressionTranslator.cs	New method translator entry point delegating to window translation when applicable.
src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MethodCallExpressionToAggregationExpressionTranslator.cs	Routes `ConcatArrays`/`SetUnion` method calls to the new translators.
src/MongoDB.Driver/Linq/Linq3Implementation/SerializerFinders/SerializerFinderVisitMethodCall.cs	Adds serializer deduction logic for `ConcatArrays`/`SetUnion` window methods.
src/MongoDB.Driver/Linq/Linq3Implementation/Reflection/WindowMethod.cs	Adds `WindowMethod.ConcatArrays` and `WindowMethod.SetUnion` reflection bindings.
src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Optimizers/AstGroupingPipelineOptimizer.cs	Adds optimizer rewrites to turn SelectMany-shaped `$reduce` patterns into `$concatArrays`/`$setUnion` accumulators.
src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstUnaryWindowOperator.cs	Adds renderable window operator enum values for `$concatArrays`/`$setUnion`.
src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Expressions/AstUnaryAccumulatorOperator.cs	Adds renderable accumulator operator enum values for `$concatArrays`/`$setUnion`.
src/MongoDB.Driver/Linq/ISetWindowFieldsPartitionExtensions.cs	Adds public LINQ window APIs: `ConcatArrays` and `SetUnion`.
src/MongoDB.Driver/Core/Misc/Feature.cs	Introduces feature gate `ConcatArraysAndSetUnionAccumulators` (WireVersion.Server81).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

The grouping/bucket SelectMany rewrite into $concatArrays/$setUnion accumulators requires server 8.1. Since that query shape was already translatable (as $push + $reduce, which runs on any server), emit the accumulator form only when the compatibility level supports it; otherwise keep the $reduce fallback. The $setWindowFields window methods are new API and stay unconditional.

- Add missing closing brace in SelectMany_Distinct_in_BucketAuto else-branch expected stage - Assert exact UniqueTags contents instead of only HaveCount - Reorder ConcatArrays/SetUnion alphabetically in WindowMethod and unary overload set

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

…ator

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.

sanych-sun · 2026-06-12T22:46:26Z

            }

+            private bool ArrayAccumulatorsSupported =>
+                Feature.ConcatArraysAndSetUnionAccumulators.IsSupported((_translationOptions?.CompatibilityLevel).ToWireVersion());


I know it is probably outside of this PR scope, but should we consider to change the ToWireVersion, to return min supported wire version for the null parameter? Because otherwise it's hardcoded to 4.0, which is 2 versions under our lowest supported version.

sanych-sun · 2026-06-12T22:48:23Z

+                _translationOptions = translationOptions;
            }

+            private bool ArrayAccumulatorsSupported =>


Property name seems to be misleading a little. Especially when we are using it to decide if we can use setUnion.

I agree, changed it.

sanych-sun · 2026-06-12T23:06:05Z

+
+    [Theory]
+    [InlineData(ServerVersion.Server80)]
+    [InlineData(ServerVersion.Server81)]


And here I'm confused again: If we run the test on MongoDB 5.0 and set compatibility level to 8.0 - should this test fail with some server side error? Should we skip the test on older versions?

This test is more of a regression check. In this case we're testing that if we have a SelectMany with index, we do not convert it to an accumulator, otherwise the index information will be unbound and rejected by the server.
In this case we want to be sure that independently from the compatibility level, we do not use the new accumulators (that were added in version 8.1). Since we use the "old" MQL, this is compatible on all versions.

If we set compatibility level to ServerVersion.Server81 shouldn't we use the new functionality?

No, if we have SelectMany with index (g.SelectMany((x, i) => ...) we can't use the new accumulator, this is what the test is verifying. If we have a SelectMany without index (g.SelectMany( x => ...), then we can use the new accumulator.

papafe · 2026-06-15T11:55:42Z

+    [Theory]
+    [InlineData(ServerVersion.Server81, true)]
+    [InlineData(ServerVersion.Server80, false)]
+    public void SelectMany_in_GroupBy_should_emit_accumulator_only_when_CompatibilityLevel_supports_it(ServerVersion compatibilityLevel, bool expectAccumulator)


Regarding this test and the following... These could be possibly considered superfluous, but they test that the MQL is actually independent from the server version but only depends on the compatibility leve.

So this test actually does not make all the way to the database? I would say it's unusual... I would prefer to actually execute the request and assert MQL based on the expected version, and skip test if we are trying to set the compatibility level to the version which is above the actual server version.

Yes, they do not go all the way to the DB, because setting up the compatibility level to an arbitrary value could generate an MQL that is not valid (for example compatibility level 8.1 on a 5.0 server).
If you think about a matrix of server version (less than 8.1, from 8.1) and compatibility level (server80, server81), all the other tests here are checking only the "diagonal", so when the compatibility level is the same as the server version. This test here instead is checking all the possible combinations, as we can also have compatibility level equal to server81 and the actual server version is 5.0.
I would say this test is more for completeness, but I can understand if we want to remove it and keep the more direct behaviour. What do you think?

papafe added the feature Adds new user-facing functionality. label May 27, 2026

sanych-sun reviewed Jun 1, 2026

View reviewed changes

Comment thread ...MongoDB.Driver/Linq/Linq3Implementation/SerializerFinders/SerializerFinderVisitMethodCall.cs

papafe added 2 commits June 2, 2026 16:06

Add SerializerFinder unit tests for ConcatArrays and SetUnion window …

f9b919a

…accumulators

Merge remote-tracking branch 'upstream/main' into csharp5495-concat-s…

975069c

…etunion-accumulators # Conflicts: # src/MongoDB.Driver/Core/Misc/Feature.cs

papafe marked this pull request as ready for review June 2, 2026 14:28

papafe requested a review from a team as a code owner June 2, 2026 14:28

papafe requested review from ajcvickers and Copilot June 2, 2026 14:28

Copilot started reviewing on behalf of papafe June 2, 2026 14:28 View session

papafe requested review from sanych-sun and removed request for ajcvickers June 2, 2026 14:28

Copilot AI reviewed Jun 2, 2026

View reviewed changes

papafe added 4 commits June 3, 2026 12:16

Small fix

6836585

Add CompatibilityLevel gating tests for SelectMany accumulator rewrite

ee29af9

papafe requested a review from Copilot June 8, 2026 13:10

Copilot started reviewing on behalf of papafe June 8, 2026 13:10 View session

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread src/MongoDB.Driver/Linq/Linq3Implementation/Ast/Optimizers/AstGroupingPipelineOptimizer.cs

Don't rewrite indexed SelectMany into $concatArrays/$setUnion accumul…

1031bd4

…ator

papafe requested a review from Copilot June 12, 2026 09:29

Copilot started reviewing on behalf of papafe June 12, 2026 09:30 View session

Copilot AI reviewed Jun 12, 2026

View reviewed changes

papafe added 3 commits June 12, 2026 12:21

Add SetUnion index-overload guard test; gate tests by CompatibilityLevel

ae97caf

Clarify IsSelectManyShapeOverElements comment with rewrite intent

d0f1f13

Simplify arrayIndexAs guard comment

a7c0aae

sanych-sun requested changes Jun 12, 2026

View reviewed changes

Rename ArrayAccumulatorsSupported to match feature name

fd0a3ee

papafe commented Jun 15, 2026

View reviewed changes

Conversation

papafe commented May 27, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

papafe Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

papafe Jun 15, 2026 •

edited

Loading