fix: groupby-sum fast path reduces multi-dim DataArray grouper over all its dims#824
Merged
Conversation
…sum fast path Detect a genuine N-D grouper (all dims are data dims) and stack the data over those dims into a single key, so the fast path reduces over every grouper dim at once instead of routing a 2-D grouper through to_pandas() into the multikey path (which leaked one dim into the result).
Merging this PR will not alter performance
Comparing 
Footnotes
|
FabianHofmann
added a commit
to tpu-brunkhorst/linopy
that referenced
this pull request
Jul 14, 2026
Collaborator
|
@FabianHofmann Great that you found that bug right away. Im really sorry about the regression! EDIT: Nevermind, seems like it wasnt my PR... |
Collaborator
|
@FabianHofmann I dived a bit deeper in and found some other issues (positional alignment when using pd.Series or xr.DataArray). This is arguably a bug, but could also be part of v1. Ill create issues for both. We should decide if we treat this as a bugfix or a semantics thing! |
Collaborator
|
See #827 |
FBumann
approved these changes
Jul 14, 2026
Collaborator
|
@FabianHofmann This should be merged no matter what happens in #827 |
FabianHofmann
added a commit
to tpu-brunkhorst/linopy
that referenced
this pull request
Jul 15, 2026
FabianHofmann
added a commit
that referenced
this pull request
Jul 16, 2026
* added two example notebooks: energy-resource-build and sudoku * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Vectorize sudoku example and wire both examples into docs Replace the sudoku notebook's loops with idiomatic linopy: groupby for the square constraints, a broadcast MultiIndex-free mask for the hints, and a weighted sum for the grid reconstruction (aligns with v1 no-MultiIndex, #803). Add nblink files and toctree entries for the energy-resource-build and sudoku examples. Square groupby uses use_fallback=True pending #823. * Improve sudoku example's pedagogy Teach the why, not just the what: motivate the binary on/off encoding as a linearization, map each sum-constraint to its Sudoku rule, explain the groupby dimension collapse and the hint-mask broadcast, and frame use_fallback=True as a temporary #823 workaround. Drop the unused square_index_3d and define the variable directly; trim the random-data 3D plot; fix typos and terminology. * Reference groupby fix PR #824 in sudoku use_fallback note * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Drop use_fallback workaround now that groupby fix #824 is merged Merged master brings the #824 groupby-sum fix, so the square constraint uses plain .sum() and the temporary note is removed. Notebook re-executed end-to-end to confirm a valid solution. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix energy-resource-build notebook for newer pandas/numpy - Use freq='ME' instead of deprecated 'M' in pd.date_range - Replace legacy np.random.seed/randn with np.random.default_rng (NPY002) Notebook re-executes end-to-end. * Polish notebook prose: neutral voice, fix typos, consistent headings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add result cross-check to sudoku notebook; fix sum labels and section flow * Improve energy notebook pedagogy: explain the model, tidy comments and headings * Shorten energy notebook title --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #823.
This aligns the grouping mechanism across fallback and non-fallback groupby operations.
Note
The following content was generated by AI.
Changes proposed in this Pull Request
LinearExpression.groupby(...).sum()returned wrong dimensions when grouping by a multi-dimensionalDataArraygrouper on its default (fast) path: the reduction happened over only one of the grouper's dimensions and the others survived in the result. Onlyuse_fallback=Truegave the correct result.Root cause: the fast path did
group.to_pandas(), which turns a 2-D grouper into aDataFramethat is then routed through the multikey encoding path — reducing over only the row axis. A genuinely N-D grouper (whose values define the groups jointly over all its dims) was never handled. This is a long-standing bug present since v0.8.0, not a regression from #802.The fix:
DataArraywithndim > 1whose dims are all data dims — and stack the data over those dims into a single key, so the fast path reduces over every grouper dim at once. This preserves the fast path's memory benefit.DataArrays whose extra axis is not a data dim (i.e. multikey frames built from aDataFrame) keep flowing through the existingDataFrame/multikey handling._grouped_sumnow accepts the (stacked) dataset to operate on.Tests
test_linear_expression_groupby_ndimnow exercises the fast path too (use_fallback=[True, False]) and asserts it equals the fallback; the stale TODO was removed.test_linear_expression_groupby_multidim_preserves_extra_dimcovers the issue's reproducer (2-D grouper over(i, j), preservedk).Checklist
AGENTS.md).doc.doc/release_notes.rstof the upcoming release is included.