indexing by a list of slices

[keewis]

Is your feature request related to a problem?

In some situations, it is important to be able to select multiple (disconnected) contiguous regions along a dimension, even if the dimension itself is very large.

If there's enough client memory, it is possible to emulate this by materializing the slices into an array of integers, but this becomes infeasible if that array is too large (and while supposedly we should be able to index by a dask integer array, I'm not sure how efficient that would be).

Examples of where this would be useful include:

• The healpix MOC index at healpix moc index xarray-contrib/xdggs#151, where cell ids are represented as a set of disconnected ranges at the smallest possible refinement level. To be able to support Index.sel, I'd need to return a IndexSelResult with either a list of slices, or materialize these into an integer array and error out if that wouldn't fit into memory (or try to use dask as an indexer).
• @tomwhite's use-case of selecting disconnected regions in a genome (see Add bcftools-style filtering sgkit-dev/sgkit#1330 (comment)). I'll let him provide further details.

cc @benbovy, @shoyer, @TomNicholas, @dcherian

Describe the solution you'd like

I'd love to be able to specify this as another kind of indexer:

indexer = SliceSet([slice(20, 5000), slice(12078432, 1850372894)])
ds.isel(cells=indexer)

but that will obviously further increase the complexity of the indexing machinery

Describe alternatives you've considered

Manually iterating of the slices, then concatenating the result is possible, but will have an additional overhead if done using the xarray API. However, I don't see a way that can work as part of IndexSelResult.

[benbovy]

Manually iterating of the slices, then concatenating the result is possible

Easiest I think would be to do that at the Variable level, i.e., implement a special case in Variable.isel() that will iterate over each slice and call Variable.concat() as needed. There may be some overhead but I think it would be limited at that level. If necessary it's still possible to introduce some internal fastpath in Variable.concat() or maintain an optimized internal version.

For coordinate variables associated with a custom index, there's the possibility of implementing custom logic in Index.isel() (returning a new index object) and Index.create_variables() (create variables from the new index object) overridden methods. This will bypass Variable.isel() for those variables.

[shoyer]

My most immediate concern is that even if we decided that this API expansion is a good idea, we currently do not have the lazy concatenate machinery that would be required to make this sort of indexing lazy, like all other forms of xarray indexing.

…

On Tue, Jul 1, 2025 at 7:18 AM Benoit Bovy ***@***.***> wrote: *benbovy* left a comment (pydata/xarray#10479) <#10479 (comment)> Manually iterating of the slices, then concatenating the result is possible Easiest I think would be to do that at the Variable level, i.e., implement a special case in Variable.isel() that will iterate over each slice and call Variable.concat() as needed. There may be some overhead but I think it would be limited at that level. If necessary it's still possible to introduce some internal fastpath in Variable.concat() or maintain an optimized internal version. For coordinate variables associated with a custom index, there's the possibility of implementing custom logic in Index.isel() (returning a new index object) and Index.create_variables() (create variables from the new index object) overridden methods. This will bypass Variable.isel() for those variables. — Reply to this email directly, view it on GitHub <#10479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJFVXXIAE7GWRPS5LUZND3GKKDRAVCNFSM6AAAAACARDVGR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMRUGI2DKNBZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[keewis]

Personally, my main motivation for this would be the return value of Index.sel, which we can't work around in user code.

As for the lazy concatenation, I believe there's #10402 as an experiment on how this could be implemented.

[TomNicholas]

I am curious if this has any crossover with what flox does to create groups

[ilan-gold]

Maybe relevant: zarr-developers/zarr-python#3175

[dcherian]

technically we don't need lazy concat here. We can hack it using the fact that LazilyIndexedArray acts as a blocker on indexing; i.e. it only passes through indexers at compute time.

So you could work with LazilyIndexedArray(MultiplySlicedArray(large_lazy_array)). This would only need concat at runtime.

[keewis]

that's a good point! This does work, but will need some thought on how to implement _updated_key (and _index_indexer_1d)

[dcherian]

But doesn't that just condense all indexers down to a unified indexer that is then passed to LazilyIndexedArray.array?

[keewis]

yes, exactly. I'll have to think a bit more about whether implementing that is actually necessary, but it doesn't currently support lists of slices.

[dcherian]

Yes I'm suggesting that you handle the list of slices elsewhere, wrap with LazilyIndexedArray and then assign to Variable