encoding and decoding using custom coders - #11541
Draft
keewis wants to merge 7 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In xarray, we have a bunch of existing coders / coding functions suitable to encode or decode data and metadata (mainly variables) encoded according to the CF conventions. However, since the CF conventions are pretty specific to the climate and forcasting community, other communities / science domains have other kinds of conventions that I believe xarray should have API to support. Additionally, supporting additional conventions (like the growing number of zarr conventions) will allow third-party packages to integrate with xarray without having to build composite or duplicate backends.
This is mainly important when doing I/O (serializing from / to disk), but could also be used to convert between the conventions from different communities.
The idea of custom coders has been around for quite a long time, with the first mention being in 2014. Related issues include:
I have also been trying to work on this for quite a while now, so to give a preview of the current state I'm opening this draft PR. To keep the scope of this PR small, I'm only including:
Dataset/Variableencodeanddecodefunctions that are new public APIIntentionally missing:
cf_codersfunction is included at the moment)The design notes are also currently outdated, my source of truth is the hackmd I've repeatedly mentioned.
cc @ianhi, @maxrjones, @benbovy, @kmuehlbauer, @shoyer
Edit: the
RangeIndexencoding is pretty much arbitrary and just there to show what we can do with dataset coders, if we find a better way to encode this than the plainrangesdict I'm happy to change the implementation