Skip to content

encoding and decoding using custom coders - #11541

Draft
keewis wants to merge 7 commits into
pydata:mainfrom
keewis:coders
Draft

encoding and decoding using custom coders#11541
keewis wants to merge 7 commits into
pydata:mainfrom
keewis:coders

Conversation

@keewis

@keewis keewis commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

In xarray, we have a bunch of existing coders / coding functions suitable to encode or decode data and metadata (mainly variables) encoded according to the CF conventions. However, since the CF conventions are pretty specific to the climate and forcasting community, other communities / science domains have other kinds of conventions that I believe xarray should have API to support. Additionally, supporting additional conventions (like the growing number of zarr conventions) will allow third-party packages to integrate with xarray without having to build composite or duplicate backends.

This is mainly important when doing I/O (serializing from / to disk), but could also be used to convert between the conventions from different communities.

The idea of custom coders has been around for quite a long time, with the first mention being in 2014. Related issues include:

I have also been trying to work on this for quite a while now, so to give a preview of the current state I'm opening this draft PR. To keep the scope of this PR small, I'm only including:

  • the protocol for custom coders for Dataset / Variable
  • the encode and decode functions that are new public API
  • some examples of where this can be used. In particular: encoding a range index, and replacing the current functions to encode which variable is a coordinate

Intentionally missing:

  • integration with backends (and the current defaults, even though a skeleton for a cf_coders function is included at the moment)
  • refactoring of the existing coder objects
  • integration with the lazy array machinery (for applying elementwise functions or array type conversions)
  • an API to allow indexes to serialize / deserialize themselves without needing to pair with a custom coder

The design notes are also currently outdated, my source of truth is the hackmd I've repeatedly mentioned.

cc @ianhi, @maxrjones, @benbovy, @kmuehlbauer, @shoyer

Edit: the RangeIndex encoding is pretty much arbitrary and just there to show what we can do with dataset coders, if we find a better way to encode this than the plain ranges dict I'm happy to change the implementation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant