Experimental lazy geff - #334
Open
JoOkuma wants to merge 4 commits into
Open
Conversation
Compare cold loading and steady-state query performance for the Rustworkx, indexed, in-memory SQL, and lazy Zarr SQL graph backends.\n\nUnify Zarr-backed xarray chunks and order query results deterministically so larger multi-chunk GEFF fixtures can be benchmarked reliably.
Preserve the full GEFF uint64 node ID range, reject unsupported undirected stores explicitly, and make read-only copies reopen their backing store. Defer DataFusion import to avoid pytest-cov logger initialization failures.\n\nExpand regression coverage for laziness, nullable values, Zarr v2, pickling, chunk layouts, and backend constraints, and document the supported query behavior.
Use the Zarr v2/v3-compatible test array creation APIs so Python 3.10's Zarr 2 environment can run the lazy backend tests. Skip the new Zarr backend benchmark when ASV evaluates the main revision where that backend does not yet exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc: @cmalinmayor @TeunHuijben
I recently learned of some SQL interfaces for zarr.
I asked an LLM to implement a backend using them for a 100% lazy geff.
It might be interesting for you guys, but it doesn't support edits.
It can be very useful for data loading graphs for model training.
Unless there's a lot of interest, I won't merge this.