-
Notifications
You must be signed in to change notification settings - Fork 17
Introduced new test_read_netcdf.py function for directly converting netCDF files into XarrayContext. #226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Introduced new test_read_netcdf.py function for directly converting netCDF files into XarrayContext. #226
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| from metpy.cbook import get_test_data | ||
| from xarray_sql import XarrayContext | ||
|
|
||
| path = get_test_data('irma_gfs_example.nc', False) | ||
| print(type(path), path) | ||
|
|
||
| ctx = XarrayContext.read_netcdf(get_test_data('irma_gfs_example.nc', False)) | ||
| print(ctx._registered_datasets.keys()) | ||
|
|
||
| result = ctx.sql("SELECT * FROM irma_gfs_example.time1_isobaric1_latitude_longitude LIMIT 5") | ||
| print(result) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,7 +7,8 @@ | |
| from .df import Chunks | ||
| from .ds import XarrayDataFrame | ||
| from .reader import read_xarray_table | ||
|
|
||
| from pathlib import Path | ||
| from matplotlib.dates import date2num | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we import this? |
||
|
|
||
| class XarrayContext(SessionContext): | ||
| """A datafusion `SessionContext` that also supports `xarray.Dataset`s.""" | ||
|
|
@@ -156,6 +157,55 @@ def _maybe_register_cftime_udf(self, ds: xr.Dataset) -> None: | |
| self.register_udf(cft.make_cftime_udf(units, cal)) | ||
| break # One UDF per context is enough. | ||
|
|
||
| @classmethod | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this should be an instance method. |
||
| def read_netcdf( | ||
| cls, | ||
| path: str | Path, | ||
| table_name: str | None = None, | ||
| chunks: dict | None = None, | ||
| engine: str | None = "netcdf4", | ||
| **open_kwargs, | ||
| ): | ||
| """ | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We use Google style docstrings. |
||
| Open a NetCDF file and register it in an XarrayContext. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| path : str | Path | ||
| Path to the .nc file. | ||
| table_name : str | None | ||
| SQL table (or schema, for mixed-dimension datasets) name. | ||
| Defaults to the file stem (e.g. 'irma_gfs_example'). | ||
| chunks : dict | None | ||
| Dask chunks. Defaults to {'time': 24} if a 'time' dim exists, | ||
| otherwise opens all dims as a single chunk. | ||
| engine : str | None | ||
| xarray backend engine. Default is 'netcdf4'. | ||
| **open_kwargs | ||
| Additional kwargs forwarded to xr.open_dataset. | ||
|
|
||
| Returns | ||
| ------- | ||
| XarrayContext | ||
| A context with the dataset already registered. | ||
| """ | ||
| path = Path(path) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think open dataset handles this. |
||
|
|
||
| ds = xr.open_dataset(path, engine=engine, **open_kwargs) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The engine should be best-effort a netcdf engine IMO, and users won't need to pass it in, since the method name implies the type of data. |
||
|
|
||
| if not isinstance(ds, xr.Dataset): | ||
| raise TypeError(f"Expected xr.Dataset, got {type(ds).__name__}") | ||
|
|
||
| if table_name is None: | ||
| table_name = path.stem | ||
|
|
||
| if chunks is None: | ||
| chunks = {"time": 24} if "time" in ds.dims else {d: -1 for d in ds.dims} | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's be correct, not polite. The users must pass in chunks. |
||
|
|
||
| ctx = cls() | ||
| ctx.from_dataset(table_name, ds, chunks=chunks) | ||
| return ctx | ||
|
|
||
| def sql(self, query: str, *args, **kwargs) -> XarrayDataFrame: | ||
| """Run a SQL query, returning an :class:`XarrayDataFrame` wrapper. | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use pytest conventions for tests.