Currently we read data using xr.open_mfdataset and create dataset using STDataset. We can specify chunks in xr.open_mfdataset when data is large, but STDataset doesnot respect it. .values and from_numpy calls in STDataset class, data is not lazy and the benefit of chunking is mostly lost. We want refactor STDataset so data stays lazy, see this comment.
Currently we read data using
xr.open_mfdatasetand create dataset using STDataset. We can specify chunks in xr.open_mfdataset when data is large, but STDataset doesnot respect it..valuesandfrom_numpycalls inSTDatasetclass, data is not lazy and the benefit of chunking is mostly lost. We want refactor STDataset so data stays lazy, see this comment.