Add alternative source for ERA5 data to eliminate queueing

[mrmorawski]

Closes #495

Changes proposed in this Pull Request

Currently the ERA5 dataset is downloaded from CDS. This requires authentication and users often get stuck in a processing queue for hours.

This PR adds support for downloading ERA5 from a mirror hosted at NCAR via OPeNDAP. This mirror requires no authentication and there is no queue.

The modification is implemented as a new dataset module called era5-ncar, which exposes the same API as the current era5.py module. All the user needs to do is change one line when creating the cutout. Code remains fully backwards-compatible.

cutout = atlite.Cutout(
    path="spain-2024.nc",
    module="era5-ncar",    # <-- only change from "era5"
    x=slice(-10, 5),
    y=slice(35, 44),
    time="2013-01-01
   )

Comparison of era5 vs era5-ncar

The NCAR mirror is stored as NetCDF files at the default 0.25/0.25 deg. resolution used by CDS. This has implications for performance and accuracy.

Performance

Tested on a Germany-sized cutout, 6 months of data, downloaded at the default grid size (CDS files served from CDS server cache, no queuing):

Source	Wall time	Peak memory usage	received files
NCAR THREDDS	239.3 s	865 MB	447.6 MB
CDS	194.2 s	802 MB	186.4 MB

As you can see, we need to download slightly more data from NCAR and do more local processing, but the extra time is typically offset by the CDS queue time (over an hour when this cutout was first requested). Coarser/larger cutouts will benefit CDS, while smaller cutouts will have an even larger speedup on NCAR.

Accuracy

Results on the default grid (and its multiples) are very close, within float32 precision. Results on other grids are slightly off, as we cannot reproduce the exact interpolation logic used by CDS. Details of the interpolation errors are explained in the example notebook examples/comparing_era5_sources_cds_vs_ncar.ipynb

When to use which source

I included the following line in the documentation:

So which source should I use? For very large cutouts (e.g. continents larger than Europe), for workflows that already use CDS, and when you require data at resolution that is not a multiple of the default 0.25/0.25 deg. grid, use CDS. For everything else, NCAR is more convenient.

Communication with UCAR

I asked UCAR, who administer this dataset for NCAR, whether they were OK with the OPeNDAP access. The response was positive. If we experience rate limiting on the server side, we may need to limit the number of simultaneous connections further (I capped it at 8).

LLM use disclosure

I used LLMs to:

• help me understand the THREDDS API
• write some of the processing code
• review the module
• write the benchmarking and plotting code

I manually wrote all the documentation and most of the docstrings. I hand-checked every line of code in this PR.

Remaining tests

I'd appreciate if someone could test the module on their machine on a large download. I've been test-running it for a week on my own setup, and it was stable (not a single failed download). However, there may still be hidden networking bugs.

Other ERA5 sources

I investigated other sources for ERA5. This is, to my knowledge, the only one fulfills the following criteria:

• ran by a large organisation
• easy spatial subsetting
• no authentication and queueing
• provides full set of the variables used by atlite

The ARCO datasets stored by Google, Microsoft and Amazon seem to be chunked along the time dimension only, so one always has to download a whole world's worth of data.

Earthmover stores era5 on s3 in Zarr, chunked in yearly, 12x12 spatial chunks, but they only do surface variables.

Checklist

• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable). I added dependencies to pyproject.toml, I don't see an environment.yaml
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the MIT license.

[fneum]

@mrmorawski, let me know when I should give this another review regarding the latest changes.

[mrmorawski]

Thanks for checking in @fneum ! I've been profiling and optimising memory usage, I think I mostly have it figured out. I will give you a heads up soon, most likely this week.

…

-------- Original Message --------

On Monday, 06/01/26 at 11:52 Fabian Neumann ***@***.***> wrote: fneum left a comment [(PyPSA/atlite#507)](#507 (comment)) ***@***.***(https://github.com/mrmorawski), let me know when I should give this another review regarding the latest changes. — Reply to this email directly, [view it on GitHub](#507?email_source=notifications&email_token=A5UF4ZWP5LS66KN3BA4I3ET45VG37A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJZGE2DIOBUHA22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4591448485), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/A5UF4ZVYPOWEG4I7O6DN5XD45VG37AVCNFSM6AAAAACYZ3YY7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKOJRGQ2DQNBYGU). Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for [iOS](https://github.com/notifications/mobile/ios/A5UF4ZQPI2UZQ3YG4E4FGDD45VG37A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJZGE2DIOBUHA22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y) and [Android](https://github.com/notifications/mobile/android/A5UF4ZWCQKL4YIZQMLRHD6L45VG37A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJZGE2DIOBUHA22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI). Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[mrmorawski]

Profiling memory usage

I did a lot of memory profiling for era5-edh. I fixed some minor issues (chunk sizes, operation order), which brought the memory down to 3.5GB for the 6-month EU cutout I was measuring above. Here is memory usage over time:

I think I have identified a key bottleneck - system malloc not releasing small chunks of memory to the OS, which accumulate in the process. It's described in more detail here , and seems to be Linux and MacOS specific. The distributed dask scheduler deals with this by launching workers with an environment variable MALLOC_TRIM_THRESHOLD_=65536, which causes malloc to release memory sooner and avoids accumulation. When I ran the same download as above with this env var, the peak memory usage dropped to 2.35GB.

Memory usage conclusions

• I think that for cutouts in the <50GB range memory usage is acceptable with the default settings.

• For cutouts above that, I think the users should use the distributed scheduler, as the default threaded one is not memory-aware and may OOM on smaller machines. I wrote an example on how to use the distributed scheduler here . It's two lines of code to give people granular control over memory usage, OOM prevention, nice graphs for monitoring progress etc. In the example notebook, I generate data for the whole world over a year while keeping memory below 12GB.

• Alternatively we could do the memory trimming in the code as done here but IMO we should avoid calling malloc directly from Python if we can help it.

Other issues (support for Python 3.10)

I also fixed issues flagged by the CI, now the tests pass (see here except on Python 3.10. Python 3.10 doesn't work because we need Zarr-3, and 3.10 only supports Zarr-2. I couldy hack around this, but I think we should either:

• require Python>=3.11, or
• make era5-edh conditional on having Python>=3.11

Would dropping 3.10 be OK with you @fneum ? It'll reach end-of-life in October 2026 anyway

[mrmorawski]

I updated the branch to pass the new ruff rules checks introduced in 9663808

And one thing that I forgot to mention above - this branch introduces a new secret to the test CI (EARTHDATAHUB_API_KEY), which is needed to download era5-edh cutouts for tests. You'd need to add one to the GitHub Actions config in this repo.

[fneum]

Yes, dropping Python 3.10 is fine (we did this in PyPSA already).

I'll add the earth data hub API key now.

[mrmorawski]

Awesome. I see there are still some issues with the CI tests (they pass, but ERA5 and ERA5EDH are being skipped in tests for this PR). I will debug those next week. Is there anything else you need to be able to merge this @fneum?

[maurerle]

did take a look on this as well (just code though, did not try it). Sounds like a good alternative! :)

The history of this PR should be squashed or cleaned up though.

[maurerle]

I don't have anything against dropping 3.10 support, but I guess that 3.10 would work without this change?

Either remove typing_extensions from pyproject.toml as well, or leave this change as before (and remove python 3.10 in a separate PR)

[mrmorawski]

this PR needs Python 3.11+ to work because era5-edh relies on Zarr v3's ObjectStore, which is not available for Python 3.10 AFAIK. I could create a separate PR for 3.10->3.11 if you'd like that, but for now I just removed typing_extensions from dependencies. Does this resolve your comment?

[mrmorawski]

thanks for the review @maurerle !

I checked out the CI, and the reason that EDH tests were being skipped is that CI on PRs doesn't have access to secrets. CDS tests are not skipped, but only on Ubuntu, because there are the nightly built cached cutouts for those.

Regarding commit history, I totally agree it should be cleaned up, but I thought you'd just squash merge it @maurerle @fneum?

If you'd prefer that I squash this branch, I'd be happy to do that of course

[fneum]

We just squash at the end.

[mrmorawski]

Then I think it's ready for merge - CI runs all tests and passes on my fork. The last remaining issue is the Python version bump - would you prefer a separate PR for that, or can we do it here?

[maurerle]

I think it is fine here.
I do not have commit rights, but I'd approve.