Error when downloading large slope file

[jnnr]

Describe the bug
Running the workflow for whole Europe runs into an error when downloading slope. We suspect that this will be fixed by pointing to the new version of the wrapper here, as soon as it is released.

To Reproduce
snakemake results/module_area_potentials/results/NUTS0/area_potential_pv_rooftop.tif

Expected behavior
Slope data should be download without any trouble.

Screenshots

Job 7: Download slope data covering the bounds of the input shapefile.                                                                                                                                                            
Reason: Missing output files: results/module_area_potentials/resources/automatic/cutout/NUTS0/slope.tif; Input files updated by another job: results/module_area_potentials/resources/user/shapes/NUTS0.parquet                   
                                                                                                                                                                                                                                  
Activating conda environment: .snakemake/conda/530cc284f69c59c16a5f8deb4e32965f_                                                                                                                                                  
Activating conda environment: .snakemake/conda/530cc284f69c59c16a5f8deb4e32965f_                                                                                                                                                  
rasterio._err.CPLE_AppDefinedError: _TIFFPartialReadStripArray:Cannot read offset/size for strile around ~304                                                                                                                     
                                                                                                                                                                                                                                  
The above exception was the direct cause of the following exception:                                                                                                                                                              
                                                                                                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                
  File "rasterio/_io.pyx", line 969, in rasterio._io.DatasetReaderBase._read                                                                                                                                                      
  File "rasterio/_io.pyx", line 199, in rasterio._io.io_multi_band                                                                                                                                                                
  File "rasterio/_io.pyx", line 205, in rasterio._io.io_multi_band                                                                                                                                                                
  File "rasterio/_err.pyx", line 325, in rasterio._err.StackChecker.exc_wrap_int                                                                                                                                                  
rasterio._err.CPLE_AppDefinedError: slope.in.degree_edtm_m_240m_s_20000101_20221231_go_epsg.4326_v20241230.tif, band 1: IReadBlock failed at X offset 40, Y offset 3: _TIFFPartialReadStripArray:Cannot read offset/size for stril
e around ~304                                                                                                                                                                                                                     
                                                                                                                                                                                                                                  
The above exception was the direct cause of the following exception:                                                                                                                                                              
                                                                                                                                                                                                                                  
Traceback (most recent call last):                                                                                                                                                                                                
  File "/data/user/jlauner/wind-value-map/.snakemake/scripts/tmpy40_zbsk.wrapper.py", line 100, in <module>                                                                                                                       
    data = src.read(window=window, boundless=True)                                                                                                                                                                                
  File "rasterio/_io.pyx", line 687, in rasterio._io.DatasetReaderBase.read                                                                                                                                                       
  File "rasterio/_io.pyx", line 688, in rasterio._io.DatasetReaderBase.read                                                                                                                                                       
  File "rasterio/_io.pyx", line 972, in rasterio._io.DatasetReaderBase._read                                                                                                                                                      
rasterio.errors.RasterioIOError: Read failed. See previous exception for details.       

Desktop (please complete the following information):

• OS: Linux
• Version: head of branch feature-full-rules

[sjpfenninger]

This is now resolved in practice, but leaving the issue open as a reminder to point to the newest wrapper version once released. For now, by default full TIF files are downloaded, and the on-demand download which triggers the above issue can be optionally configured.

[irm-codebase]

The general issue is Zenodo's rate limits.

We are limited to max 60 requests per minute or 2000 per hour. This can happen when requesting many small files in tandem, or a large chunked file. Rasterio is hard-coded to a 512x512 read/write window, with each being a request.
https://rasterio.groups.io/g/main/topic/how_to_politely_clip_large/114209356

The new version of the rio-clip wrapper is already merged, but we need to wait for the next snakemake-wrappers release. It lets you modify the GDAL environment during the request to hopefully avoid this failure.

These settings were suggested by GDAL devs:

GDAL_ENV_OVERRIDE = {
    "GDAL_HTTP_MAX_RETRY": "10000",  # retry a lot
    "GDAL_HTTP_RETRY_DELAY": "60",  # if rate limit hit, wait 1 minute
    "GDAL_HTTP_RETRY_CODES": "429,500",  # retry if 429 (too many requests), 500 (internal server error)
}