Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
7babf71
Initial setup of rules: download data, cutout, process slope
sjpfenninger Jun 25, 2025
49ad79c
Use 60m GEDTM30 for slope via COG wrapper
sjpfenninger Jun 26, 2025
75d91b0
Move bathymetry download to COG; cleanup file naming
sjpfenninger Jun 26, 2025
b5a36c7
Use 30 arcsec GHSL
sjpfenninger Jun 26, 2025
4597abf
First attempt rules for onshore
LinhHo Jun 30, 2025
335ba06
Update calling script for snakemake
LinhHo Jun 30, 2025
3d614cd
Resample with dictionary from config
LinhHo Jun 30, 2025
21eec60
Cannot save netcdf AtttributeError
LinhHo Jun 30, 2025
b8671a9
Coordinates error?
LinhHo Jun 30, 2025
84321a2
Update messages in rules
LinhHo Jun 30, 2025
5dea4f2
Fix mixing rasterio and xarray dataset
LinhHo Jun 30, 2025
deace67
Get full workflow to run
sjpfenninger Jun 30, 2025
cc93098
Cleanup and merge scripts
sjpfenninger Jun 30, 2025
a2fa551
Add wind offshore
LinhHo Jul 2, 2025
08e39f1
Add wind offshore
LinhHo Jul 2, 2025
2c7acc7
Remove land sea mask
LinhHo Jul 2, 2025
bee7ac5
Add reports
sjpfenninger Jul 4, 2025
25b86d2
Improve file naming consistency
sjpfenninger Jul 4, 2025
9e1e131
Move more of the geo-processing into resample.py; Add summary report
sjpfenninger Jul 7, 2025
a95ae05
Further clean up processing steps; Report by shape
sjpfenninger Jul 7, 2025
55808e1
Process bathymetry at the start; minor cleanup
sjpfenninger Jul 8, 2025
8440e6e
Make shape a wildcard
sjpfenninger Jul 8, 2025
98e959f
Make UTM-based buffering configurable
sjpfenninger Jul 8, 2025
9b39da0
Simplify and generalise the area potential processing; Cleanup
sjpfenninger Jul 8, 2025
b88df0d
User must manually download WDPA
sjpfenninger Jul 8, 2025
c061cc5
Add integration test
sjpfenninger Jul 8, 2025
f061c68
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 8, 2025
684fa8b
Emulate `unzip` in Python
sjpfenninger Jul 8, 2025
aa58edf
Attempt to fix Windows tests
sjpfenninger Jul 8, 2025
966e463
Add docstrings and logfiles
sjpfenninger Jul 9, 2025
42c4e11
Fix savefig call
sjpfenninger Jul 9, 2025
8e81fd9
Add INTERFACE.yaml
sjpfenninger Jul 10, 2025
47136ee
Template updates, README cleanup, add pixi.lock
sjpfenninger Jul 11, 2025
287004f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 11, 2025
6e91b28
Add missing pixi task update
sjpfenninger Jul 11, 2025
ea72ceb
Update integration test Snakefile
sjpfenninger Jul 11, 2025
50b6dce
Separate download from unzip in integration test, scale down to 1 core
sjpfenninger Jul 11, 2025
834ff15
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 11, 2025
4cb8bf6
One core only to avoid parallelism on Windows
sjpfenninger Jul 11, 2025
48c520f
Improve unzip_like and remove bash-based tempdirs
sjpfenninger Jul 11, 2025
4ca80ae
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 11, 2025
b031d79
Move shell-based raster clipping to Python script
sjpfenninger Jul 11, 2025
862611c
Remove 'set -x'
sjpfenninger Jul 11, 2025
d36bbd0
Land cover types configurable, memory improvements
sjpfenninger Jul 15, 2025
8879741
Update test config; increase figure resolution
sjpfenninger Jul 15, 2025
5d04201
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 15, 2025
d18037d
Write CRS to TIFFs before saving
sjpfenninger Jul 15, 2025
5718c8a
Allow workflow to break larger shapes into subunits; other improvements
sjpfenninger Jul 22, 2025
42c3285
Improve memory use in area_potential_report rule
sjpfenninger Jul 23, 2025
36b6091
Replace gdal_merge with gdalwarp, clean up report
sjpfenninger Jul 25, 2025
f7fca2a
Run `copier recopy`
sjpfenninger Aug 1, 2025
899642e
Changes in response to review
sjpfenninger Oct 31, 2025
ebcca38
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 31, 2025
51010fa
Make tiny_files optional
sjpfenninger Oct 31, 2025
5009d12
Zero out rather than NaN areas with no potential
sjpfenninger Oct 31, 2025
d7e054a
Add basic documentation
sjpfenninger Nov 4, 2025
8fa413f
Add schema validation for shapes
sjpfenninger Nov 14, 2025
5fc043c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 14, 2025
d4c59be
Improve schema
sjpfenninger Nov 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@
__pycache__
*.pyc

### Environments
.pixi/

### Snakemake
.snakemake/
gurobi.log
Expand Down
1 change: 1 addition & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ This does not necessarily list everyone who has contributed to this software's
code or documentation. For a full contributor list, see:
<https://github.com/calliope-project/module_area_potentials/graphs/contributors>

Linh Ho-Tran, <L.Ho@tudelft.nl>
Stefan Pfenninger-Lee, <s.pfenninger@tudelft.nl>
2 changes: 2 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,7 @@ title: "clio - module_area_potentials: Area potentials"
repository: "https://github.com/calliope-project/module_area_potentials"
license: Apache-2.0
authors:
- given-names: Linh
family-names: Ho-Tran
- given-names: Stefan
family-names: Pfenninger-Lee
28 changes: 12 additions & 16 deletions INTERFACE.yaml
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
# Module Input-Output structure for automated doc. generation
resources:
user:
user_message.md: "Example file emulating external files requested from a user."
automatic:
dummy_readme.md: "Example file emulating downloads determined by internal settings."
"shapes/{shape}.parquet": "Region geometries in parquet format. These should conform to the schema defined in https://github.com/calliope-project/module_geo_boundaries/blob/main/workflow/internal/shape.schema.yaml"
"wdpa.gdb": "WDPA protected areas database from https://www.protectedplanet.net/ in GeoDB format (choose 'File Geodatabase' when downloading)."
results:
combined_text.md: "Example file emulating module results."

# Wildcard example:
# resources:
# user:
# shapes_{resolution}.geojson: region geometries.
# automatic:
# technology_data.parquet: dataset with technology characteristics.
# results:
# '{resolution}/tech_capacity.parquet': description of output data.
# '{resolution}/results_image.png': description of output image.
# wildcards:
# resolution: description of the wildcard’s purpose.
"{shape}/{subunit}/area_potential_{tech}.tif": "Area potential GeoTIFF raster for the specified technology and subunit."
"{shape}/{subunit}/area_potential_{tech}.png": "Area potential GeoTIFF raster for the specified technology and subunit."
"{shape}/area_potential_{tech}.tif": "Area potential GeoTIFF raster for the specified technology across all subunits."
"{shape}/area_potential_{tech}.png": "Area potential PNG for the specified technology across all subunits."
"{shape}/area_potential_report.csv": "CSV summary report of area potential for all techs, across all subunits."
"{shape}/area_potential_report.html": "HTML summary report of area potential for all techs, across all subunits."
wildcards:
shape: "Name of the shape to be processed, e.g., 'world', 'europe', 'MEX'."
subunit: "Name of a subunit from within the shape, e.g 'IRL' for Ireland within 'europe'."
tech: "Name of the technology, e.g., 'pv_rooftop' or 'wind_offshore'. Available technologies are defined in the module configuration."
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ pixi install --all
For testing, simply run:

```shell
pixi run test
pixi run test-integration
```

To view the documentation locally, use:
Expand All @@ -37,5 +37,5 @@ To test a minimal example of a workflow using this module:
```shell
pixi shell # activate this project's environment
cd tests/integration/ # navigate to the integration example
snakemake --use-conda # run the workflow!
snakemake --use-conda --cores 2 # run the workflow!
```
96 changes: 94 additions & 2 deletions config/config.yaml
Original file line number Diff line number Diff line change
@@ -1,2 +1,94 @@
# A minimal example of how to configure this module
dummy_text: This is a user input.
# Options for buffering: either a string of the form "epsg:xxxx" or "UTM"/"utm"
# - "UTM": project each shape to the UTM zone of its centroid for buffering
# - "epsg:xxxx": use the specified CRS for all buffering
# A good option is "epsg:8857" (WGS 84 / Equal Earth Greenwich) for global coverage
buffer_crs: "epsg:8857"

split_by: country_id # likely country_id or shape_id

# Optional - specify land_cover_types here to override the internal defaults.

techs:
pv_rooftop:
initial_area: settlement_area
continuous_layers:
settlement_share:
min: 0.01
max: 1
share: 0.8
binary_layers:
regions_maritime: 0
regions_land: 1
protected: 0
# Include all land cover types; selection is done by settlement_share
landcover_FARM: 1
landcover_FOREST: 1
landcover_URBAN: 1
landcover_OTHER: 1
landcover_NOT_SUITABLE: 0
landcover_WATER: 0
pv_open_field:
initial_area: pixel_area
continuous_layers:
slope:
min: 0
max: 3
settlement_share:
min: 0
max: 0.01
binary_layers:
regions_maritime: 0
regions_land: 1
protected: 0
landcover_FARM: 0.1
landcover_FOREST: 0
landcover_URBAN: 0
landcover_OTHER: 0.2
landcover_NOT_SUITABLE: 0
landcover_WATER: 0
wind_onshore:
initial_area: pixel_area
continuous_layers:
slope:
min: 0
max: 20
settlement_share:
min: 0
max: 0.01
binary_layers:
regions_maritime: 0
regions_land: 1
protected: 0
landcover_FARM: 0.2
landcover_FOREST: 0.05
landcover_URBAN: 0
landcover_OTHER: 0.3
landcover_NOT_SUITABLE: 0
landcover_WATER: 0
wind_offshore:
initial_area: pixel_area
continuous_layers:
bathymetry:
min: -50
max: 0
share: 0.8
binary_layers:
regions_land: 0
regions_maritime: 1
protected: 0
shapes_buffer:
land: 10000 # meters

# Optional: override settings for specific subunits (countries, regions, etc.)
# This allows you to set specific parameters that differ from the defaults,
# or apply settings that are not defined in the defaults (defaults = `techs` section).
# The subunit keys should match the subunit IDs used in column selected in `split_by`.
overrides:
PRT: # Inside a subunit, any setting from `techs` can be overridden
wind_offshore:
shapes_buffer:
land: 2000 # meters
NLD:
wind_offshore:
shapes_buffer:
land: 22000 # 22 km = 12 nautical miles
84 changes: 81 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,82 @@
# Home
# Area potentials

Welcome to the documentation of the `module_area_potentials` data module!
Please consult the [specification guidelines](./specification.md) and the [`clio` documentation](https://clio.readthedocs.io/) for more information.
This module performs geospatial analyses to determine the available land area for specific technologies, on a by-pixel basis and aggregated to given geographic boundaries (e.g. generated by [module_geo_boundaries](https://github.com/calliope-project/module_geo_boundaries).

## Overview

The analysis in this module is structured as follows:

* Geospatial input data (vector and raster) are acquired. This is automatic for most data, but the following data need to be manually supplied:
* Geographic boundaries in the parquet format
* WDPA protected areas database from https://www.protectedplanet.net/ in GeoDB format (choose 'File Geodatabase' when downloading).
* For the extent of the provided boundaries, the input data are reprojected and rasterised to the resolution of the land cover data (GlobCover), and merged into a single dataset for further processing.
* Based on the supplied configuration, the land-use analysis is done for each defined technology.
* Results can be reported on a per-geography and per-technology basis, as pixel surface area values (TIFF files), images for reporting purposes (PNG files), and also as a summary report with per-region capacities (CSV and HTML files)

See below for the [data sources](#data-sources).

## Configuration

Configure the analysis in the `config.yaml` file. Examples are visible in `config/config.yaml` and `tests/integration/test_config.yaml`.

In the configuration, you can define any number of `techs`, and for each of them, specify the `initial_area`, `continuous_layers`, and `binary_layers`.

By example, here is a `pv_rooftop` tech. We use the `settlement_area`, which is the settlement area in m² in each pixel, as the initial area from which the further analysis proceeds. In `continuous_layers`, we use the `settlement_share`, which is the share (0-1) of area covered by settlement, and exclude pixels with less than 0.01 settlement share while assuming that of those pixels not excluded by that, 0.8 (80%) of the settled area can be used for rooftop PV. Finally, in the `binary_layers`, we include all land use types except `NOT_SUITABLE` (since the main selection is done via the settlement_share). This means that, for example, `FOREST` pixels with a `settlement_area` > 0 can be included.

```yaml
pv_rooftop:
initial_area: settlement_area
continuous_layers:
settlement_share:
min: 0.01
max: 1
share: 0.8
binary_layers:
regions_maritime: 0
regions_land: 1
protected: 0
landcover_FARM: 1
landcover_FOREST: 1
landcover_URBAN: 1
landcover_OTHER: 1
landcover_NOT_SUITABLE: 0
landcover_WATER: 0
```

Here is a `wind_offshore` example. We start with the `pixel_area`, the total surface area in m² for each pixel. We include pixels with a slope up to and including 20 degrees, and exclude pixels with a settlement share above 0.01. Furthermore, we include only land areas (`regions_land: 1` and `regions_maritime: 0`) and completely exclude some areas like protected areas or urban areas (`protected: 0`, `landcover_URBAN: 0`), while including only a fraction of other areas (e.g. if a pixel is considered farmland, only 20% of its surface is available: `landcover_FARM: 0.2`).

```yaml
wind_onshore:
initial_area: pixel_area
continuous_layers:
slope:
min: 0
max: 20
settlement_share:
min: 0
max: 0.01
binary_layers:
regions_maritime: 0
regions_land: 1
protected: 0
landcover_FARM: 0.2
landcover_FOREST: 0.05
landcover_URBAN: 0
landcover_OTHER: 0.3
landcover_NOT_SUITABLE: 0
landcover_WATER: 0

```

## Data sources

* [GEDTM30](https://github.com/openlandmap/GEDTM30) for slope
* License: Creative Commons Attribution 4.0 International
* [GlobCover land cover data](https://due.esrin.esa.int/page_globcover.php)
* License: "You may use the GlobCover land cover map for educational and/or scientific purposes, without any fee on the condition that you credit ESA and the Université Catholique de Louvain as the source of the GlobCover products."
* [GEBCO (General Bathymetric Chart of the Oceans)](https://www.gebco.net/data-products/gridded-bathymetry-data) 15 arc-second data
* License: "The GEBCO Grid is placed in the public domain and may be used free of charge. [...] Users must: Acknowledge the source of The GEBCO Grid. A suitable form of attribution is given in the documentation that accompanies The GEBCO Grid."
* [GHSL (Global Human Settlement Layer)](https://human-settlement.emergency.copernicus.eu/download.php) built-up surface data (R2023, GHS-BUILT-S, 100m resolution)
* License: "The GHSL has been produced by the EC JRC as open and free data. Reuse is authorised, provided the source is acknowledged."
* [WDPA (World Database on Protected Areas)](https://www.protectedplanet.net/)
* License: Non-commercial allowed. Citation: "UNEP-WCMC and IUCN (2025), Protected Planet: The World Database on Protected Areas (WDPA) and World Database on Other Effective Area-based Conservation Measures (WD-OECM) [Online], June 2025, Cambridge, UK: UNEP-WCMC and IUCN. Available at: www.protectedplanet.net."
1 change: 1 addition & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
[mypy]
disable_error_code = import-untyped
exclude = (^|/)\.(snakemake|pixi)(/|$)
Loading