-
Notifications
You must be signed in to change notification settings - Fork 1
Feature full rules #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
59 commits
Select commit
Hold shift + click to select a range
7babf71
Initial setup of rules: download data, cutout, process slope
sjpfenninger 49ad79c
Use 60m GEDTM30 for slope via COG wrapper
sjpfenninger 75d91b0
Move bathymetry download to COG; cleanup file naming
sjpfenninger b5a36c7
Use 30 arcsec GHSL
sjpfenninger 4597abf
First attempt rules for onshore
LinhHo 335ba06
Update calling script for snakemake
LinhHo 3d614cd
Resample with dictionary from config
LinhHo 21eec60
Cannot save netcdf AtttributeError
LinhHo b8671a9
Coordinates error?
LinhHo 84321a2
Update messages in rules
LinhHo 5dea4f2
Fix mixing rasterio and xarray dataset
LinhHo deace67
Get full workflow to run
sjpfenninger cc93098
Cleanup and merge scripts
sjpfenninger a2fa551
Add wind offshore
LinhHo 08e39f1
Add wind offshore
LinhHo 2c7acc7
Remove land sea mask
LinhHo bee7ac5
Add reports
sjpfenninger 25b86d2
Improve file naming consistency
sjpfenninger 9e1e131
Move more of the geo-processing into resample.py; Add summary report
sjpfenninger a95ae05
Further clean up processing steps; Report by shape
sjpfenninger 55808e1
Process bathymetry at the start; minor cleanup
sjpfenninger 8440e6e
Make shape a wildcard
sjpfenninger 98e959f
Make UTM-based buffering configurable
sjpfenninger 9b39da0
Simplify and generalise the area potential processing; Cleanup
sjpfenninger b88df0d
User must manually download WDPA
sjpfenninger c061cc5
Add integration test
sjpfenninger f061c68
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 684fa8b
Emulate `unzip` in Python
sjpfenninger aa58edf
Attempt to fix Windows tests
sjpfenninger 966e463
Add docstrings and logfiles
sjpfenninger 42c4e11
Fix savefig call
sjpfenninger 8e81fd9
Add INTERFACE.yaml
sjpfenninger 47136ee
Template updates, README cleanup, add pixi.lock
sjpfenninger 287004f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 6e91b28
Add missing pixi task update
sjpfenninger ea72ceb
Update integration test Snakefile
sjpfenninger 50b6dce
Separate download from unzip in integration test, scale down to 1 core
sjpfenninger 834ff15
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 4cb8bf6
One core only to avoid parallelism on Windows
sjpfenninger 48c520f
Improve unzip_like and remove bash-based tempdirs
sjpfenninger 4ca80ae
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] b031d79
Move shell-based raster clipping to Python script
sjpfenninger 862611c
Remove 'set -x'
sjpfenninger d36bbd0
Land cover types configurable, memory improvements
sjpfenninger 8879741
Update test config; increase figure resolution
sjpfenninger 5d04201
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] d18037d
Write CRS to TIFFs before saving
sjpfenninger 5718c8a
Allow workflow to break larger shapes into subunits; other improvements
sjpfenninger 42c3285
Improve memory use in area_potential_report rule
sjpfenninger 36b6091
Replace gdal_merge with gdalwarp, clean up report
sjpfenninger f7fca2a
Run `copier recopy`
sjpfenninger 899642e
Changes in response to review
sjpfenninger ebcca38
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] 51010fa
Make tiny_files optional
sjpfenninger 5009d12
Zero out rather than NaN areas with no potential
sjpfenninger d7e054a
Add basic documentation
sjpfenninger 8fa413f
Add schema validation for shapes
sjpfenninger 5fc043c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] d4c59be
Improve schema
sjpfenninger File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,9 @@ | |
| __pycache__ | ||
| *.pyc | ||
|
|
||
| ### Environments | ||
| .pixi/ | ||
|
|
||
| ### Snakemake | ||
| .snakemake/ | ||
| gurobi.log | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,20 +1,16 @@ | ||
| # Module Input-Output structure for automated doc. generation | ||
| resources: | ||
| user: | ||
| user_message.md: "Example file emulating external files requested from a user." | ||
| automatic: | ||
| dummy_readme.md: "Example file emulating downloads determined by internal settings." | ||
| "shapes/{shape}.parquet": "Region geometries in parquet format. These should conform to the schema defined in https://github.com/calliope-project/module_geo_boundaries/blob/main/workflow/internal/shape.schema.yaml" | ||
| "wdpa.gdb": "WDPA protected areas database from https://www.protectedplanet.net/ in GeoDB format (choose 'File Geodatabase' when downloading)." | ||
| results: | ||
| combined_text.md: "Example file emulating module results." | ||
|
|
||
| # Wildcard example: | ||
| # resources: | ||
| # user: | ||
| # shapes_{resolution}.geojson: region geometries. | ||
| # automatic: | ||
| # technology_data.parquet: dataset with technology characteristics. | ||
| # results: | ||
| # '{resolution}/tech_capacity.parquet': description of output data. | ||
| # '{resolution}/results_image.png': description of output image. | ||
| # wildcards: | ||
| # resolution: description of the wildcard’s purpose. | ||
| "{shape}/{subunit}/area_potential_{tech}.tif": "Area potential GeoTIFF raster for the specified technology and subunit." | ||
| "{shape}/{subunit}/area_potential_{tech}.png": "Area potential GeoTIFF raster for the specified technology and subunit." | ||
| "{shape}/area_potential_{tech}.tif": "Area potential GeoTIFF raster for the specified technology across all subunits." | ||
| "{shape}/area_potential_{tech}.png": "Area potential PNG for the specified technology across all subunits." | ||
| "{shape}/area_potential_report.csv": "CSV summary report of area potential for all techs, across all subunits." | ||
| "{shape}/area_potential_report.html": "HTML summary report of area potential for all techs, across all subunits." | ||
| wildcards: | ||
| shape: "Name of the shape to be processed, e.g., 'world', 'europe', 'MEX'." | ||
| subunit: "Name of a subunit from within the shape, e.g 'IRL' for Ireland within 'europe'." | ||
| tech: "Name of the technology, e.g., 'pv_rooftop' or 'wind_offshore'. Available technologies are defined in the module configuration." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,94 @@ | ||
| # A minimal example of how to configure this module | ||
| dummy_text: This is a user input. | ||
| # Options for buffering: either a string of the form "epsg:xxxx" or "UTM"/"utm" | ||
| # - "UTM": project each shape to the UTM zone of its centroid for buffering | ||
| # - "epsg:xxxx": use the specified CRS for all buffering | ||
| # A good option is "epsg:8857" (WGS 84 / Equal Earth Greenwich) for global coverage | ||
| buffer_crs: "epsg:8857" | ||
|
|
||
| split_by: country_id # likely country_id or shape_id | ||
|
|
||
| # Optional - specify land_cover_types here to override the internal defaults. | ||
|
|
||
| techs: | ||
| pv_rooftop: | ||
| initial_area: settlement_area | ||
| continuous_layers: | ||
| settlement_share: | ||
| min: 0.01 | ||
| max: 1 | ||
| share: 0.8 | ||
| binary_layers: | ||
| regions_maritime: 0 | ||
| regions_land: 1 | ||
| protected: 0 | ||
| # Include all land cover types; selection is done by settlement_share | ||
| landcover_FARM: 1 | ||
| landcover_FOREST: 1 | ||
| landcover_URBAN: 1 | ||
| landcover_OTHER: 1 | ||
| landcover_NOT_SUITABLE: 0 | ||
| landcover_WATER: 0 | ||
| pv_open_field: | ||
| initial_area: pixel_area | ||
| continuous_layers: | ||
| slope: | ||
| min: 0 | ||
| max: 3 | ||
| settlement_share: | ||
| min: 0 | ||
| max: 0.01 | ||
| binary_layers: | ||
| regions_maritime: 0 | ||
| regions_land: 1 | ||
| protected: 0 | ||
| landcover_FARM: 0.1 | ||
| landcover_FOREST: 0 | ||
| landcover_URBAN: 0 | ||
| landcover_OTHER: 0.2 | ||
| landcover_NOT_SUITABLE: 0 | ||
| landcover_WATER: 0 | ||
| wind_onshore: | ||
| initial_area: pixel_area | ||
| continuous_layers: | ||
| slope: | ||
| min: 0 | ||
| max: 20 | ||
| settlement_share: | ||
| min: 0 | ||
| max: 0.01 | ||
| binary_layers: | ||
| regions_maritime: 0 | ||
| regions_land: 1 | ||
| protected: 0 | ||
| landcover_FARM: 0.2 | ||
| landcover_FOREST: 0.05 | ||
| landcover_URBAN: 0 | ||
| landcover_OTHER: 0.3 | ||
| landcover_NOT_SUITABLE: 0 | ||
| landcover_WATER: 0 | ||
| wind_offshore: | ||
| initial_area: pixel_area | ||
| continuous_layers: | ||
| bathymetry: | ||
| min: -50 | ||
| max: 0 | ||
| share: 0.8 | ||
| binary_layers: | ||
| regions_land: 0 | ||
| regions_maritime: 1 | ||
| protected: 0 | ||
| shapes_buffer: | ||
| land: 10000 # meters | ||
|
|
||
| # Optional: override settings for specific subunits (countries, regions, etc.) | ||
| # This allows you to set specific parameters that differ from the defaults, | ||
| # or apply settings that are not defined in the defaults (defaults = `techs` section). | ||
| # The subunit keys should match the subunit IDs used in column selected in `split_by`. | ||
| overrides: | ||
| PRT: # Inside a subunit, any setting from `techs` can be overridden | ||
| wind_offshore: | ||
| shapes_buffer: | ||
| land: 2000 # meters | ||
| NLD: | ||
| wind_offshore: | ||
| shapes_buffer: | ||
| land: 22000 # 22 km = 12 nautical miles | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,82 @@ | ||
| # Home | ||
| # Area potentials | ||
|
|
||
| Welcome to the documentation of the `module_area_potentials` data module! | ||
| Please consult the [specification guidelines](./specification.md) and the [`clio` documentation](https://clio.readthedocs.io/) for more information. | ||
| This module performs geospatial analyses to determine the available land area for specific technologies, on a by-pixel basis and aggregated to given geographic boundaries (e.g. generated by [module_geo_boundaries](https://github.com/calliope-project/module_geo_boundaries). | ||
|
|
||
| ## Overview | ||
|
|
||
| The analysis in this module is structured as follows: | ||
|
|
||
| * Geospatial input data (vector and raster) are acquired. This is automatic for most data, but the following data need to be manually supplied: | ||
| * Geographic boundaries in the parquet format | ||
| * WDPA protected areas database from https://www.protectedplanet.net/ in GeoDB format (choose 'File Geodatabase' when downloading). | ||
| * For the extent of the provided boundaries, the input data are reprojected and rasterised to the resolution of the land cover data (GlobCover), and merged into a single dataset for further processing. | ||
| * Based on the supplied configuration, the land-use analysis is done for each defined technology. | ||
| * Results can be reported on a per-geography and per-technology basis, as pixel surface area values (TIFF files), images for reporting purposes (PNG files), and also as a summary report with per-region capacities (CSV and HTML files) | ||
|
|
||
| See below for the [data sources](#data-sources). | ||
|
|
||
| ## Configuration | ||
|
|
||
| Configure the analysis in the `config.yaml` file. Examples are visible in `config/config.yaml` and `tests/integration/test_config.yaml`. | ||
|
|
||
| In the configuration, you can define any number of `techs`, and for each of them, specify the `initial_area`, `continuous_layers`, and `binary_layers`. | ||
|
|
||
| By example, here is a `pv_rooftop` tech. We use the `settlement_area`, which is the settlement area in m² in each pixel, as the initial area from which the further analysis proceeds. In `continuous_layers`, we use the `settlement_share`, which is the share (0-1) of area covered by settlement, and exclude pixels with less than 0.01 settlement share while assuming that of those pixels not excluded by that, 0.8 (80%) of the settled area can be used for rooftop PV. Finally, in the `binary_layers`, we include all land use types except `NOT_SUITABLE` (since the main selection is done via the settlement_share). This means that, for example, `FOREST` pixels with a `settlement_area` > 0 can be included. | ||
|
|
||
| ```yaml | ||
| pv_rooftop: | ||
| initial_area: settlement_area | ||
| continuous_layers: | ||
| settlement_share: | ||
| min: 0.01 | ||
| max: 1 | ||
| share: 0.8 | ||
| binary_layers: | ||
| regions_maritime: 0 | ||
| regions_land: 1 | ||
| protected: 0 | ||
| landcover_FARM: 1 | ||
| landcover_FOREST: 1 | ||
| landcover_URBAN: 1 | ||
| landcover_OTHER: 1 | ||
| landcover_NOT_SUITABLE: 0 | ||
| landcover_WATER: 0 | ||
| ``` | ||
|
|
||
| Here is a `wind_offshore` example. We start with the `pixel_area`, the total surface area in m² for each pixel. We include pixels with a slope up to and including 20 degrees, and exclude pixels with a settlement share above 0.01. Furthermore, we include only land areas (`regions_land: 1` and `regions_maritime: 0`) and completely exclude some areas like protected areas or urban areas (`protected: 0`, `landcover_URBAN: 0`), while including only a fraction of other areas (e.g. if a pixel is considered farmland, only 20% of its surface is available: `landcover_FARM: 0.2`). | ||
|
|
||
| ```yaml | ||
| wind_onshore: | ||
| initial_area: pixel_area | ||
| continuous_layers: | ||
| slope: | ||
| min: 0 | ||
| max: 20 | ||
| settlement_share: | ||
| min: 0 | ||
| max: 0.01 | ||
| binary_layers: | ||
| regions_maritime: 0 | ||
| regions_land: 1 | ||
| protected: 0 | ||
| landcover_FARM: 0.2 | ||
| landcover_FOREST: 0.05 | ||
| landcover_URBAN: 0 | ||
| landcover_OTHER: 0.3 | ||
| landcover_NOT_SUITABLE: 0 | ||
| landcover_WATER: 0 | ||
|
|
||
| ``` | ||
|
|
||
| ## Data sources | ||
|
|
||
| * [GEDTM30](https://github.com/openlandmap/GEDTM30) for slope | ||
| * License: Creative Commons Attribution 4.0 International | ||
| * [GlobCover land cover data](https://due.esrin.esa.int/page_globcover.php) | ||
| * License: "You may use the GlobCover land cover map for educational and/or scientific purposes, without any fee on the condition that you credit ESA and the Université Catholique de Louvain as the source of the GlobCover products." | ||
| * [GEBCO (General Bathymetric Chart of the Oceans)](https://www.gebco.net/data-products/gridded-bathymetry-data) 15 arc-second data | ||
| * License: "The GEBCO Grid is placed in the public domain and may be used free of charge. [...] Users must: Acknowledge the source of The GEBCO Grid. A suitable form of attribution is given in the documentation that accompanies The GEBCO Grid." | ||
| * [GHSL (Global Human Settlement Layer)](https://human-settlement.emergency.copernicus.eu/download.php) built-up surface data (R2023, GHS-BUILT-S, 100m resolution) | ||
| * License: "The GHSL has been produced by the EC JRC as open and free data. Reuse is authorised, provided the source is acknowledged." | ||
| * [WDPA (World Database on Protected Areas)](https://www.protectedplanet.net/) | ||
| * License: Non-commercial allowed. Citation: "UNEP-WCMC and IUCN (2025), Protected Planet: The World Database on Protected Areas (WDPA) and World Database on Other Effective Area-based Conservation Measures (WD-OECM) [Online], June 2025, Cambridge, UK: UNEP-WCMC and IUCN. Available at: www.protectedplanet.net." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,3 @@ | ||
| [mypy] | ||
| disable_error_code = import-untyped | ||
| exclude = (^|/)\.(snakemake|pixi)(/|$) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.