-
Notifications
You must be signed in to change notification settings - Fork 10
Update episode 9 "CMORization" for v2.14.0 #420
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
LisaBock
wants to merge
2
commits into
main
Choose a base branch
from
update_epsiode_09
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+34
−30
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,7 +2,7 @@ | |
| title: "CMORization: adding new datasets to ESMValTool" | ||
| teaching: 15 | ||
| exercises: 45 | ||
| compatibility: ESMValTool v2.13.0 | ||
| compatibility: ESMValTool v2.14.0 | ||
|
|
||
| questions: | ||
| - "CMORization: what is it and why do we need it?" | ||
|
|
@@ -68,8 +68,8 @@ that is important for calculating components of the global carbon cycle. | |
| See the next section on how to obtain data. | ||
|
|
||
| As in the previous episode | ||
| ([Development and Contribution episode]({{ page.root }}{% link | ||
| _episodes/07-development-setup.md %})), | ||
| ["Development and Contribution"]({{ page.root }}{% link | ||
| _episodes/07-development-setup.md %}), | ||
| we will be using the development installation of ESMValTool. | ||
|
|
||
|
|
||
|
|
@@ -123,28 +123,17 @@ run the existing one. There is a specific command available in the ESMValTool to | |
| run the CMORizer scripts: | ||
|
|
||
| ```bash | ||
| esmvaltool data format --config_dir <directory of config-user.yml> <dataset-name> | ||
| esmvaltool data format --original-data-dir </path/to/original/data> <dataset-name> | ||
| ``` | ||
|
|
||
| The options `--start` and `--end` can be added to command above to restrict the | ||
| This will look for the original data in the `original-data-dir` folder and store the | ||
| formatted data in the the output_dir path given in the configuration file. If | ||
| `original-data-dir` is not specified ESMValTool looks for the data in the current directory. The | ||
| options `--start` and `--end` can be added to command above to restrict the | ||
| formatting of raw data to a time range. They will be ignored if a specific | ||
| dataset does not support this option (i.e. because all the data is provided as a single file). | ||
| Valid formats are `YYYY`, `YYYYMM`, `YYYYMMDD`. The same way is also applicable for | ||
| the option `esmvaltool data download`. | ||
|
|
||
| The ``config-user.yml`` is the file in which we define the different data | ||
| paths, see the episode on [Configuration]({{ page.root }}{% link _episodes/03-configuration.md %}). | ||
| In the ``rootpath`` of your ``config-user.yml``, make sure to add the right | ||
| directory for "RAWOBS" data in which you downloaded the FLUXCOM dataset: | ||
|
|
||
| ```yaml | ||
| rootpath: | ||
| RAWOBS: ~/data/RAWOBS | ||
| ``` | ||
|
|
||
| This enables ESMValTool to find the raw observational datasets stored in the | ||
| "RAWOBS" folder. The ``dataset-name`` needs to be identical to the folder | ||
| name that was created to store the raw observation data files, i.e. | ||
| Valid formats are `YYYY`, `YYYYMM`, `YYYYMMDD`. The ``dataset-name`` needs to be identical to the | ||
| folder name that was created to store the raw observation data files, i.e. | ||
| ``RAWOBS/TierX/dataset-name``. In our case this would be "FLUXCOM". | ||
|
|
||
| If everything is okay, the output should look something like this: | ||
|
|
@@ -207,8 +196,20 @@ called `~/data/OBS/Tier3/FLUXCOM` and make sure the path to ``OBS`` is set | |
| correctly in our config-user file: | ||
|
|
||
| ```yaml | ||
| rootpath: | ||
| OBS: ~/data/OBS | ||
| OBS6: | ||
| data: | ||
| dkrz: | ||
| type: esmvalcore.io.local.LocalDataSource | ||
| rootpath: ~/data/OBS | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this not be "rootpath: ~/data/OBS6"? |
||
| dirname_template: "Tier{tier}/{dataset}" | ||
| filename_template: "{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc" | ||
| OBS: | ||
| data: | ||
| dkrz: | ||
| type: esmvalcore.io.local.LocalDataSource | ||
| rootpath: ~/data/OBS | ||
| dirname_template: "Tier{tier}/{dataset}" | ||
| filename_template: "{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc" | ||
| ``` | ||
|
|
||
| You can also see the path where ESMValTool stores the reformatting script: | ||
|
|
@@ -290,8 +291,7 @@ CMORized, ESMValTool will give a warning or error. | |
| Try to run the example recipe with | ||
|
|
||
| ```bash | ||
| esmvaltool run recipe_check_fluxcom.yml --config_dir <directory of config-user.yml> | ||
| --log_level debug | ||
| esmvaltool run recipe_check_fluxcom.yml --log_level debug | ||
| ``` | ||
|
|
||
| If everything is okay, the recipe should run without problems. | ||
|
|
@@ -414,7 +414,7 @@ You can try running the CMORizer at this point, and it should work without | |
| errors. However, it doesn't produce any output yet: | ||
|
|
||
| ```bash | ||
| esmvaltool data format --config_dir <directory of config-user.yml> FLUXCOM | ||
| esmvaltool data format --original-data-dir </path/to/original/data> FLUXCOM | ||
| ``` | ||
|
|
||
| ### 1. Find the input data | ||
|
|
@@ -637,15 +637,19 @@ upgrading the CMORizer to newer standards here!). Make sure the path to ``OBS6`` | |
| is set correctly in our config-user file: | ||
|
|
||
| ```yaml | ||
| rootpath: | ||
| OBS6: ~/data/OBS6 | ||
| OBS6: | ||
| data: | ||
| dkrz: | ||
| type: esmvalcore.io.local.LocalDataSource | ||
| rootpath: ~/data/OBS6/ | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this need the "/" at the end of the line? It is not given earlier when the rootpath is specified... |
||
| dirname_template: "Tier{tier}/{dataset}" | ||
| filename_template: "{project}_{dataset}_{type}_{version}_{mip}_{short_name}[_.]*nc" | ||
| ``` | ||
|
|
||
| If we now run the test recipe on our newly 'CMORized' data, | ||
|
|
||
| ```bash | ||
| esmvaltool run recipe_check_fluxcom.yml --config_dir <directory of config-user.yml> | ||
| --log_level debug | ||
| esmvaltool run recipe_check_fluxcom.yml --log_level debug | ||
| ``` | ||
|
|
||
| it should be able to find the correct file, but it does not succeed yet. The ESMValTool CMOR checker | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there is a "the" missing after "added to..."