Regrid ancillary variables - #3205
Conversation
valeriupredoi
left a comment
valeriupredoi
left a comment
There was a problem hiding this comment.
looks good, but I has a quick question pls 🍺
| # regridded data variable. | ||
| ancillary_result.data = ancillary_result.lazy_data().rechunk( | ||
| result[ancillary_slice].lazy_data().chunks, | ||
| ) |
There was a problem hiding this comment.
thanks @bouweandela - I am a bit puzzled as to why all the rechunking - why not perform a single rechunk of the ancillary_result at the end, after regridding? What are you going to do if the cube data is not chunked (one single slab)?
There was a problem hiding this comment.
The initial rechunk is needed to avoid getting huge chunks out of regridding when regridding to a higher resolution. The final rechunk is to align the chunks inside the cube, this gives better performance when combining one of the ancillary variables with the main variable in some later preprocessing step (e.g. area statistics).
There was a problem hiding this comment.
yes, that much I understood myself - but the question is why is the first rechunk needed since I thought the ancil data is chunked like the main var data? They have different chunking? That's insane if they do - they are both produced by the same model with the same data specs
There was a problem hiding this comment.
side note: huge chunks will not be a problem for CMIP7 anymore since the data will pass through cmip7_repack, so we'll have to be a bit more careful then
There was a problem hiding this comment.
BTW I saw your comment from the other PR #3204 - don't hold the merge of this hanging on my rechunking comments, we'll have to rethink that for CMIP7 light anyway - I am just being a bit anal 😁
There was a problem hiding this comment.
I thought the ancil data is chunked like the main var data? They have different chunking?
There is nothing in iris that enforces that.
huge chunks will not be a problem for CMIP7 anymore
Regridding is done on a per chunk basis: one chunk is regridded to one other chunk. Even if the input chunks are reasonably sized, the output chunks may be too large when the resolution is increased by regridding. I believe this has been addressed in iris in SciTools/iris#6730, but I do not see the corresponding changes in iris-esmf-regrid, so it's probably best to keep our own regridding code. This applies to CMIP7 just the same.
There was a problem hiding this comment.
that's mental! Thanks, bud - not auto scaling chunks upon regridding is bad, not having chunks harmonized between main var and ancil var, equally. OK nevermind my comments above 😁
2 participants
Description
Regrid ancillary variables
Closes part of #3138
Link to documentation:
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
To help with the number pull requests: