Fix MPI Divergent Grid Stretching

[danieljvickers]

Description

When we run grid stretching in MFC pre-processing, we first normalize the cell boundary array by the local length of the mpi rank. We then convert back later. This has the clear flaw that as we add more MPI ranks then the grid is stretched in different ways. For example, a case that runs fine in 32 ranks will suddenly break at 128 ranks because the grid was stretched more to infinity. This can also cause some frustration because the grid stretching has to be completely recomputed in order to properly strong-scale a case. When more ranks are added, it doesn't just double the resolution, it may just increase the domain size and leave the same resolution.

Both of the issues above make this clearly undesired/unintended behavior. To fix this, I added a global domain boundary in pre-processing that is used to normalize the array, instead of the local domain length. This eliminates the processor-rank-number-dependent domain stretching and makes it a little easier to set up proper grid-stretching cases.

Type of change (delete unused ones)

• Bug fix

Testing

Ran on 1 and 1024 ranks, but obtained the same grid size.

Checklist

Check these like this [x] to indicate which of the below applies.

• I added or updated tests for new behavior

See the developer guide for full coding standards.

GPU changes (expand if you modified src/simulation/)

This is a CPU-only change.

[sbryngelson]

it baffles me that this was broken. surely we've done multirank grid stretching before, but the fix looks correct.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.43%. Comparing base (66254db) to head (6dd5701).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1625   +/-   ##
=======================================
  Coverage   60.43%   60.43%           
=======================================
  Files          83       83           
  Lines       19868    19871    +3     
  Branches     2956     2956           
=======================================
+ Hits        12007    12010    +3     
  Misses       5860     5860           
  Partials     2001     2001           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[danieljvickers]

it baffles me that this was broken. surely we've done multirank grid stretching before, but the fix looks correct.

I spent quite a while trying to work around it because I couldn't believe that it was broken either. But idk. Either way, I have been using this to do all of my analysis on Frontier, and it has made things a lot easier.

I think it may also be neat to add a grid-stretching helper to the toolchain or a smarter parameter grid stretching in the case files to make getting your parameters more obvious. Something to help people with parameter selection based on their desired domain size, uniform region, uniform spacing, etc.