Reduce the number of PIO sync during IOStream write by grnydawn · Pull Request #463 · E3SM-Project/Omega

grnydawn · 2026-07-07T19:47:58Z

This PR reduces the number of PIO sync operations performed during IOStream writes for field data.

Instead of executing a PIO sync for each field variable, it synchronizes the physical file only when the file is closed.

All CTest tests pass on pm-gpu, pm-cpu, and chrysalis.

The IOSTREAM_TEST on pm-cpu is approximately 30% faster than the original implementation.

This PR provides a partial fix for this issue. Additional performance optimizations may be achieved by implementing caching and reuse of decompositions.

Checklist

vanroekel · 2026-07-08T14:55:04Z

@grnydawn This seems to be working for my column testing. It's getting far more than 30% reduction. Here is the original timing from the PIO section

*   "Omega:PIO:PIOc_read_darray"                                 -      46    -       0.501140     0.012541     0.009970         0.000003
      "Omega:PIO:read_darray_nc"                                 -      46    -       0.120856     0.002865     0.002319         0.000003
      "Omega:PIO:rearrange_io2comp"                              -      46    -       0.000350     0.000027     0.000004         0.000003
  "Omega:RunLoop"                                                -       1    -      55.019615    55.019615    55.019615         0.000000
*   "Omega:PIO:PIOc_initdecomp"                                  -    5808    -       0.669450     0.000581     0.000006         0.000412
      "Omega:PIO:box_rearrange_create"                           -    5808    -       0.641387     0.000572     0.000004         0.000412
*       "Omega:PIO:pio_swapm"                                    -   23236    -       0.042159     0.000027     0.000000         0.001648
*   "Omega:PIO:PIOc_freedecomp"                                  -    5808    -       0.011113     0.000026     0.000001         0.000412
*   "Omega:PIO:PIOc_sync"                                        -    5916    -      17.350798     0.006201     0.000000         0.000420
*     "Omega:PIO:write_total"                                    -   12934    -      52.854229     0.007719     0.000002         0.000917
        "Omega:PIO:flush_buffer"                                 -   11532    -       0.180795     0.000071     0.000000         0.000818
          "Omega:PIO:PIOc_write_darray_multi"                    -    5766    -       0.159620     0.000066     0.000013         0.000409
            "Omega:PIO:rearrange_comp2io"                        -    5766    -       0.100254     0.000041     0.000005         0.000409
            "Omega:PIO:write_darray_multi_par"                   -    5766    -       0.026767     0.000029     0.000002         0.000409
*       "Omega:PIO:flush_output_buffer"                          -   11967    -      17.145353     0.006151     0.000000         0.000849
*   "Omega:PIO:PIOc_closefile"                                   -     150    -       0.074891     0.002532     0.000028         0.000011
      "Omega:PIO:PIOc_closefile_write_mode"                      -     145    -       0.065644     0.002459     0.000333         0.000010
*   "Omega:PIO:spio_get_att_tc"                                  -     150    -       0.000724     0.000011     0.000002         0.000011
*   "Omega:PIO:spio_get_vars_tc"                                 -   10297    -       0.165971     0.003559     0.000008         0.000730
    "Omega:Forcing:edge1"                                        -     144    -       0.001040     0.000025     0.000006         0.000010
    "Omega:Stepper:firstDoStep"                                  -       1    -       0.004757     0.004757     0.004757         0.000000
    "Omega:PIO:PIOc_createfile"                                  -       2    -       0.009834     0.007720     0.002115         0.000000
    "Omega:PIO:spio_put_att_tc"                                  -     965    -       0.002869     0.000012     0.000002         0.000068
    "Omega:PIO:PIOc_write_darray"                                -    5766    -      35.425842     0.007411     0.005544         0.000409
    "Omega:PIO:spio_put_vars_tc"                                 -     145    -       0.001435     0.000025     0.000008         0.000010

here is with your changes

*   "Omega:PIO:PIOc_read_darray"                                 -      46    -       0.500419     0.012231     0.009727         0.000003
      "Omega:PIO:read_darray_nc"                                 -      46    -       0.117463     0.002857     0.002119         0.000003
      "Omega:PIO:rearrange_io2comp"                              -      46    -       0.000341     0.000026     0.000004         0.000003
  "Omega:RunLoop"                                                -       1    -       3.779160     3.779160     3.779160         0.000000
*   "Omega:PIO:PIOc_initdecomp"                                  -    5232    -       0.604450     0.000327     0.000003         0.000371
      "Omega:PIO:box_rearrange_create"                           -    5232    -       0.584499     0.000318     0.000002         0.000371
*       "Omega:PIO:pio_swapm"                                    -   20932    -       0.018200     0.000018     0.000000         0.001485
*   "Omega:PIO:PIOc_freedecomp"                                  -    5232    -       0.004225     0.000010     0.000000         0.000371
*   "Omega:PIO:PIOc_sync"                                        -     150    -       0.745799     0.011380     0.000000         0.000011
*   "Omega:PIO:PIOc_closefile"                                   -     150    -       0.060659     0.001070     0.000029         0.000011
      "Omega:PIO:PIOc_closefile_write_mode"                      -     145    -       0.052187     0.001005     0.000206         0.000010
*   "Omega:PIO:spio_get_att_tc"                                  -     150    -       0.000685     0.000010     0.000002         0.000011
*   "Omega:PIO:spio_get_vars_tc"                                 -   10297    -       0.162443     0.001094     0.000008         0.000730
    "Omega:Forcing:edge1"                                        -     144    -       0.001005     0.000030     0.000005         0.000010
    "Omega:Stepper:firstDoStep"                                  -       1    -       0.005467     0.005467     0.005467         0.000000
    "Omega:PIO:PIOc_createfile"                                  -       2    -       0.005314     0.002848     0.002466         0.000000
    "Omega:PIO:spio_put_att_tc"                                  -     865    -       0.002492     0.000012     0.000002         0.000061
    "Omega:PIO:PIOc_write_darray"                                -    5190    -       1.032859     0.007272     0.000006         0.000368
*     "Omega:PIO:write_total"                                    -    6492    -       1.839735     0.011380     0.000002         0.000461
        "Omega:PIO:flush_buffer"                                 -    5335    -       0.117411     0.000077     0.000000         0.000378
          "Omega:PIO:PIOc_write_darray_multi"                    -    5190    -       0.079605     0.000064     0.000005         0.000368
*           "Omega:PIO:flush_output_buffer"                      -    5625    -       0.624651     0.010452     0.000000         0.000399
            "Omega:PIO:rearrange_comp2io"                        -    5190    -       0.058122     0.000059     0.000002         0.000368
            "Omega:PIO:write_darray_multi_par"                   -    5190    -       0.005046     0.000019     0.000000         0.000368
    "Omega:PIO:spio_put_vars_tc"                                 -     145    -       0.001118     0.000034     0.000006         0.000010

I see a massive reduction in the number of sync calls (as I think is expected), write totals go down in many places. I now wonder if some of the improvement is due to other changes in my rebase. Let me test again without this set of changes but with all the rebase changes.

vanroekel · 2026-07-08T15:03:58Z

Here is with the updated code, but not your PR

38 *   "Omega:PIO:PIOc_read_darray"                                 -      46    -       0.510933     0.013034     0.009770         0.000003
 39       "Omega:PIO:read_darray_nc"                                 -      46    -       0.119977     0.003046     0.002131         0.000003
 40       "Omega:PIO:rearrange_io2comp"                              -      46    -       0.000461     0.000030     0.000006         0.000003
 41   "Omega:RunLoop"                                                -       1    -      49.700504    49.700504    49.700504         0.000000
 42 *   "Omega:PIO:PIOc_initdecomp"                                  -    5232    -       0.641882     0.000312     0.000006         0.000370
 43       "Omega:PIO:box_rearrange_create"                           -    5232    -       0.615471     0.000302     0.000004         0.000370
 44 *       "Omega:PIO:pio_swapm"                                    -   20932    -       0.045506     0.000025     0.000000         0.001481
 45 *   "Omega:PIO:PIOc_freedecomp"                                  -    5232    -       0.010339     0.000018     0.000001         0.000370
 46 *   "Omega:PIO:PIOc_sync"                                        -    5340    -      15.635663     0.005643     0.000000         0.000378
 47 *     "Omega:PIO:write_total"                                    -   11682    -      47.650967     0.007380     0.000002         0.000827
 48         "Omega:PIO:flush_buffer"                                 -   10380    -       0.177444     0.000080     0.000000         0.000734
 49           "Omega:PIO:PIOc_write_darray_multi"                    -    5190    -       0.159332     0.000077     0.000014         0.000367
 50             "Omega:PIO:rearrange_comp2io"                        -    5190    -       0.104993     0.000066     0.000006         0.000367
 51             "Omega:PIO:write_darray_multi_par"                   -    5190    -       0.023294     0.000038     0.000002         0.000367
 52 *       "Omega:PIO:flush_output_buffer"                          -   10815    -      15.437488     0.005610     0.000000         0.000765
 53 *   "Omega:PIO:PIOc_closefile"                                   -     150    -       0.076801     0.003252     0.000031         0.000011
 54       "Omega:PIO:PIOc_closefile_write_mode"                      -     145    -       0.068330     0.003185     0.000316         0.000010
 55 *   "Omega:PIO:spio_get_att_tc"                                  -     150    -       0.000750     0.000014     0.000002         0.000011
 56 *   "Omega:PIO:spio_get_vars_tc"                                 -   10297    -       0.166392     0.001107     0.000008         0.000729
 57     "Omega:Forcing:edge1"                                        -     144    -       0.001059     0.000025     0.000006         0.000010
 58     "Omega:Stepper:firstDoStep"                                  -       1    -       0.005398     0.005398     0.005398         0.000000
 59     "Omega:PIO:PIOc_createfile"                                  -       2    -       0.005102     0.002926     0.002176         0.000000
 60     "Omega:PIO:spio_put_att_tc"                                  -     865    -       0.002602     0.000014     0.000002         0.000061
 61     "Omega:PIO:PIOc_write_darray"                                -    5190    -      31.938751     0.007379     0.005475         0.000367
 62     "Omega:PIO:spio_put_vars_tc"                                 -     145    -       0.001444     0.000038     0.000008         0.000010

So the large timing reduction is indeed the changes in this PR. This is all on pm-cpu. Are there other tests you'd like me to run.

And I didn't say this explicitly but I also confirmed the output data is unchanged with this PR.

vanroekel · 2026-07-08T15:23:06Z

Sorry disregard my last comment, there is something wrong with the output. Looking into it to see what the cause is

philipwjones · 2026-07-08T16:58:03Z

@grnydawn thanks for looking into this. As the comment suggested, the sync was there for safety and tests were failing without it, probably because of the Decomp issue that you found. I'm still maybe a little worried about the general PIO buffers, especially if we remove the decomp computation that's adding some overhead and allowing time for buffers to clear. I guess we'll see what Luke's testing finds. I'll use a separate comment for the Decomp caching.

philipwjones · 2026-07-08T17:14:58Z

On the decomp caching. One of the reasons I recomputed every time is that it requires storing address offset arrays for each unique array size. At a minimum, it would require 3 full-sized arrays to manage each of the mesh locations (and a similar set for 2-d fields - tho maybe we could just extract the k=0 slice from the full-sized decomp). And then it would need additional Decomps for any array that didn't conform to the generic sizes (eg NCells x NVertLayers). I suspect we could precompute decomps for the generic sizes and only compute new ones for the outliers. That wouldn't be too large a cost.

grnydawn · 2026-07-08T17:41:33Z

@grnydawn thanks for looking into this. As the comment suggested, the sync was there for safety and tests were failing without it, probably because of the Decomp issue that you found. I'm still maybe a little worried about the general PIO buffers, especially if we remove the decomp computation that's adding some overhead and allowing time for buffers to clear. I guess we'll see what Luke's testing finds. I'll use a separate comment for the Decomp caching.

@philipwjones , I am learning the IOStream implementation while debugging, so there could be some misunderstanding on my part. My understanding is that the key issue was deleting the decomposition after every field write while the PIO buffer still had buffered data. The sync operation forces the data to be flushed to the file before tearing the decomposition down. Yes, let’s wait to see what Luke finds.

grnydawn · 2026-07-08T17:46:14Z

On the decomp caching. One of the reasons I recomputed every time is that it requires storing address offset arrays for each unique array size. At a minimum, it would require 3 full-sized arrays to manage each of the mesh locations (and a similar set for 2-d fields - tho maybe we could just extract the k=0 slice from the full-sized decomp). And then it would need additional Decomps for any array that didn't conform to the generic sizes (eg NCells x NVertLayers). I suspect we could precompute decomps for the generic sizes and only compute new ones for the outliers. That wouldn't be too large a cost.

This approach of precalculating the common sizes and only computing new ones when necessary seems like the right direction to me.

vanroekel · 2026-07-08T22:09:37Z

Okay sorry for the slow return to this. I had some issues to work through in my codebase. I was able to test again with reasonable results. From what I can tell with this PR and without there are no diffs in the output. This could be a consequence of the configuration I'm testing (16 cell planar), so should probably take results with a grain of salt

grnydawn · 2026-07-09T14:31:07Z

Okay sorry for the slow return to this. I had some issues to work through in my codebase. I was able to test again with reasonable results. From what I can tell with this PR and without there are no diffs in the output. This could be a consequence of the configuration I'm testing (16 cell planar), so should probably take results with a grain of salt

@vanroekel , Good to know that you got the same output. So far, I have tested this PR only with the Omega ctests on PM-CPU, PM-GPU, and Chrysalis. I think the omega_pr Polaris test suite with this PR will provide a more thorough validation. I will post an update here once I get the test results from the test suite.

grnydawn · 2026-07-10T13:39:22Z

I got the test results from running the Polaris omega_pr test suite on Chrysalis, but they don't seem to be definitive yet. Three out of 21 tests failed. One failed due to insufficient allocation time, and the other two failed with the error: "Error: order of convergence for tracer1". These same failures also occur without applying this PR.

I used the latest commit from the develop branch, which may not be compatible with the current Polaris version. I'll do some more testing.

sbrus89 · 2026-07-14T14:31:43Z

@grnydawn, the polaris/Omega compatibility issue has been sorted out as of yesterday. Sorry if that caused any problems in your testing.

xylar · 2026-07-22T16:12:15Z

@grnydawn, as you guessed, this needs to be rebased onto develop. We also need to lint the commit (see CI).

xylar · 2026-07-22T19:35:34Z

Testing

After merging develop into this branch, I ran:

Polaris `omega_pr` suite

Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-baseline-cpu
Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-baseline-cpu/build
PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-iostream2-cpu/build
PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-iostream2-cpu
Machine: aurora
Compiler: oneapi-ifx
Build type: Release
Log: not found
Result: All tests passed

Polaris `omega_pr` suite

Baseline workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-baseline-gpu
Baseline build: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-baseline-gpu/build
PR build: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-iostream2-gpu/build
PR workdir: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-iostream2-gpu
Machine: aurora
Compiler: oneapi-ifxgpu
Build type: Release
Log: not found
Result: All tests passed

CTest unit tests:

Machine: aurora
Compiler: oneapi-ifx
Build type: Release
Result: All tests passed
Log: /lus/flare/projects/E3SM_Dec/xylar/polaris_0.10/aurora/test_20260722/omega-pr-iostream2-cpu/build/ctests.log

Note: GPU CTests are expected to fail with the issues reported in #368

xylar · 2026-07-22T19:53:34Z

My testing didn't show any particular speed up but these tests aren't really meant to exercise that. I don't see any signs of trouble in the omega_pr suite. But I agree with the discussion above that testing is needed on a large, realistic mesh before we sign off on these changes.

grnydawn · 2026-07-23T19:45:31Z

@xylar , Thanks for the testing results. Your results align with mine in that all tests passed without errors.

Regarding timing, I am collecting more detailed timing results from Chrysalis and Aurora to compare performance with and without this PR. PM-CPU is the primary target for this evaluation. Unfortunately, the system is currently down.

xylar · 2026-07-23T20:39:55Z

Perlmutter will be down for several weeks so I would suggest finding an alternative system (Aurora or Frontier) to do timing on.

grnydawn · 2026-07-24T14:42:00Z

I ran the omega_pr suite on Aurora and Chrysalis with and without this PR and collected the timing results from all *.timing.0 files generated by each tests in the suite.

Metric	Aurora	Chrysalis
`PIOc_sync` calls (nofix → fix)	5,422 → 878 (6.2× fewer)	4,737 → 793 (6.0× fewer)
`PIOc_sync` wall time (nofix → fix)	55.04 s → 13.36 s	125.24 s → 34.26 s
`PIOc_sync` time saved	41.68 s (4.12×)	90.98 s (3.66×)
Median per‑case sync speedup	6.8×	8.2×
Cases improved / total	83 / 85	80 / 80
Net total runtime saved	39.93 s (1.7%)	183.69 s (6.8%)

The speedup is directly proportional to the number of I/O synchronization calls eliminated by this update on both systems.

However, Aurora's net overall runtime improvement (1.7%) is much smaller than Chrysalis's gain because PIOc_createfile regresses significantly: 32.7 s → 67.3 s (+34.6 s), with the fix version being slower in 55 of 85 cases (in the worst cases, a single PIOc_createfile call increases from approximately 0.14 s to 3.1 s). On Chrysalis, this timer remains essentially unchanged. There is currently no clear explanation for this behavior on the Aurora file system.

pr463_pio_timing_percase.csv

cbegeman

Approving on the basis of @grnydawn 's timing study and @xylar 's polaris PR testing on aurora.

philipwjones

I'll go ahead and approve so I'm not in the way, but it would still be nice to make sure this is ok on a larger production mesh.

grnydawn requested review from cbegeman, philipwjones and vanroekel July 7, 2026 19:47

grnydawn added the PIO Parallel IO label Jul 7, 2026

Reduce the number of PIO sync during IOStream write

d115dae

grnydawn force-pushed the ykim/omega/iostream branch from 0c52877 to d115dae Compare July 24, 2026 14:20

cbegeman approved these changes Jul 28, 2026

View reviewed changes

philipwjones approved these changes Jul 31, 2026

View reviewed changes

Uh oh!

Conversation

grnydawn commented Jul 7, 2026 • edited by xylar Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanroekel commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanroekel commented Jul 8, 2026

Uh oh!

vanroekel commented Jul 8, 2026

Uh oh!

philipwjones commented Jul 8, 2026

Uh oh!

philipwjones commented Jul 8, 2026

Uh oh!

grnydawn commented Jul 8, 2026

Uh oh!

grnydawn commented Jul 8, 2026

Uh oh!

vanroekel commented Jul 8, 2026

Uh oh!

grnydawn commented Jul 9, 2026

Uh oh!

grnydawn commented Jul 10, 2026

Uh oh!

sbrus89 commented Jul 14, 2026

Uh oh!

xylar commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Polaris omega_pr suite

Polaris omega_pr suite

CTest unit tests:

Uh oh!

xylar commented Jul 22, 2026

Uh oh!

grnydawn commented Jul 23, 2026

Uh oh!

xylar commented Jul 23, 2026

Uh oh!

grnydawn commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman left a comment

Choose a reason for hiding this comment

Uh oh!

philipwjones left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

grnydawn commented Jul 7, 2026 •

edited by xylar

Loading

vanroekel commented Jul 8, 2026 •

edited

Loading

xylar commented Jul 22, 2026 •

edited

Loading

xylar commented Jul 22, 2026 •

edited

Loading

Polaris `omega_pr` suite

Polaris `omega_pr` suite

grnydawn commented Jul 24, 2026 •

edited

Loading