Document Gadi MPI rank-placement performance for MatMult-dominated Stokes solves

[gthyagi]

Summary

A controlled Underworld3 Stokes benchmark on NCI Gadi shows a large dependence on MPI rank
count, node type, and launch configuration. The numerical workload remains effectively
unchanged, and approximately 97% of the solve time is spent in PETSc MatMult. This would be
useful to investigate and document so Gadi users can choose appropriate MPI resources and rank
placement.

This is a performance and deployment question, not a reported numerical-correctness failure.
Every run completed successfully and produced the expected output.

Environment

• Underworld3 commit: 5595b665
• PETSc: 3.25.4
• OpenMPI: 4.1.7
• HDF5: 1.12.2p
• Platform: NCI Gadi
• Queues compared: normal (Cascade Lake nodes) and normalsr (Sapphire Rapids nodes)
• One OpenMP/BLAS thread per MPI rank

Benchmark

The test is a 2-D nonlinear, variable-viscosity Stokes model based on the Barr--Houseman fault
benchmark:

• P2/P1 velocity-pressure elements on the same triangular mesh for every run
• nominal global cell size: 0.01 model units
• 200 tagged rectangular-fault cells
• power-law exponent: n=3
• viscosity contrast: 1e5
• Stokes tolerance: 1e-8
• four SNES solves and 24 KSP solves in every completed case
• identical mesh SHA-256:
  f9ade30ff26dbb9cfdb671148f6f1dcec70473cbe18ae4ad8f126d6fda60623b

Representative launch command:

export OPENBLAS_NUM_THREADS=1
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1

mpiexec -n ${NP} ${MPI_ARGS} python3 bh92_rectangular_fault_uw_surface.py \
    -uw_run_on_gadi true \
    -uw_cellsize 0.01 \
    -uw_n 3 \
    -uw_fault_hw 0.0005 \
    -uw_fault_nx 101 \
    -uw_fault_ny 2 \
    -uw_fault_identification element_mask \
    -uw_visc_diff 1e5 \
    -uw_flow transFlow \
    -uw_stokes_pen 1 \
    -uw_stokes_tol 1e-8 \
    -uw_projection_tol 1e-8 \
    -uw_ncpus ${NP} \
    -uw_render_plots false \
    -uw_proj_monitor false

The explicit-binding cases used:

MPI_ARGS="--map-by core --bind-to core --report-bindings"

The attempted no-ML cases additionally set OMPI_MCA_coll=^ml.

Results

Queue and launch configuration	Ranks	Stokes solve (s)	PBS wall time	Peak memory	MatMult time (s)	MatMult rate (Mflop/s)
normal, default MPI placement	8	4932.1	1:23:34	8.99 GB	4811.2	6707
normal, explicit core binding	8	5751.4	1:37:02	9.13 GB	5624.2	5738
normal, binding + attempted no-ML	8	5708.0	1:36:25	8.97 GB	5581.4	5782
normal, binding + attempted no-ML	4	8963.9	2:30:48	7.31 GB	8739.4	3842
normal, binding + attempted no-ML	12	3288.3	0:56:02	10.1 GB	3201.7	9995
normalsr, binding + attempted no-ML	8	3992.2	1:07:43	9.25 GB	3884.4	8308

An earlier 8-rank normal run with default placement took 4540.6 s for the Stokes solve,
compared with 4932.1 s for the repeat above. This is an 8.6% variation between normal nodes.
The campaign configurations also ran concurrently on different nodes, so node-to-node variation
partly confounds direct attribution of the observed difference to binding alone.

Observations

1. The solver path did not change: all configurations used four SNES solves and 24 KSP solves.
2. Runtime differences closely follow MatMult throughput rather than nonlinear convergence.
3. Twelve ranks gave the best Gadi wall time in this sample (56:02), while four ranks reduced
   memory use but required 2:30:48.
4. Explicit compact core binding did not improve the 8-rank result in this campaign. A controlled
   same-node comparison or a socket-distributed mapping test is needed before drawing a firm
   affinity conclusion.
5. The normalsr 8-rank run was faster than the repeated normal 8-rank run, but slower than the
   12-rank normal run.
6. OMPI_MCA_coll=^ml neither materially changed the bound-run timing nor suppressed messages of
   the following form:

[LOG_CAT_ML] component basesmuma is not available but requested in hierarchy
[LOG_CAT_ML] ml_discover_hierarchy exited with error

These messages were nonfatal.

Requested guidance

Could the recommended Gadi launch configuration for memory-bandwidth-dominated UW3/PETSc solves
be investigated and documented, particularly:

• rank count and placement across sockets for a single Gadi node;
• whether UW3's Gadi examples should recommend explicit OpenMPI mapping/binding options;
• the correct way to disable or silence the unavailable OpenMPI basesmuma/ML component; and
• whether a small repeatable PETSc timing benchmark should be included with the Gadi setup notes.

Mac comparison

The identical mesh and model were also run locally on an Apple Silicon Mac:

• host: Apple M2 Pro (arm64), 10 CPU cores, 16 GiB RAM;
• MPI ranks: 8;
• PETSc recorded by the run: 3.25.0;
• same mesh SHA-256 as the Gadi campaign:
  f9ade30ff26dbb9cfdb671148f6f1dcec70473cbe18ae4ad8f126d6fda60623b;
• same nonlinear workload: four SNES solves, 24 KSP solves, and 155,552 MatMult calls.

Platform	Ranks	Mesh creation (s)	Stokes solve (s)	Total through HDF5 (s)	MatMult time (s)	KSPSolve time (s)	MatMult rate (Mflop/s)
Apple M2 Pro	8	1.1428	2421.7	2432	2341.8	1815.2	13780
Gadi normal, original default-placement baseline	8	2.1225	4540.6	4572	4415.8	3403.4	7308

The Mac completed the Stokes solve 1.88 times faster than the original Gadi 8-rank baseline.
The difference again follows MatMult throughput: 13,780 Mflop/s on the Mac versus 7,308
Mflop/s on that Gadi node.

This is a cross-platform observation, not a controlled hardware comparison. The Mac run used
PETSc 3.25.0, whereas Gadi used PETSc 3.25.4. The Mac output did not record the exact UW3 Git
SHA or OpenMPI version used at run time, and the machines differ in processor architecture,
memory subsystem, MPI implementation/configuration, and filesystem. It nevertheless provides a
useful indication that the Gadi timing difference is dominated by sparse matrix-vector throughput
rather than additional nonlinear or linear solver work.