Skip to content

ci(test): run module unittests and integration suites in parallel - #7754

Open
MrLi000001 wants to merge 2 commits into
deepmodeling:developfrom
MrLi000001:ci/test-parallel
Open

ci(test): run module unittests and integration suites in parallel#7754
MrLi000001 wants to merge 2 commits into
deepmodeling:developfrom
MrLi000001:ci/test-parallel

Conversation

@MrLi000001

Copy link
Copy Markdown

The test workflow executed 16 module-unittest steps plus 10 integration suite steps fully serially - measured ~18-21 min of wall time even when the Build step only takes ~3 min with a warm ccache.

  • Merge the 16 MODULE_* steps into one 'ctest -j $(nproc) -R MODULE_' invocation (PERF_MODULE_HSOLVER_KERNELS still excluded).
  • Merge the 10 integration suite steps into one 'ctest -j 4' invocation. Each suite is an independent ctest entry running Autotest.sh in its own tests/ directory, so they can run concurrently. -j4 is used instead of nproc to avoid extreme oversubscription, since every suite already spawns MPI ranks with OMP_NUM_THREADS=2.

Test selection is unchanged (same -R/-E filters). Expected: the test portion drops from ~18-21 min to ~6-8 min of wall time.

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

Fix #

Unit Tests and/or Case Tests for my changes

  • Commands run:
  • Result summary:
  • Checks not run, with reason:

What's changed?

  • Example: brief summary of the user-visible or developer-facing change.

Governance Notes

  • INPUT/docs changes:
  • Core module impact:
  • Exceptions requested:

The test workflow executed 16 module-unittest steps plus 10 integration
suite steps fully serially - measured ~18-21 min of wall time even when
the Build step only takes ~3 min with a warm ccache.

- Merge the 16 MODULE_* steps into one 'ctest -j $(nproc) -R MODULE_'
  invocation (PERF_MODULE_HSOLVER_KERNELS still excluded).
- Merge the 10 integration suite steps into one 'ctest -j 4' invocation.
  Each suite is an independent ctest entry running Autotest.sh in its
  own tests/<suite> directory, so they can run concurrently. -j4 is
  used instead of nproc to avoid extreme oversubscription, since every
  suite already spawns MPI ranks with OMP_NUM_THREADS=2.

Test selection is unchanged (same -R/-E filters). Expected: the test
portion drops from ~18-21 min to ~6-8 min of wall time.
@mohanchen mohanchen added the Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS label Aug 2, 2026
…erial

The first version attempted to run all 10 integration suites concurrently
with ctest -j4, but this caused severe oversubscription on the X64
self-hosted runner (each suite spawns 4 MPI ranks with OMP_NUM_THREADS=2,
so 4 concurrent suites = 32 threads on an 8-core runner). The test
step took 91 min and failed with ctest exit code 8 (timeout).

This version keeps the 16 MODULE_* steps merged into a single parallel
ctest -j $(nproc) invocation (gtest binaries, no MPI oversubscription,
measured ~4.5 min serial -> ~1 min parallel) but reverts the integration
suite steps back to serial execution.
@Stardust0831

Stardust0831 commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Parallelizing the module tests is a reasonable direction, and this step took 1m49s in the actual run. However, MODULE_ESOLVER now runs once under All Module Unittests and again under Other Unittests. Some MODULE tests also launch 2 or 4 MPI processes internally, so please add CTest PROCESSORS metadata or use a conservative fixed parallelism instead of nproc to avoid oversubscription. The stated 6–8-minute target was not reached; the full test section still took about 19 minutes.

@Critsium-xy

Copy link
Copy Markdown
Collaborator

One blocking issue that hasn't been raised yet: -j breaks test isolation that this repo never had.

AddTest sets WORKING_DIRECTORY $<TARGET_FILE_DIR:...> (cmake/Testing.cmake:65) and the hand-written MPI add_tests use ${CMAKE_CURRENT_BINARY_DIR} — the same directory. So all ctest entries in one test subdir share a CWD. Concrete case, source/source_io/test/CMakeLists.txt:22-30:

AddTest(TARGET MODULE_IO_read_exit_file_test ...)              # serial entry
add_test(NAME MODULE_IO_read_exit_file_test_para_4
      COMMAND mpirun -np 4 ./MODULE_IO_read_exit_file_test     # same binary
      WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})           # same directory

Two independent ctest entries, same executable, same CWD — and the test creates and std::removes a fixed-name file "EXIT" (read_exit_file_test.cpp:26,51,102). Serially they could never overlap; under -j they can, and one deletes the file the other is still using. The same "serial + _para_N twin sharing a CWD" shape exists for input_test_para, read_wfc_pw_test, read_wf2rho_pw_test, cif_io_test, orb_io_test, write_dmk, read_wfc_nao_test, plus source_base/test_parallel:50, source_basis/module_pw/test:22, source_estate/test_mpi:16, source_hsolver/test:181.

Options: RESOURCE_LOCK per test directory, a per-test WORKING_DIRECTORY, or RUN_SERIAL on the known offenders. Either way please run the workflow several times before merging — one green run doesn't disprove a race.

Two additions to @Stardust0831's points, both root causes rather than new findings:

  • MODULE_ESOLVER duplication: the trailing "Other Unittests" -E list enumerates 15 MODULE_* prefixes and omits MODULE_ESOLVER, which is one of the 17 prefixes actually present in the tree. -R 'MODULE_' is broader than the union of the old 15 steps, hence the double run. Replacing those 15 entries in -E with a single MODULE_ is less fragile than adding one more.
  • Why the 6–8 min target was missed: no PROCESSORS metadata anywhere, so the MPI entries (-np 2/3/4) are scheduled as cost 1. With OMP_NUM_THREADS=2, a single -np 4 entry already wants 8 threads, so -j $(nproc) oversubscribes heavily and the gain is eaten by contention. set_tests_properties(... PROPERTIES PROCESSORS N) is what makes -j meaningful. Note source_basis/module_pw/test:22 runs mpirun -np 3 and -np 4 inside a single entry, and nproc in a container reports host cores, not the cgroup quota.

Finally, the description says the 10 integration suite steps were merged into one ctest -j 4, but the diff doesn't contain that — 01_PW through 10_others are still 10 serial steps. That likely explains the missed target more than anything else, since those dominate wall time. Please either implement that half or update the description.

Minor: with the steps merged, --output-on-failure instead of -V keeps the same debugging value with a far smaller log, which matters now that the GitHub UI no longer shows which module failed at a glance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants