Skip to content

Enable OpenMP Enzyme fixture checks#47

Open
krystophny wants to merge 21 commits into
developfrom
ad-13-openmp-enzyme-gradient
Open

Enable OpenMP Enzyme fixture checks#47
krystophny wants to merge 21 commits into
developfrom
ad-13-openmp-enzyme-gradient

Conversation

@krystophny

Copy link
Copy Markdown
Member

Depends on #46.
Closes #14.

Summary

  • seed flang OpenMP discovery with -fopenmp and libomp before find_package(OpenMP)
  • pass both global and build-type Fortran flags through the generated Enzyme pipeline so differentiated compilation sees -fopenmp
  • avoid omp_lib.mod in OpenMP-enabled Fortran sources by declaring the OpenMP runtime calls directly
  • run the Enzyme AD fixture with one OpenMP thread by default and add a four-thread CTest variant

Verification

Test fails on main

Before this PR, the stacked parent commit fails while configuring the flang OpenMP Enzyme build:

$ cmake -S <parent-source> -B <build> -G Ninja -DCMAKE_Fortran_COMPILER=flang-new -DCMAKE_BUILD_TYPE=Release -DGVEC_FIX_HMAP=RZ -DUSE_OPENMP=ON -DLINK_GVEC_TO_NETCDF=OFF -DUSE_ENZYME=ON
cmake exit=1
-- The Fortran compiler identification is LLVMFlang 22.1.6
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:290 (message):
  Could NOT find OpenMP_Fortran (missing: OpenMP_Fortran_FLAGS
  OpenMP_Fortran_LIB_NAMES)
-- Configuring incomplete, errors occurred!

Test passes after fix

$ cmake -S . -B <build-rz-omp> -G Ninja -DCMAKE_Fortran_COMPILER=flang-new -DCMAKE_BUILD_TYPE=Release -DGVEC_FIX_HMAP=RZ -DUSE_OPENMP=ON -DLINK_GVEC_TO_NETCDF=OFF -DUSE_ENZYME=ON
-- The Fortran compiler identification is LLVMFlang 22.1.6
-- Found OpenMP_Fortran: -fopenmp
-- Found OpenMP: TRUE  found components: Fortran
-- Configuring done
-- Build files have been written to: <build-rz-omp>

$ cmake --build <build-rz-omp> --target gvec_enzyme_ad_fixture_gradient -j$(nproc)
[134/134] Linking Fortran executable bin/gvec_enzyme_ad_fixture_gradient

$ rg -n "COMPILE_OPTIONS|OMP_NUM_THREADS" <build-rz-omp>/gvec_enzyme_ad_fixture_gradient.enzyme/run_pipeline.cmake <build-rz-omp>/CTestTestfile.cmake
COMPILE_OPTIONS "...|-fopenmp|...|-DPP_WHICH_HMAP=1"
set_tests_properties([=[gvec_enzyme_ad_fixture_gradient]=] PROPERTIES ENVIRONMENT "OMP_NUM_THREADS=1" ...)
set_tests_properties([=[gvec_enzyme_ad_fixture_gradient_omp4]=] PROPERTIES ENVIRONMENT "OMP_NUM_THREADS=4" ...)

$ ctest --test-dir <build-rz-omp> -R '^gvec_enzyme_ad_fixture_gradient$' --output-on-failure
1/1 Test #5: gvec_enzyme_ad_fixture_gradient ...   Passed
100% tests passed, 0 tests failed out of 1

$ ctest --test-dir <build-rz-omp> -R '^gvec_enzyme_ad_fixture_gradient_omp4$' --output-on-failure
1/1 Test #6: gvec_enzyme_ad_fixture_gradient_omp4 ...   Passed
100% tests passed, 0 tests failed out of 1
$ cmake -S . -B <build-cyl-omp> -G Ninja -DCMAKE_Fortran_COMPILER=flang-new -DCMAKE_BUILD_TYPE=Release -DGVEC_FIX_HMAP=cyl -DUSE_OPENMP=ON -DLINK_GVEC_TO_NETCDF=OFF -DUSE_ENZYME=ON
-- The Fortran compiler identification is LLVMFlang 22.1.6
-- Found OpenMP_Fortran: -fopenmp
-- Found OpenMP: TRUE  found components: Fortran
-- Configuring done

$ rg -n "COMPILE_OPTIONS.*fopenmp|OMP_NUM_THREADS" <build-cyl-omp>/gvec_enzyme_ad_fixture_gradient.enzyme/run_pipeline.cmake <build-cyl-omp>/CTestTestfile.cmake
COMPILE_OPTIONS "...|-fopenmp|...|-DPP_WHICH_HMAP=3"
set_tests_properties([=[gvec_enzyme_ad_fixture_gradient]=] PROPERTIES ENVIRONMENT "OMP_NUM_THREADS=1" ...)
set_tests_properties([=[gvec_enzyme_ad_fixture_gradient_omp4]=] PROPERTIES ENVIRONMENT "OMP_NUM_THREADS=4" ...)

$ ctest --test-dir <build-cyl-omp> -R '^gvec_enzyme_ad_fixture_gradient(_omp4)?$' --output-on-failure
1/2 Test #5: gvec_enzyme_ad_fixture_gradient ........   Passed
2/2 Test #6: gvec_enzyme_ad_fixture_gradient_omp4 ...   Passed
100% tests passed, 0 tests failed out of 2
$ cmake -S . -B <build-serial-rz> -G Ninja -DCMAKE_Fortran_COMPILER=flang-new -DCMAKE_BUILD_TYPE=Release -DGVEC_FIX_HMAP=RZ -DUSE_OPENMP=OFF -DLINK_GVEC_TO_NETCDF=OFF -DUSE_ENZYME=ON
-- The Fortran compiler identification is LLVMFlang 22.1.6
-- Configuring done

$ ctest --test-dir <build-serial-rz> -R '^gvec_enzyme_ad_fixture_gradient$' --output-on-failure
1/1 Test #5: gvec_enzyme_ad_fixture_gradient ...   Passed
100% tests passed, 0 tests failed out of 1
$ cmake -S . -B <build-gfortran-omp> -G Ninja -DCMAKE_BUILD_TYPE=Release -DGVEC_FIX_HMAP=RZ -DUSE_OPENMP=ON -DLINK_GVEC_TO_NETCDF=OFF
-- The Fortran compiler identification is GNU 16.1.1
-- Found OpenMP_Fortran: -fopenmp (found version "5.2")
-- Found OpenMP: TRUE (found version "5.2") found components: Fortran
-- Configuring done

$ cmake --build <build-gfortran-omp> -j$(nproc)
[140/141] Linking Fortran executable bin/gvec
SUCCESS: GVEC BUILD COMPLETE!
[141/141] Linking Fortran executable bin/gvec_post
SUCCESS: GVEC_POST BUILD COMPLETE!
$ git diff --check
<no output>

$ fo
fo: no Fortran project detected

@krystophny

Copy link
Copy Markdown
Member Author

Fixed the AD-13 OpenMP Enzyme path in this branch.

The change covers three failure points:

  • flang now gets OpenMP discovery seeded with -fopenmp and libomp before CMake calls find_package(OpenMP);
  • the generated Enzyme Fortran pipeline now receives both global and build-type Fortran flags, so -fopenmp is present while Enzyme emits and recompiles LLVM IR;
  • OpenMP runtime calls no longer require a compiler-specific omp_lib.mod, which keeps flang and gfortran builds on the same source path.

Local verification covers RZ serial Enzyme, RZ OpenMP Enzyme with one and four threads, cyl OpenMP Enzyme with one and four threads, and a normal gfortran OpenMP build.

@krystophny krystophny force-pushed the ad-13-openmp-enzyme-gradient branch from d5ddc97 to ee7dc00 Compare July 9, 2026 20:01
@krystophny krystophny force-pushed the ad-13-openmp-enzyme-gradient branch from ee7dc00 to 1924c34 Compare July 9, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant