Problem loading from-source build on Cray machine: libStgDomain.so undefined MPI symbols not found in MPI library

[aproeme]

Hello,

Error description

I've built Underworld2 from source on a Cray machine (ARCHER2), but importing fails with an undefined MPI symbol (mpi_pack_):

(uw2pyenv)> python -m "import underworld"
[...rest of stack trace...]
ImportError: /mnt/lustre/a2fs-work2/work/z19/shared/underworld2/uw2pyenv/lib/python3.10/site-packages/underworld/lib/libStgDomain.so: undefined symbol: mpi_pack_

I can see that libStgDomain.so is linked to the correct MPI library (Cray MPICH):

> ldd /mnt/lustre/a2fs-work2/work/z19/shared/underworld2/uw2pyenv/lib/python3.10/site-packages/underworld/lib/libStGermain.so 
[...various libs...]
libmpi_gnu_91.so.12 => /opt/cray/pe/lib64/libmpi_gnu_91.so.12 (0x00007f366b200000)
[...various libs...]

This MPI library contains the symbol definition for MPI_Pack:

> nm -a /opt/cray/pe/lib64/libmpi_gnu_91.so.12 | grep -w MPI_Pack
0000000000721e00 W MPI_Pack

but not for for mpi_pack_:

> nm -a /opt/cray/pe/lib64/libmpi_gnu_91.so.12 | grep -w mpi_pack_
>

I think the issue is therefore a case mismatch in the symbol names. Would you be able to advise why libStGermain.so expects mpi_pack_ (all lowercase) rather than MPI_Pack, wether I've overlooked anything and how I might fix this?

Build procedure

My build procedure for Underworld2 was as follows:

Load relevant modules:

module load PrgEnv-gnu
module load petsc
module load cray-hdf5-parallel
module load cray-python

Create virtual environment (based off of cray-python, which provides mpi4py linked to the Cray MPICH library):

python -m venv --system-site-packages ./uw2pyenv
source ./uw2pyenv/bin/activate

Install h5py using the guidance provided for MPI-parallel h5py (note the Cray compiler wrapper cc links to the Cray MPICH library):

CC=cc HDF5_MPI="ON" pip install --no-binary=h5py h5py

Set compilers to be the Cray compiler wrappers:

export MPICC=cc
export MPICXX=CC
export MPIF90=ftn

Make sure cmake can find the LibXml2 and PETSc installs provided by the modules:

export PKG_CONFIG_PATH=$PETSC_DIR/lib/pkgconfig:$PKG_CONFIG_PATH
export PKG_CONFIG_PATH=$LIBXML2_DIR/lib/pkgconfig:$PKG_CONFIG_PATH

Then in src/underworld/libUnderworld/CMakeLists.txt change

pkg_check_modules(PETSc PETSc) 

to

pkg_check_modules(PETSc petsc)

Since the pkconfig file in our PETSc install is called petsc.pc not PETSc.pc <--- possibly related to the lowercase/uppercase symbol issue?

Finally pip install, with output shown:

(uw2pyenv)> pip install .

Processing /mnt/lustre/a2fs-work2/work/z19/shared/underworld2/underworld2-2.17.0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
[...etc...]
Successfully built underworld
Installing collected packages: underworld
Successfully installed underworld-2.17.0

The attempted import of underworld in a python session fails as described above.

[julesghub]

Hi @aproeme,

Thanks for raising this. Great you're trying a new setup of uw-2.17 on your HPC.

Interesting about the upper/lower case symbol issues.
Does a ldd for libStDomain.so, not libStGermain.so show the correct libmpi with the same upper/lower case issue?

Thanks.

[aproeme]

Hi!

Apologies for the 'bug' in my issue report, not sure why the reference to libStGermain.so crept in: that library does not appear in the error on import.

I can confirm that what I reported holds for libStgDomain.so:

> ldd libStgDomain.so 
[...various libs...]
	libmpi_gnu_91.so.12 => /opt/cray/pe/lib64/libmpi_gnu_91.so.12 (0x00007f7b36800000)
[...various libs...]

i.e. it's linked to the same correct Cray MPICH library as reported above for libStGermain.so.

Confirmation that mpi_pack_ is indeed undefined in libStgDomain.so:

> nm -a libStgDomain.so | grep -w mpi_pack_
                 U mpi_pack_

Interestingly libStgDomain.so does actually seem to contain many uppercase MPI symbols:

> nm -a libStgDomain.so | grep " MPI_" 
                 U MPI_Abort
                 U MPI_Accumulate
                 U MPI_Add_error_class
                 U MPI_Add_error_code
                 U MPI_Allgather
                 U MPI_Allgatherv
                 U MPI_Alloc_mem
                 U MPI_Allreduce
                 U MPI_Alltoall
                 U MPI_Alltoallv
                 U MPI_Barrier
[...many more symbols....]

> nm -a libStgDomain.so | grep " MPI_" | wc -l
144

These will match the uppercase form found in Cray MPICH:

> nm -a /opt/cray/pe/lib64/libmpi_gnu_91.so.12 | grep " MPI_"
0000000000749700 W MPI_Abort
00000000011f04e0 W MPI_Accumulate
0000000000739e80 W MPI_Add_error_class
0000000000739ab0 W MPI_Add_error_code
000000000073a260 W MPI_Add_error_string
0000000000720860 W MPI_Address
000000000074b9d0 W MPI_Aint_add
000000000074ba10 W MPI_Aint_diff
00000000000c1380 W MPI_Allgather
00000000000c2b70 W MPI_Allgatherv
000000000122b970 W MPI_Alloc_mem
000000000020dc90 W MPI_Allreduce
000000000020f2e0 W MPI_Alltoall
00000000002105b0 W MPI_Alltoallv
0000000000211a60 W MPI_Alltoallw
00000000000b6490 W MPI_Attr_delete
00000000000b6ce0 W MPI_Attr_get
00000000000b7290 W MPI_Attr_put
000000000029d230 W MPI_Barrier

These are all the lowercase MPI symbols in libStgDomain.so:

> nm -a libStgDomain.so | grep " mpi_"
                 U mpi_abort_
                 U mpi_allgatherv_
                 U mpi_allreduce_
                 U mpi_alltoall_
                 U mpi_barrier_
                 U mpi_bcast_
                 U mpi_cancel_
                 U mpi_comm_dup_
                 U mpi_comm_free_
                 U mpi_comm_rank_
                 U mpi_comm_size_
                 U mpi_comm_split_
                 U mpi_gather_
                 U mpi_get_count_
                 U mpi_get_processor_name_
                 U mpi_init_
                 U mpi_initialized_
                 U mpi_iprobe_
                 U mpi_irecv_
                 U mpi_isend_
                 U mpi_op_create_
                 U mpi_op_free_
                 U mpi_pack_
                 U mpi_pack_size_
                 U mpi_probe_
                 U mpi_recv_
                 U mpi_reduce_
                 U mpi_reduce_scatter_
                 U mpi_request_free_
                 U mpi_send_
                 U mpi_ssend_
                 U mpi_test_
                 U mpi_type_commit_
                 U mpi_type_contiguous_
                 U mpi_type_free_
                 U mpi_unpack_
                 U mpi_wait_
                 U mpi_waitall_
                 U mpi_waitany_
                 U mpi_wtime_

> nm -a libStgDomain.so | grep " mpi_" | wc -l
40

Clearly there are duplicates in libStgDomain.so (e.g. mpi_abort_ and MPI_Abort). In other words it looks like for a subset of the MPI symbols there are also Fortran name-mangled duplicates. I wonder where these came from.

[julesghub]

Hi @aproeme,

I just tried compiling the uw-2.17 on a Cray machine I have access to and import underworld worked for me - no doubt my machine is different from ARCHER2.

Rereading you're original post - it might be helpful to put these exports BEFORE you create your python virtual environment.

export MPICC=cc
export MPICXX=CC
export MPIF90=ftn

Does that help?

Sorry for the late response on this. Let me know the latest.

[aproeme]

Thank you for the suggestion, I tried it but got the same result.

However I believe I have since narrowed this down to two closely related things that are to do with the software stack configuration specific to ARCHER2:

1. The version of PETSc on our system was built to only link against a Fortran-specific variant of Cray's MPICH library (which does supply lowercase mpi symbols). This was responsible for inclusion of mpi_abort_ and mpi_init_.

2. module load petsc also loads other third party scientific libraries (SuperLU, MUMPS, etc.) that were built similarly and cause unintentional inclusion of additional lowercase mpi symbols as they are implicitly linked to by the cc compiler wrapper.

I therefore believe this is not an issue with Underworld2 and we should be able to resolve it locally.

[julesghub]

Hi @aproeme,
I see - thanks for clarification on this.
Keen to hear more when you get to the bottom of it on ARCHER2.