diff --git a/_quarto.yml b/_quarto.yml index 060f154..1f773d9 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -11,26 +11,75 @@ book: repo-branch: main repo-actions: [edit, issue] favicon: figures/ippl-logo.png + sidebar: + collapse-level: 1 chapters: - index.qmd - - sections/overview/index.qmd - - sections/theory/index.qmd - - sections/getting-started/index.qmd - - sections/examples/alpine-manager-hierarchy.qmd - - sections/core-concepts/index.qmd - - sections/fields/index.qmd - - sections/particles/index.qmd - - sections/particle-mesh/index.qmd - - sections/fft/index.qmd - - sections/poisson-solvers/index.qmd - - sections/maxwell-solvers/index.qmd - - sections/fem/index.qmd - - sections/communication/index.qmd - - sections/performance-portability/index.qmd - - sections/examples/index.qmd - - sections/developer-guide/index.qmd - - sections/api-reference/index.qmd - - sections/literature/index.qmd + + - part: sections/introduction/index.qmd + chapters: + - sections/introduction/theory/index.qmd + - sections/introduction/overview/index.qmd + + # - part: "Getting Started" + - part: sections/getting-started/index.qmd + chapters: + - sections/getting-started/installation/index.qmd + - sections/getting-started/hello-ippl/index.qmd + - sections/getting-started/slurm/index.qmd + + - part: sections/ippl-core/index.qmd + chapters: + - sections/ippl-core/core-concepts/index.qmd + - sections/ippl-core/fields/index.qmd + - sections/ippl-core/particles/index.qmd + - sections/ippl-core/communication/index.qmd + + - part: sections/methods-operators/index.qmd + chapters: + - sections/methods-operators/particle-mesh-interpolation/index.qmd + - sections/methods-operators/differential-operators/index.qmd + - sections/methods-operators/cg/index.qmd + - sections/methods-operators/fft/index.qmd + - file: sections/methods-operators/nufft/index.qmd + text: "NUFFT" + - sections/methods-operators/fem/index.qmd + + - part: sections/field-solvers/index.qmd + chapters: + - sections/field-solvers/poisson-solvers/index.qmd + - sections/field-solvers/maxwell-solvers/index.qmd + + - part: sections/analysis-performance/index.qmd + chapters: + - sections/analysis-performance/profiling/index.qmd + - sections/analysis-performance/in-situ/index.qmd + + - part: sections/tests/index.qmd + chapters: + - sections/tests/integration-tests/index.qmd + - sections/tests/unit-tests/index.qmd + + - part: sections/demos/index.qmd + chapters: + - sections/demos/alpine/index.qmd + - sections/demos/alpine-pif/index.qmd + - sections/demos/cosmology/index.qmd + - sections/demos/collisions/index.qmd + - sections/demos/fel/index.qmd + + - part: sections/for-developers/index.qmd + chapters: + - sections/for-developers/developer-guide/index.qmd + - sections/for-developers/workflow/index.qmd + - sections/for-developers/writing-unit-tests/index.qmd + - sections/for-developers/performance-portability/index.qmd + - sections/for-developers/api-reference/index.qmd + + - part: sections/appendix/index.qmd + chapters: + - sections/appendix/literature/index.qmd + - sections/appendix/playground/index.qmd number-sections: true number-depth: 3 @@ -39,6 +88,7 @@ format: html: toc: true toc-depth: 3 + # toc-location: right number-sections: true html-math-method: mathjax code-fold: false diff --git a/index.qmd b/index.qmd index be013a6..fbad779 100644 --- a/index.qmd +++ b/index.qmd @@ -1,8 +1,9 @@ --- -title: "IPPL Manual" +unnumbered: true number-sections: false --- +# Welcome {.unnumbered} ::: {.hero} ::: {} IPPL, the Independent Parallel Particle Layer, is a C++20 library for performance-portable particle, field, and particle-mesh simulations. It provides dimension-independent building blocks for Eulerian, Lagrangian, and hybrid methods on CPUs and GPUs. @@ -19,10 +20,11 @@ IPPL uses Kokkos for performance portability, MPI for distributed memory communi For users: -1. Start with @sec-overview and @sec-getting-started. +1. Start with @sec-overview and @sec-hello-ippl. 2. Read @sec-core-concepts, then the data model chapters on fields and particles. 3. Move to FFT, Poisson, Maxwell, or FEM depending on the application. -4. Use @sec-examples as executable documentation. +4. Use @sec-tests and @sec-demos as executable documentation. + For developers: diff --git a/sections/analysis-performance/in-situ/index.qmd b/sections/analysis-performance/in-situ/index.qmd new file mode 100644 index 0000000..a856b42 --- /dev/null +++ b/sections/analysis-performance/in-situ/index.qmd @@ -0,0 +1,12 @@ +--- +subtitle: "In situ analysis and visualization with ParaView Catalyst." +--- + +# In Situ {#sec-in-situ} + +coming soon! + +## Visualization + + +## Steering \ No newline at end of file diff --git a/sections/analysis-performance/index.qmd b/sections/analysis-performance/index.qmd new file mode 100644 index 0000000..7a47e66 --- /dev/null +++ b/sections/analysis-performance/index.qmd @@ -0,0 +1,14 @@ +--- +title: "Analysis & Performance" +subtitle: "Performance evaluation and in-situ analysis" +--- + +# Analysis & Performance {#sec-analysis-performance} + + +This chapter covers tools and techniques for analyzing the performance of IPPL applications and visualizing data in situ. + +- @sec-profiling: Profiling tools and memory events tracking. +- @sec-in-situ: In situ analysis and visualization with ParaView Catalyst. diff --git a/sections/analysis-performance/profiling/index.qmd b/sections/analysis-performance/profiling/index.qmd new file mode 100644 index 0000000..426ec00 --- /dev/null +++ b/sections/analysis-performance/profiling/index.qmd @@ -0,0 +1,284 @@ +--- +subtitle: "Profiling tools and memory events tracking." +--- + + + +# Profiling {#sec-profiling} + +## MPI Calls +You can use the mpiP tool (https://github.com/LLNL/mpiP) to get statistics about the MPI calls in IPPL. + +To use it, download it from [Github](https://github.com/LLNL/mpiP) and follow the instructions to install it. You may run into some issues while installing, here is a list of common issues and the solution: +- On Cray systems "MPI_Init not defined": This I fixed by passing the correct Cray wrappers for the compilers to the configure: `./configure CC=cc FC=ftn F77=ftn` +- If you have an issue with it not recognizing a function symbol in Fortran 77, you need to substitute the line `echo "main(){ FF(); return 0; }" > flink.c` (line 706) in the file `configure.ac` by the following line `echo "extern void FF(); int main() { FF(); return 0; }" > flink.c` +- During the `make all`, if you run into an issue of some Testing file not recognizing mpi.h, then you need to add the following line `CXX = CC` in the file `Testing/Makefile`. + +If the installation was successful, you should have the library `libmpip.so` in the mpiP directory. + +To instument your application with the mpiP library, add the following line to your jobscript (or run it in your command line if you are running locally/on an interactive node): +`export LD_PRELOAD=$[path to mpip directory]/libmpiP.so` +To pass any options to mpiP, you can export the variable MPIP with the options you want. For example, if you would like to get a histogram of the data sent by MPI calls (option `-y`), you would need to add the following line to your jobscript: +`export MPIP="-y"` + +If you application has been correctly instrumented, you will see that mpiP has been found and its version is printed at the top of the standard output. At the end of the standard output, you will get the name of the file containing the MPI statistics: +`Storing mpiP output in ...` + +To get a total amount of bytes moved around by your application, you can use the python script mpiP.py (found in the top level IPPL directory) in the following form: +`python3 mpiP.py [path/to/directory]` +where path/to/directory refers to the place where the .mpiP output can be found. This python script will then print out the total amount of Bytes moved by MPI in your application. + +## Profiling on LUMI + +### rocprof + +Analysis with: https://ui.perfetto.dev/ + +``` +#!/bin/bash -l +# +#SBATCH --job-name=opalx1 +#SBATCH --error=opalx-%j.error +#SBATCH --output=opalx-2-%j.out +#SBATCH --time=00:05:00 +#SBATCH --partition=standard-g +#SBATCH --nodes 1 +#SBATCH --ntasks-per-core=1 +#SBATCH -c 56 --threads-per-core=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --gpus-per-node=8 +#SBATCH --account=project_465001705 +#SBATCH --hint=nomultithread +#SBATCH --hint=exclusive +CPU_BIND="map_cpu:49,57,17,25,1,9,33,41" +export MPICH_GPU_SUPPORT_ENABLED=1 + +ulimit -s unlimited +export EXE_DIR=/users/adelmann/sandbox/opalx/build/src/ +module load cray-python/3.11.7 +module use /appl/local/containers/test-modules +module load LUMI/24.03 partition/G cpeAMD rocm/6.1.3 buildtools/24.03 + +cat << EOF > select_gpu +#!/bin/bash +export HIP_VISIBLE_DEVICES=\$SLURM_LOCALID +exec \$* +EOF +chmod +x ./select_gpu +srun ./select_gpu rocprof --hip-trace ${EXE_DIR}/opalx input.in --info 5 +rm -rf ./select_gpu + +``` + + +### omniperf (do not use omnitrace) + +doc url: https://rocm.docs.amd.com/projects/rocprofiler-compute/en/docs-6.2.4/how-to/profile/mode.html + +``` +#!/bin/bash -l +# +#SBATCH --job-name=opalx1 +#SBATCH --error=opalx-%j.error +#SBATCH --output=opalx-2-%j.out +#SBATCH --time=00:05:00 +#SBATCH --partition=standard-g +#SBATCH --nodes 1 +#SBATCH --ntasks-per-core=1 +#SBATCH -c 56 --threads-per-core=1 +#SBATCH --ntasks-per-node=1 +#SBATCH --gpus-per-node=8 +#SBATCH --account=project_465001705 +#SBATCH --hint=nomultithread +#SBATCH --hint=exclusive +CPU_BIND="map_cpu:49,57,17,25,1,9,33,41" +export MPICH_GPU_SUPPORT_ENABLED=1 + +ulimit -s unlimited +export EXE_DIR=/users/adelmann/sandbox/opalx/build/src/ +module load cray-python/3.11.7 +module use /appl/local/containers/test-modules +module load LUMI/24.03 partition/G cpeAMD rocm/6.1.3 buildtools/24.03 +module load omniperf +cat << EOF > select_gpu +#!/bin/bash +#export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID +export HIP_VISIBLE_DEVICES=\$SLURM_LOCALID +exec \$* +EOF +chmod +x ./select_gpu +srun ./select_gpu omniperf profile --name opalx --roof-only --kernel-names -- ${EXE_DIR}/opalx input.in --info 5 +rm -rf ./select_gpu +``` + + + + + + + + +## Kokkos Profiling Tools (MemoryEvents) +> **!Ported from old Doxygen, review required!** + +In certain applications, you might want to use profiling tools for debugging and testing. Since IPPL uses **Kokkos** as a backend, you can leverage Kokkos' built-in profiling tools. + +This guide explains how to use Kokkos' profiling tools, using the **MemoryEvents** tool as an example. + +### Description of MemoryEvents + +MemoryEvents tracks a timeline of allocation and deallocation events in Kokkos Memory Spaces. It records time, pointer, size, memory-space-name, and allocation-name. This is in particular useful for debugging purposes to understand where all the memory is going. + +Additionally, the tool provides a timeline of memory usage for each individual Kokkos Memory Space. + +The tool is located at: https://github.com/kokkos/kokkos-tools/tree/develop/profiling/memory-events + + +### 1. Clone the Kokkos Tools Repository + +First, clone the Kokkos tools repository, which contains a variety of profiling tools: +```bash +git clone https://github.com/kokkos/kokkos-tools +``` + +### 2. Build and Install the Tools + +Navigate into the repository and build the tools using CMake: + +```bash +cd kokkos-tools +cmake .. +make -j +sudo make install +``` + +### 3. Set Up the Profiling Tool + +Before running your application, export the Kokkos Tools environment variable to point to the `kp_memory_events.so` tool: +```bash +export KOKKOS_TOOLS_LIBS={PATH_TO_TOOL_DIRECTORY}/kp_memory_events.so +``` +Replace `{PATH_TO_TOOL_DIRECTORY}` with the actual path where the tool is located. + + +### 4. Run your Application + +Execute your application normally. The MemoryEvents tool will automatically collect data during execution. For example: + +```bash +./application COMMANDS +``` + +### 5. Output Files + +The MemoryEvents tool will generate the following files: + +- `HOSTNAME-PROCESSID.mem_events:` Lists memory events. +- `HOSTNAME-PROCESSID-MEMSPACE.memspace_usage:` Provides a utilization timeline for each active memory space. + +### 6. Example on with SLURM + +Here’s an example of how to run the profiling with a SLURM system using `sbatch`: +```bash +sbatch -n 2 --wrap="export KOKKOS_TOOLS_LIBS=$HOME/kokkos-tools/kp_memory_events.so; \ +mpirun -n 2 LandauDamping 128 128 128 10000 10 FFT 0.01 LeapFrog --overallocate 2.0 --info 10" +``` + +In this example: + +- `sbatch -n 2` specifies 2 nodes. +- The Kokkos tool is exported and applied to the `LandauDamping` application. + +This guide provides the basic steps for integrating Kokkos profiling tools into your IPPL-based projects. You can adjust the commands as needed depending on your specific application and environment. + +### Example + +Consider the following code: + +```cpp +#include + + typedef Kokkos::View a_type; + typedef Kokkos::View b_type; + typedef Kokkos::View c_type; + +int main() { + Kokkos::initialize(); + { + int N = 10000000; + for(int i =0; i<2; i++) { + a_type a("A",N); + { + b_type b("B",N); + c_type c("C",N); + for(int j =0; j +#sec-appendix > h1 { display: none; } + + +This chapter contains supplementary materials, literature references, and a playground for exploring IPPL. + +- @sec-literature: Literature references and bibliography. +- @sec-playground: Interactive browser playground for exploring IPPL features. diff --git a/sections/literature/index.qmd b/sections/appendix/literature/index.qmd similarity index 97% rename from sections/literature/index.qmd rename to sections/appendix/literature/index.qmd index cd59e13..8c694be 100644 --- a/sections/literature/index.qmd +++ b/sections/appendix/literature/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "Literature references and bibliography." +--- + # Literature {#sec-literature} This chapter collects IPPL-related papers, software records, and presentation material used by the manual. diff --git a/sections/playground/index.qmd b/sections/appendix/playground/index.qmd similarity index 98% rename from sections/playground/index.qmd rename to sections/appendix/playground/index.qmd index 927e916..2c645b6 100644 --- a/sections/playground/index.qmd +++ b/sections/appendix/playground/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "Interactive browser playground for exploring IPPL features." +--- + # Browser Playground {#sec-playground} The playground keeps the manual page open while compilation and execution happen in a local Docker container. The first example is deliberately small: edit a `Hello IPPL` program, compile it, run it with MPI, inspect stdout, and edit again. diff --git a/sections/demos/alpine-pif/index.qmd b/sections/demos/alpine-pif/index.qmd new file mode 100644 index 0000000..922d8fb --- /dev/null +++ b/sections/demos/alpine-pif/index.qmd @@ -0,0 +1,7 @@ +--- +subtitle: "Demonstration of the Particle in Fourier Method." +--- + +# ALPINE-PIF {#sec-demo-alpine-pif} + +coming soon! \ No newline at end of file diff --git a/sections/examples/alpine-manager-hierarchy.qmd b/sections/demos/alpine/index.qmd similarity index 99% rename from sections/examples/alpine-manager-hierarchy.qmd rename to sections/demos/alpine/index.qmd index f83bb21..602b71f 100644 --- a/sections/examples/alpine-manager-hierarchy.qmd +++ b/sections/demos/alpine/index.qmd @@ -1,10 +1,14 @@ -# ALPINE Manager Hierarchy {#sec-alpine-manager-hierarchy} +--- +subtitle: "The ALPINE application manager hierarchy and PIC mini applications." +--- + +# ALPINE {#sec-alpine-manager-hierarchy} ALPINE is IPPL's application layer: a collection of plasma-physics **mini-apps** in the `alpine/` directory that run full particle-in-cell (PIC) simulations end to end. They are not isolated library demos. Each mini-app wires together particles, fields, interpolation, Poisson solvers, MPI load balancing, and diagnostics the way a real simulation code would. This chapter explains what those mini-apps are for, how they are organized, and how the **manager hierarchy** keeps application logic separate from reusable PIC infrastructure. We use **Landau damping** as the main walkthrough because it is the smallest complete example; the same pattern applies to the other ALPINE applications. -For the underlying PIC model, see @sec-theory. For gather/scatter and load balancing, see @sec-particle-mesh. For solver options, see @sec-poisson-solvers and the runnable `LandauDamping` example in @sec-examples. +For the underlying PIC model, see @sec-theory. For gather/scatter and load balancing, see @sec-particle-mesh. For solver options, see @sec-poisson-solvers and the runnable `LandauDamping` example in @sec-tests. ## What ALPINE mini-apps are diff --git a/sections/demos/collisions/index.qmd b/sections/demos/collisions/index.qmd new file mode 100644 index 0000000..476bed8 --- /dev/null +++ b/sections/demos/collisions/index.qmd @@ -0,0 +1,7 @@ +--- +subtitle: "Approaches to integrate coulomb collisions into PIC." +--- + +# Collisions {#sec-demo-collisions} + +TODO: \ No newline at end of file diff --git a/sections/demos/cosmology/index.qmd b/sections/demos/cosmology/index.qmd new file mode 100644 index 0000000..bbb35ef --- /dev/null +++ b/sections/demos/cosmology/index.qmd @@ -0,0 +1,7 @@ +--- +subtitle: "Cosmology and structure formation." +--- + +# Cosmology {#sec-demo-cosmology} + +TODO: \ No newline at end of file diff --git a/sections/demos/fel/index.qmd b/sections/demos/fel/index.qmd new file mode 100644 index 0000000..81dcdeb --- /dev/null +++ b/sections/demos/fel/index.qmd @@ -0,0 +1,7 @@ +--- +subtitle: "Simulation of a Free-Electron-Laser Undulator." +--- + +# FEL {#sec-demo-fel} + +Coming soon! \ No newline at end of file diff --git a/sections/demos/index.qmd b/sections/demos/index.qmd new file mode 100644 index 0000000..b733f1e --- /dev/null +++ b/sections/demos/index.qmd @@ -0,0 +1,17 @@ +--- +title: "Demos" +subtitle: "Applications and Demos of IPPL" +--- + +# Demos {#sec-demos} + + +This chapter showcases various applications and demonstrations built using the IPPL framework. + +- @sec-alpine-manager-hierarchy: The ALPINE application manager hierarchy and PIC mini applications. +- @sec-demo-alpine-pif: Demonstration of the Particle in Fourier Method. +- @sec-demo-cosmology: Cosmology and structure formation. +- @sec-demo-collisions: Approaches to integrate coulomb collisions into PIC. +- @sec-demo-fel: Simulation of a Free-Electron-Laser Undulator. diff --git a/sections/field-solvers/index.qmd b/sections/field-solvers/index.qmd new file mode 100644 index 0000000..8630d25 --- /dev/null +++ b/sections/field-solvers/index.qmd @@ -0,0 +1,15 @@ +--- +title: "Field Solvers" +subtitle: "Solving Differential Equations on ippl fields" +--- + +# Field Solvers {#sec-field-solvers} + + + +This chapter explains how to configure and run field solvers on IPPL fields. + +- @sec-poisson-solvers: Solving Laplace, Poisson, and Biharmonic equations. +- @sec-maxwell-solvers: Solving Maxwell equations. \ No newline at end of file diff --git a/sections/maxwell-solvers/index.qmd b/sections/field-solvers/maxwell-solvers/index.qmd similarity index 99% rename from sections/maxwell-solvers/index.qmd rename to sections/field-solvers/maxwell-solvers/index.qmd index 7c3a4f6..bc13759 100644 --- a/sections/maxwell-solvers/index.qmd +++ b/sections/field-solvers/maxwell-solvers/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "Solving Maxwell equations." +--- + # Maxwell Solvers {#sec-maxwell-solvers} While Poisson solvers are appropriate for many use cases of \ac{pic} codes—for example for computation of Coulomb forces between particles in particle accelerator beams (known as space charge)—certain phenomena, such as laser interactions with plasma \cite{birdsallPlasmaPhysicsComputer2018}, are only captured with full electromagnetic simulations. This results in \ac{empic} and requires Maxwell field solvers [@mayani2026massivelyParallelPhd]. diff --git a/sections/poisson-solvers/index.qmd b/sections/field-solvers/poisson-solvers/index.qmd similarity index 99% rename from sections/poisson-solvers/index.qmd rename to sections/field-solvers/poisson-solvers/index.qmd index faca096..5a1fead 100644 --- a/sections/poisson-solvers/index.qmd +++ b/sections/field-solvers/poisson-solvers/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "Solving Laplace, Poisson, and Biharmonic equations." +--- + # Poisson Solvers {#sec-poisson-solvers} IPPL provides many different methods for solving the Poisson equation: diff --git a/sections/api-reference/index.qmd b/sections/for-developers/api-reference/index.qmd similarity index 93% rename from sections/api-reference/index.qmd rename to sections/for-developers/api-reference/index.qmd index c441a1f..4af3ab5 100644 --- a/sections/api-reference/index.qmd +++ b/sections/for-developers/api-reference/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "Detailed API reference documentation." +--- + # API Reference {#sec-api-reference} The manual should not duplicate every class signature. API details should be generated from Doxygen comments in the IPPL source tree. @@ -35,11 +39,11 @@ The manual should link to API pages for signatures and overloads, while keeping | Header | Manual chapter | API page | |---|---|---| -| `Ippl.h` | @sec-getting-started | [source](../../api/Ippl_8h_source.html) | -| `Utility/Inform.h` | @sec-getting-started, @sec-communication | [`Inform`](../../api/classInform.html) | -| `Utility/IpplInfo.h` | @sec-getting-started | [source](../../api/IpplInfo_8h_source.html) | +| `Ippl.h` | @sec-hello-ippl | [source](../../api/Ippl_8h_source.html) | +| `Utility/Inform.h` | @sec-hello-ippl, @sec-communication | [`Inform`](../../api/classInform.html) | +| `Utility/IpplInfo.h` | @sec-hello-ippl | [source](../../api/IpplInfo_8h_source.html) | | `Utility/IpplException.h` | @sec-developer-guide | [`IpplException`](../../api/classIpplException.html) | -| `Utility/IpplTimings.h` | @sec-examples, @sec-performance-portability | [`IpplTimings`](../../api/classIpplTimings.html) | +| `Utility/IpplTimings.h` | @sec-tests, @sec-performance-portability | [`IpplTimings`](../../api/classIpplTimings.html) | | `Index/Index.h`, `Index/NDIndex.h` | @sec-core-concepts | [`Index`](../../api/classippl_1_1Index.html), [`NDIndex`](../../api/classippl_1_1NDIndex.html) | | `Region/NDRegion.h`, `Region/PRegion.h` | @sec-core-concepts, @sec-particles | [`NDRegion`](../../api/classippl_1_1NDRegion.html), [`PRegion`](../../api/classippl_1_1PRegion.html) | | `Meshes/UniformCartesian.h` | @sec-core-concepts, @sec-fields | [`UniformCartesian`](../../api/classippl_1_1UniformCartesian.html) | diff --git a/sections/developer-guide/index.qmd b/sections/for-developers/developer-guide/index.qmd similarity index 85% rename from sections/developer-guide/index.qmd rename to sections/for-developers/developer-guide/index.qmd index 8626bfa..2ca5832 100644 --- a/sections/developer-guide/index.qmd +++ b/sections/for-developers/developer-guide/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "General guidelines for extending IPPL." +--- + # Developer Guide {#sec-developer-guide} The developer guide documents how to extend IPPL without breaking portability, correctness, or downstream applications. @@ -92,3 +96,21 @@ Tests should cover both the numerical contract and the distributed-memory contra | FFT and Poisson | Forward/backward consistency, manufactured solution error, backend options. | | FEM and Maxwell | Basis correctness, DOF mapping, residuals, convergence or error norms. | | Communication | Blocking/nonblocking exchange, collectives, serialization, tag isolation. | + + +## Debugging apps with gdb +> **!Ported from old Doxygen, review required!** + +### LandauDamping +Some applications (LandauDamping currently) accept a command line option `--debug`, this causes them to pause on startup and allows you to attach a debugger and then resume (by pressing ). This is useful when debugging N mpi ranks and you want to attach gdb to one or more of them, but can't launch using gdb directly. + +Usually one would attach gdb using a launch command such as this (on a remote machine supporting xterm over X11 for example, to open N xterm windows with gdb attached to one rank in each) +```bash +srun -N 1 -n 2 wrapper-script.sh xterm -fn 10x20 -e gdb \ + -ex='set confirm on' \ + -ex=record \ + -ex=run \ + -ex="set pagination off" \ + --args my/path/to/executable +``` +The extra `-ex` parameters tell gdb to launch the application and start executing and not wait for additional prompting. diff --git a/sections/for-developers/index.qmd b/sections/for-developers/index.qmd new file mode 100644 index 0000000..2002d27 --- /dev/null +++ b/sections/for-developers/index.qmd @@ -0,0 +1,18 @@ +--- +title: "For Developers" +subtitle: "Resources for developers contributing to IPPL" +--- + +# For Developers {#sec-for-developers} + + + +This chapter contains information and guidelines for developers contributing to IPPL. + +- @sec-developer-guide: General guidelines for extending IPPL. +- @sec-workflow: Development workflow and tools. +- @sec-writing-unit-tests: Implementation and writing of unit tests. +- @sec-performance-portability: Strategies for achieving performance portability. +- @sec-api-reference: Detailed API reference documentation. \ No newline at end of file diff --git a/sections/performance-portability/index.qmd b/sections/for-developers/performance-portability/index.qmd similarity index 98% rename from sections/performance-portability/index.qmd rename to sections/for-developers/performance-portability/index.qmd index 1237aea..abb6755 100644 --- a/sections/performance-portability/index.qmd +++ b/sections/for-developers/performance-portability/index.qmd @@ -1,3 +1,7 @@ +--- +subtitle: "Strategies for achieving performance portability." +--- + # Performance Portability {#sec-performance-portability} IPPL's portability model is based on Kokkos execution and memory spaces, MPI for distributed memory, and optional HeFFTe FFT backends. The same algorithmic code path is intended to run across CPU and GPU systems with only compile-time type selection and runtime parameter choices. diff --git a/sections/for-developers/workflow/index.qmd b/sections/for-developers/workflow/index.qmd new file mode 100644 index 0000000..faf9fb4 --- /dev/null +++ b/sections/for-developers/workflow/index.qmd @@ -0,0 +1,124 @@ +--- +subtitle: "Development workflow and tools." +--- + +# Workflow {#sec-workflow} + +## Naming Conventions + +- Variables should use camel casing +- Compile time constants should use capital casing +- Member variables should be suffixed with `_m` + +## Math + +Mathematical constants should be obtained from `Kokkos::numbers` (moved out of `experimental` in Kokkos 4). Any instance of a mathematical function in host-only code should use symbols from the standard library, e.g. `std::sqrt`. Any instances occurring in device-code or code that might be run on a device (such as those marked with `KOKKOS_INLINE_FUNCTION`) should use the symbols from Kokkos, e.g. `Kokkos::sqrt`, to ensure performance and portability on GPUs. + +## Other Formatting Conventions + +IPPL uses `clang-format` for automatic formatting. For explanations of the available settings, refer to the [clang documentation](https://clang.llvm.org/docs/ClangFormatStyleOptions.html). +IPPL's settings are available in the repository root. For more information about the formatting, refer to the workflow below. + +## Automatic Code Formatting + +IPPL uses `clang-format` to automatically format the codebase to maintain a consistent code style. When modifying IPPL, the changes can be reformatted automatically by +using a *git hook*. A git hook is a script that is executed by `git` before completing certain actions. The `hooks/` top-level directory includes a `pre-commit` script and a +shell script for setting up the git hook. Git hooks are stored in `.git/hooks`; the shell script creates a symbolic link in `.git/hooks` pointing to the `pre-commit` script, +effectively creating a pre-commit hook for the local repository. A pre-commit hook is run automatically before every commit. + +The provided script automatically runs `clang-format` on all files with *staged* changes, i.e. files that have been added to the commit, then re-stages these files to include +the reformatted changes in the commit. The result is that, even if the code is not properly formatted when you run `git commit`, the code is automatically reformatted before the +commit is created, so all committed code will always have the proper formatting. + + +## Installing Formatters +> **!Ported from old Doxygen, review required!** + +### 1. Installing clang-format-21 + +#### Ubuntu/Debian +Add the LLVM repository and install clang-format-21: +```bash +# Add LLVM repository (Ubuntu 20.04+, Debian 11+) +wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add - +sudo apt-add-repository "deb http://apt.llvm.org/$(lsb_release -cs)/ llvm-toolchain-$(lsb_release -cs)-21 main" +sudo apt update + +# Install clang-format-21 +sudo apt install clang-format-21 +``` +Verify the installation: +```bash +clang-format-21 --version +``` +Note: If you have `clang-format` installed but it doesn't use the full name `clang-format-21`, then just create a symlink in your `.local` path that redirects `clang-format-21` to your real `clang-format` executable + +#### APPLE macOS Tahoe +```bash +brew install llvm@21 +clang-format --version +``` + +#### Other Systems +For other operating systems, visit the [LLVM download page](https://releases.llvm.org/) or use your system's package manager to locate `clang-format` version 21 or later. + +### 2. Installing cmake-format +cmake-format is a Python package available via pip: +```bash +# Install cmake-format (Python 3.x required) +pip install cmake-format +``` +Verify the installation: +```bash +cmake-format --version +``` + +## Troubleshooting + +### Hook not executing? +- Verify the script is executable: `chmod +x .git/hooks/pre-commit` +- Verify the symlink points to the correct location: `ls -la .git/hooks/pre-commit` +- Ensure `clang-format-21` and `cmake-format` are in your `$PATH`: `which clang-format-21 cmake-format` + +### Formatting succeeds locally but hook still rejects? +This is typically due to: +- Different `clang-format-21` versions installed (ensure you have version 21) +- Nested `.clang-format` configuration files in subdirectories that override the root config + +### Path issues on HPC systems? +If you're on a shared HPC system where multiple compiler modules are available: +- Load the correct module before working: `module load llvm/21` or similar +- Create a symlink to `clang-format` in `~/.local/bin` if module loading is inconvenient + +## Caveats + +- `clang-format` must be available to use the automatic formatting; Beware of the order in which modules are loaded, as the presence of `clang` can affect project configuration. + If you newly configure a project and `clang` has higher priority in your `$PATH` than `gcc` (or another desired compiler), the project may become configured to compile with clang + instead of your desired compiler. In general, it is advisable to only load the clang module once the project is configured, or to load clang before your desired compiler such + that it has lower priority in your `$PATH`. +- To avoid misconfigurations due to the Clang module, a simple workaround is to create a symbolic link to `clang-format` in a folder that is always in your `$PATH`. + To locate `clang-format`, run `which clang-format` while the module is loaded. Create a symbolic link with `ln -s "$(which clang-format)" /path/to/link`, where `/path/to` is + included in your `$PATH`. Conventional paths include `~/.local/bin` (this folder may need to be created). If this is not already in your `$PATH`, you can modify + your `~/.bash_profile` to add this to your path: `PATH=$PATH:~/.local/bin`. +- The pre-commit script always reformats and re-stages *entire* files. If you used `git add -i` to only stage a subset of the changes in a file, the pre-commit script will + render the interactive staging moot by staging the entire file. To prevent this, use the `--no-verify` flag when committing; this makes `git` skip the pre-commit script. Note that + this means that your changes will not be reformatted before committing. If you would like to commit only a subset of the changes made to a file but still want to run the formatter, + you can stash the changes you don't want to commit by stashing in patch mode (`git stash save -p`) and shelving the other changes. After committing, run `git stash pop` to restore + the uncommitted changes. +- The pre-commit script only reformats *staged* files. It does not affect unstaged or unmodified files. To reformat unchanged files, you will have to run the formatter manually. + This is also true if you stage a subset of modified files; the staged files will be reformatted, but unstaged files will not be reformatted even if they have modifications. + +Refer to our git introduction if the git-related terms on this page are unclear. + +# Git brief introduction + +For those in need of a refresher or an introduction, here is a glossary of Git terms relevant to the workflow caveats, as well as some useful commands and flags: + +- A file is *tracked* if it is under version control. A file that has never been committed is *untracked* and will show up under "untracked files" when running `git status`. +- Files whose names match patterns in a `.gitignore` file will be ignored by git and do not show up as untracked. To start tracking an ignored file, use the `-f` flag when staging it. +- The *working tree* is the current state of the repository, excluding untracked files. Changes to the working tree are shown by `git diff`. +- Before committing, changes have to be *staged*. This is done by running `git add` to stage the changes that need to be committed. You can view the staged changes using `git diff --staged`. This represents the changes that will be recorded by running `git commit`. +- You can stage any subset of the changes to the working tree. You can also stage only a subset of changes to a file by using *interactive staging* with `git add -i` and choosing "patch" at the interactive prompt. +- A *commit* is a set of changes that constitute a fixed point in the repository history. Running `git commit` creates a new commit out of the staged changes. Unstaged changes are untouched. + + diff --git a/sections/for-developers/writing-unit-tests/index.qmd b/sections/for-developers/writing-unit-tests/index.qmd new file mode 100644 index 0000000..ceeb1d2 --- /dev/null +++ b/sections/for-developers/writing-unit-tests/index.qmd @@ -0,0 +1,63 @@ +--- +subtitle: "Implementation and writing of unit tests." +--- + +# Writing Unit Tests {#sec-writing-unit-tests} + +The generation of test configurations is implemented in `unit_tests/TestUtils.h`. It is controlled using the following struct: + +```cpp +template +struct Parameters { ... }; + +struct TestParams { + using Spaces = ippl::detail::TypeForAllSpaces::exec_spaces_type; + using Precisions = std::tuple; + using Combos = CreateCombinations::type; + + template + using Ranks = std::tuple...>; + + template + using CombosWithRanks = typename CreateCombinations>::type; + + template + using tests = typename TestForTypes< + std::conditional_t>>::type; +... +}; +``` + +First, we generate a tuple containing all the available execution spacs (see `Utility/TypeUtils.h`). We then create tuples with the other parameters we want to test. The `CreateCombinations` struct recursively generates all combinations of the chosen parameters at compile time and instantiates the `Parameter` type to hold these combinations. We then use GoogleTest's `testing::Types<...>` to instantiate all the unit tests for each combination. Example: + +```cpp +template +class FieldTest; + +template +class FieldTest>> : public ::testing::Test { ... }; + +using Tests = TestParams::tests<1, 2, 3, 4, 5, 6>; +TYPED_TEST_CASE(FieldTest, Tests); + +TYPED_TEST(FieldTest, DeepCopy) { ... } +``` + +If we want to generate combinations beyond just ranks/precision/execution space like above, we can use the tuple generation with other types. For example, the particle bunch tests use two execution spaces: + +```cpp +template +class ParticleBaseTest; + +template +class ParticleBaseTest>> : public ::testing::Test { ... }; + +using Precisions = TestParams::Precisions; +using Spaces = TestParams::Spaces; +using Ranks = TestParams::Ranks<1, 2, 3, 4, 5, 6>; +using Combos = CreateCombinations::type; +using Tests = TestForTypes::type; +TYPED_TEST_CASE(ParticleBaseTest, Tests); + +TYPED_TEST(ParticleBaseTest, CreateAndDestroy) { ... } +``` diff --git a/sections/getting-started/hello-ippl/index.qmd b/sections/getting-started/hello-ippl/index.qmd new file mode 100644 index 0000000..9b40892 --- /dev/null +++ b/sections/getting-started/hello-ippl/index.qmd @@ -0,0 +1,150 @@ +--- +subtitle: "A simple Hello World example." +--- + +# Hello IPPL {#sec-hello-ippl} + + +IPPL is configured with CMake. A minimal build needs a C++20 compiler, MPI, and Kokkos. FFT and solver chapters require builds with `IPPL_ENABLE_FFT=ON` and usually `IPPL_ENABLE_SOLVERS=ON`, but the first program does not need that stack. + +## Minimal program + +```cpp +#include "Ippl.h" + +int main(int argc, char* argv[]) { + ippl::initialize(argc, argv); + + *ippl::Info << "Hello from rank " + << ippl::Comm->rank() + << " of " + << ippl::Comm->size() + << ippl::endl; + + ippl::finalize(); + return 0; +} +``` + +## Compile a small IPPL program + +A minimal standalone program can be developed as a small CMake project. During early example development, the most robust pattern is to consume IPPL as a CMake subproject so that the example uses the same dependency configuration as the IPPL source tree. + +Create `hello_ippl.cpp`: + +```cpp +#include "Ippl.h" + +int main(int argc, char* argv[]) { + ippl::initialize(argc, argv); + + *ippl::Info << "Hello from rank " + << ippl::Comm->rank() + << " of " + << ippl::Comm->size() + << ippl::endl; + + ippl::finalize(); + return 0; +} +``` + +Create `CMakeLists.txt` next to it: + +```cmake +cmake_minimum_required(VERSION 3.24) + +project(ippl_hello LANGUAGES CXX) + +set(IPPL_PLATFORMS "SERIAL" CACHE STRING "" FORCE) +set(IPPL_ENABLE_FFT OFF CACHE BOOL "" FORCE) +set(IPPL_ENABLE_SOLVERS OFF CACHE BOOL "" FORCE) +set(IPPL_ENABLE_TESTS OFF CACHE BOOL "" FORCE) +set(IPPL_ENABLE_UNIT_TESTS OFF CACHE BOOL "" FORCE) +set(Kokkos_VERSION "git.4.7.02" CACHE STRING "" FORCE) +set(IPPL_SOURCE_DIR "/workspaces/ippl" CACHE PATH "IPPL source tree") + +add_subdirectory("${IPPL_SOURCE_DIR}" ippl-build) + +add_executable(hello_ippl hello_ippl.cpp) +target_link_libraries(hello_ippl PRIVATE IPPL::ippl) +``` + +Configure, build, and run it inside the devcontainer: + +```bash +cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER=mpicxx +cmake --build build --target hello_ippl --parallel 2 +mpiexec --oversubscribe -n 2 build/hello_ippl +``` + +Expected output: + +```text +Hello from rank 0 of 2 +Hello from rank 1 of 2 +``` + +For installed IPPL packages, the `CMakeLists.txt` becomes shorter: + +```cmake +cmake_minimum_required(VERSION 3.24) + +project(ippl_hello LANGUAGES CXX) + +find_package(IPPL REQUIRED CONFIG) + +add_executable(hello_ippl hello_ippl.cpp) +target_link_libraries(hello_ippl PRIVATE IPPL::ippl) +``` + +Use the same compiler/MPI stack that was used to build IPPL: + +```bash +cmake -S . -B build -G Ninja \ + -DCMAKE_CXX_COMPILER=mpicxx \ + -DCMAKE_PREFIX_PATH=/path/to/ippl/install +cmake --build build +mpiexec -n 2 build/hello_ippl +``` + +Do not mix a host compiler, a different MPI installation, or a different Kokkos installation with an existing IPPL build. + +## Common configure modes + +```bash +cmake -S /path/to/ippl -B build-serial \ + -DCMAKE_BUILD_TYPE=RelWithDebInfo \ + -DCMAKE_CXX_STANDARD=20 \ + -DIPPL_PLATFORMS=SERIAL +``` + +```bash +cmake -S /path/to/ippl -B build-openmp \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_CXX_STANDARD=20 \ + -DIPPL_PLATFORMS=OPENMP \ + -DIPPL_ENABLE_FFT=OFF \ + -DIPPL_ENABLE_SOLVERS=OFF \ + -DIPPL_ENABLE_TESTS=ON +``` + +```bash +cmake -S /path/to/ippl -B build-cuda \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_CXX_STANDARD=20 \ + -DIPPL_PLATFORMS=CUDA \ + -DKokkos_ARCH_AMPERE80=ON \ + -DCMAKE_CUDA_ARCHITECTURES=80 +``` + +## Build options to document first + +| Option | Meaning | +|---|---| +| `IPPL_PLATFORMS` | Kokkos backends requested by IPPL, such as `SERIAL`, `OPENMP`, `CUDA`, or `HIP`. | +| `IPPL_ENABLE_FFT` | Enables FFT support and HeFFTe dependency logic. | +| `IPPL_ENABLE_SOLVERS` | Enables solver components that depend on FFT or linear solver infrastructure. | +| `IPPL_ENABLE_TESTS` | Enables integration tests and examples used as documentation. | +| `IPPL_ENABLE_UNIT_TESTS` | Enables GoogleTest-based unit tests. | +| `IPPL_USE_ALTERNATIVE_VARIANT` | Compatibility path for some CUDA/GCC combinations. | diff --git a/sections/getting-started/index.qmd b/sections/getting-started/index.qmd index 86ed2ee..aa2ee83 100644 --- a/sections/getting-started/index.qmd +++ b/sections/getting-started/index.qmd @@ -1,145 +1,16 @@ -# Getting Started {#sec-getting-started} - -IPPL is configured with CMake. A minimal build needs a C++20 compiler, MPI, and Kokkos. FFT and solver chapters require builds with `IPPL_ENABLE_FFT=ON` and usually `IPPL_ENABLE_SOLVERS=ON`, but the first program does not need that stack. - -## Minimal program - -```cpp -#include "Ippl.h" - -int main(int argc, char* argv[]) { - ippl::initialize(argc, argv); - - *ippl::Info << "Hello from rank " - << ippl::Comm->rank() - << " of " - << ippl::Comm->size() - << ippl::endl; - - ippl::finalize(); - return 0; -} -``` - -## Compile a small IPPL program - -A minimal standalone program can be developed as a small CMake project. During early example development, the most robust pattern is to consume IPPL as a CMake subproject so that the example uses the same dependency configuration as the IPPL source tree. - -Create `hello_ippl.cpp`: - -```cpp -#include "Ippl.h" - -int main(int argc, char* argv[]) { - ippl::initialize(argc, argv); - - *ippl::Info << "Hello from rank " - << ippl::Comm->rank() - << " of " - << ippl::Comm->size() - << ippl::endl; - - ippl::finalize(); - return 0; -} -``` - -Create `CMakeLists.txt` next to it: - -```cmake -cmake_minimum_required(VERSION 3.24) - -project(ippl_hello LANGUAGES CXX) - -set(IPPL_PLATFORMS "SERIAL" CACHE STRING "" FORCE) -set(IPPL_ENABLE_FFT OFF CACHE BOOL "" FORCE) -set(IPPL_ENABLE_SOLVERS OFF CACHE BOOL "" FORCE) -set(IPPL_ENABLE_TESTS OFF CACHE BOOL "" FORCE) -set(IPPL_ENABLE_UNIT_TESTS OFF CACHE BOOL "" FORCE) -set(Kokkos_VERSION "git.4.7.02" CACHE STRING "" FORCE) -set(IPPL_SOURCE_DIR "/workspaces/ippl" CACHE PATH "IPPL source tree") +--- +title: "Getting Started" +subtitle: "Getting Started with IPPL" +--- -add_subdirectory("${IPPL_SOURCE_DIR}" ippl-build) - -add_executable(hello_ippl hello_ippl.cpp) -target_link_libraries(hello_ippl PRIVATE IPPL::ippl) -``` - -Configure, build, and run it inside the devcontainer: - -```bash -cmake -S . -B build -G Ninja -DCMAKE_CXX_COMPILER=mpicxx -cmake --build build --target hello_ippl --parallel 2 -mpiexec --oversubscribe -n 2 build/hello_ippl -``` - -Expected output: - -```text -Hello from rank 0 of 2 -Hello from rank 1 of 2 -``` - -For installed IPPL packages, the `CMakeLists.txt` becomes shorter: - -```cmake -cmake_minimum_required(VERSION 3.24) - -project(ippl_hello LANGUAGES CXX) - -find_package(IPPL REQUIRED CONFIG) - -add_executable(hello_ippl hello_ippl.cpp) -target_link_libraries(hello_ippl PRIVATE IPPL::ippl) -``` - -Use the same compiler/MPI stack that was used to build IPPL: - -```bash -cmake -S . -B build -G Ninja \ - -DCMAKE_CXX_COMPILER=mpicxx \ - -DCMAKE_PREFIX_PATH=/path/to/ippl/install -cmake --build build -mpiexec -n 2 build/hello_ippl -``` - -Do not mix a host compiler, a different MPI installation, or a different Kokkos installation with an existing IPPL build. - -## Common configure modes - -```bash -cmake -S /path/to/ippl -B build-serial \ - -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DCMAKE_CXX_STANDARD=20 \ - -DIPPL_PLATFORMS=SERIAL -``` - -```bash -cmake -S /path/to/ippl -B build-openmp \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_CXX_STANDARD=20 \ - -DIPPL_PLATFORMS=OPENMP \ - -DIPPL_ENABLE_FFT=OFF \ - -DIPPL_ENABLE_SOLVERS=OFF \ - -DIPPL_ENABLE_TESTS=ON -``` +# Getting Started {#sec-getting-started} + -```bash -cmake -S /path/to/ippl -B build-cuda \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_CXX_STANDARD=20 \ - -DIPPL_PLATFORMS=CUDA \ - -DKokkos_ARCH_AMPERE80=ON \ - -DCMAKE_CUDA_ARCHITECTURES=80 -``` -## Build options to document first +This chapter helps you set up and run your first IPPL programs. -| Option | Meaning | -|---|---| -| `IPPL_PLATFORMS` | Kokkos backends requested by IPPL, such as `SERIAL`, `OPENMP`, `CUDA`, or `HIP`. | -| `IPPL_ENABLE_FFT` | Enables FFT support and HeFFTe dependency logic. | -| `IPPL_ENABLE_SOLVERS` | Enables solver components that depend on FFT or linear solver infrastructure. | -| `IPPL_ENABLE_TESTS` | Enables integration tests and examples used as documentation. | -| `IPPL_ENABLE_UNIT_TESTS` | Enables GoogleTest-based unit tests. | -| `IPPL_USE_ALTERNATIVE_VARIANT` | Compatibility path for some CUDA/GCC combinations. | +- @sec-installation: Instructions for building and installing IPPL. +- @sec-hello-ippl: A simple Hello World example. +- @sec-slurm: Running IPPL jobs on HPC systems with SLURM. \ No newline at end of file diff --git a/sections/getting-started/installation/index.qmd b/sections/getting-started/installation/index.qmd new file mode 100644 index 0000000..77c17a0 --- /dev/null +++ b/sections/getting-started/installation/index.qmd @@ -0,0 +1,340 @@ +--- +subtitle: "Instructions for building and installing IPPL." +--- + +# Installation {#sec-installation} +All the new developments of IPPL are merged into the `master` branch which can make it potentially unstable from time to time. So if you want a stable and more tested version +please checkout the tagged branch corresponding to the latest release (e.g. `git checkout tags/IPPL-x.x.x`). Otherwise if you want the latest developments go with the master with the above caveat in mind. + +## Requirements + +* [CMake](https://cmake.org/download/) +* A C++ compilation toolchain (GPU-capable for GPU builds, e.g. [nvcc](https://developer.nvidia.com/cuda-downloads), [clang]() or [rocmcc](https://rocm.docs.amd.com/en/docs-5.0.2/reference/rocmcc/rocmcc.html)) +* MPI (GPU-aware if building for GPUs) + +### Optional requirements +* FFTW +* CuFFT + +## Compilation +IPPL is a CMake Project and can be configured by passing options in CMake syntax: +``` +cmake -D