diff --git "a/CI\346\217\220\351\200\237\346\226\271\346\241\210.md" "b/CI\346\217\220\351\200\237\346\226\271\346\241\210.md" new file mode 100644 index 0000000000..2d643ea937 --- /dev/null +++ "b/CI\346\217\220\351\200\237\346\226\271\346\241\210.md" @@ -0,0 +1,307 @@ +# ABACUS CI/CD 提速方案(可落地版) + +> 配套文档:《CI速度差异分析.md》(同目录,含全部实测数据) +> 原则:每个方案给出**具体文件改动 + 预期收益 + 风险 + 验证方法**,可直接拆成 PR。 +> 已验证的关键事实: +> - ccache 已通过 `CMakeLists.txt:299-303` 接入全部语言(含 CUDA),装了就用,无需改 CMake +> - CMake 只用 `git log -1` 取 commit 信息(`CMakeLists.txt:163-174`),**不需要 fetch-depth: 0** +> - `Autotest.sh -n` 是每个用例的 MPI 进程数,不是用例并行数 +> - 上游 cuda.yml 仍是 `-j4` 单 job;你的 f3e844ff7(nproc+matrix)只在本地/已关闭的 #7713 + +--- + +## 方案总览(按 ROI 排序) + +| 方案 | 改动量 | 预期收益 | 风险 | 建议批次 | +|---|---|---|---|---| +| B. CUDA 编译提速 | cuda.yml 2 行 | Build 33min → 8~15min | 极低 | **PR-1(立即)** | +| D3. checkout 浅克隆 | test.yml 1 行 | 冷节点省 ~12min | 极低 | **PR-1(立即)** | +| C. ccache 可靠化 | 2 个 yml 数行 + 1 个新 yml | 消除 2.7 vs 37.5min 波动 | 低 | PR-2 | +| A. 镜像稳定化 | 2 个 Dockerfile 删段落 | init 8~78min → ~1min | 中(需维护者共识) | PR-3 | +| E. toolchain 归零 | 随 A 一起做 | 冷节点省 3~10min | 低 | 随 PR-3 | +| D. 测试并行化 | test.yml 重构 / cuda.yml | CPU 18min→~6min | 中 | PR-4 | + +预期总效果:**test.yml 24~100min → 12~20min;cuda.yml 53~131min → 15~25min。** + +--- + +## 方案 B:CUDA 编译提速(PR-1,最先做) + +### B1. 裁剪 CUDA 架构(7 种 → 1~2 种) + +**问题**:`CMakeLists.txt:445-476` 未指定时,CUDA 12.2 下默认编 60/70/75/80/86/89/90 共 7 种架构,61 个 `.cu` 每个编 7 遍。CI 只需要能在 CI 的那块 GPU 上跑。 + +**改动**(`.github/workflows/cuda.yml` Configure & Build 步骤): + +```yaml + - name: Configure & Build + run: | + nvidia-smi + source toolchain/install/setup + rm -rf build + cmake -B build -G Ninja \ + -DUSE_CUDA=ON \ + -DBUILD_TESTING=ON \ + -DENABLE_FLOAT_FFTW=ON \ + -DCMAKE_CUDA_ARCHITECTURES=80 \ # ← 新增:按 CI 实际 GPU 填 + -DCMAKE_INSTALL_PREFIX="${GITHUB_WORKSPACE}/install" + cmake --build build -j $(nproc) # ← B2:-j4 改掉 + cmake --install build +``` + +**架构怎么填**:打开最近一次 CUDA run 的 Configure & Build 日志,第一行 `nvidia-smi` 输出就是卡型: + +| GPU | Compute Capability | 填 | +|---|---|---| +| V100 | 7.0 | `70` | +| T4 | 7.5 | `75` | +| A100 | 8.0 | `80` | +| A10 / RTX 30 系 | 8.6 | `86` | +| L40 / RTX 40 系 | 8.9 | `89` | +| H100 | 9.0 | `90` | + +池子里若有多种卡,填 2 个(如 `80;90`),仍比 7 种省 ~70% nvcc 工作量。**不要**用 `native`——你本地 matrix 版把 build 和 test 拆成了不同 job,可能落在不同型号的机器上。 + +**预期收益**:33min → 8~15min(-70%~-85% nvcc 时间)。测试有效性不受影响(二进制只需在 CI GPU 上可运行)。 + +### B2. 放开并行度 `-j4` → `-j $(nproc)` + +**问题**:`-j4` 自 #4032 时代写死。GPU 节点通常 16+ 核。 + +**注意**:必须和 B1 一起做。7 架构 × 高并行 → nvcc 单进程 2~4GB 内存,容易 OOM(你之前关心的 OOM 问题就在这里);架构裁到 1~2 种后放开并行是安全的。你本地 commit f3e844ff7 已包含此项,可直接摘出来。 + +--- + +## 方案 D3:checkout 浅克隆(并入 PR-1) + +**问题**:test.yml `fetch-depth: 0` 全历史克隆,冷节点实测 11.9min。已验证 CMake 只执行 `git log -1`,不需要历史。 + +**改动**(`.github/workflows/test.yml`): + +```yaml + - name: Checkout repository + uses: actions/checkout@v7 + with: + fetch-depth: 1 # ← 0 改 1 + submodules: 'false' +``` + +**风险排查**:`git submodule update --init --recursive` 不受 depth 影响;`version_check.yml` 是独立 workflow,如它依赖 tag 历史则保持它自己的 depth 不变(与本 PR 无关)。 + +--- + +## 方案 C:ccache 可靠化(PR-2,消除最大波动源) + +### C1. 显式容量 + 可观测(改动极小,先做) + +**问题**:两个 workflow 只挂载 `/tmp/ccache`,从未设置容量(ccache 4.x 默认约 10GB,7 架构 CUDA 产物 + 日常 churn 有驱逐压力),也从不打印命中率——慢的时候无法自证。 + +**改动**(test.yml 和 cuda.yml 的 Build 步骤前后各加一步): + +```yaml + - name: Setup ccache + run: | + ccache --max-size=30G + ccache --zero-stats + ccache -s + + # ... Configure / Build ... + + - name: ccache statistics + if: always() + run: ccache -s +``` + +**收益**:①容量兜底;②每次运行的日志里直接看到 `cache hit rate`,下次再出现"2min vs 37min"一眼定位是缓存问题还是机器问题。这也是给维护者讲故事的证据。 + +### C2. 确认动态池的缓存持久化(需要 runner 管理员配合) + +**问题**:6/17 #7476 切到动态 runner 池(K8s Pod),`/tmp/ccache` 是宿主机目录还是 emptyDir 决定了缓存能不能跨 run 存活。实测同一 PR 越推越快,说明**部分节点**是能保住的,但新扩的节点是冷的。 + +**行动**(在 issue/PR 里 @ 维护者确认两件事): +1. `/tmp/ccache` 是否 hostPath/PV?多节点池是否每节点独立一份?(独立 → 命中率随节点数稀释) +2. 若是 emptyDir,改为 hostPath 或 PVC; +3. 终极方案:ccache 4.x 支持二级远程缓存,池内共享一份: + ```yaml + env: + CCACHE_REMOTE_STORAGE: "http://内网缓存服务|layout=bazel" # 或 redis:/s3 + CCACHE_REMOTE_ONLY: "false" # 本地+远程双层 + ``` + 这样新扩的冷节点也能命中远程缓存,**直接消除"节点运气"因子**。 + +### C3. cache-warmer:给池子预热(消除"合并风暴后第一批 PR 变慢") + +**问题**:大重构合入 develop 后,全池缓存失效,之后第一批 PR 的 CI 全部变慢(7/29-31 的 347 文件/125 头文件风暴就是案例)。 + +**方案**:新增 `.github/workflows/cache_warmer.yml`,在 develop 有合并时自动"只编不测",让池子缓存始终贴着最新 develop: + +```yaml +name: Cache Warmer +on: + push: + branches: [develop] + workflow_dispatch: + +concurrency: + group: cache-warmer + cancel-in-progress: true + +jobs: + warm: + runs-on: X64 + if: github.repository_owner == 'deepmodeling' + container: + image: ghcr.io/deepmodeling/abacus-gnu # 方案 A 落地后换稳定 tag + volumes: + - /tmp/ccache:/github/home/.ccache + steps: + - uses: actions/checkout@v7 + with: + fetch-depth: 1 + submodules: recursive + - name: Build only (prime ccache) + run: | + sudo apt-get update && sudo apt-get install -y ccache ninja-build gfortran + cmake -B build -G Ninja -DBUILD_TESTING=ON + cmake --build build -j $(nproc) + ccache -s +``` + +GPU 池同理可加一条 `runs-on: gpu` 的 warm job(USE_CUDA=ON)。成本是 runner 时间,收益是所有 PR 的 Build 稳定在"只差 PR 自己改动"的水平。 + +--- + +## 方案 A:镜像稳定化(PR-3,收益最大,需维护者共识) + +### 问题回顾 + +- `devcontainer.yml`:**每次 push 到 develop** 都重建并推送 `abacus-{gnu,intel,cuda}:latest`; +- `Dockerfile.gnu:31-41` / `Dockerfile.cuda:27-36`:cache-bust(`ADD .../develop /dev/null`)+ **在镜像里完整编译一份 ABACUS**; +- 结果:镜像几乎天天变 → 动态 runner 天天重拉(init 实测 8~78min); +- 而 CI 从不用镜像里那份二进制(每个 PR 都 checkout 自己重编)——**镜像内编译对 CI 是纯浪费**。 + +### 改法 A1(推荐:删,不加新东西,最容易过审) + +从 `Dockerfile.gnu` 删 31-41 行、`Dockerfile.cuda` 删 27-36 行(cache-bust + 镜像内编译段落)。效果: + +- 镜像内容只在**依赖本身变化**(改 Dockerfile)时才变 → devcontainer 每次重建产出的镜像层摘要不变 → runner 端 `docker pull` 变 no-op → **Initialize containers 稳定 ~0-1min**; +- 镜像体积:gnu 减 ~1GB、cuda 减数 GB(abacus 构建产物 + 源码)→ 真需要重拉时也更快; +- 符合 AGENTS.md「复用现有 Docker 资产、不新增容器」——这是做减法。 + +**需要处理的一个副作用**:这两个镜像的定位本来是"给用户体验 ABACUS"(Dockerfile 注释写明 aimed for evaluating ABACUS),删掉内置二进制后,想直接 `docker run` 用 abacus 的用户会受影响。PR 里提供两个选项让维护者选: +- 选项 1:接受——evaluating 用户改用 release 二进制/自行编译,README/文档更新一句; +- 选项 2:把"含 abacus 的 eval 镜像"拆成单独的 `Dockerfile.eval`(CI 用纯净 deps 镜像,eval 镜像照旧天天新)——代价是多一个 Dockerfile,需要在 PR 里按 AGENTS.md 要求写明理由。 + +### 改法 A2(可叠加:拉镜像换国内源) + +`Dockerfile.gnu` 注释里已写明镜像同时发布在 `registry.dp.tech/deepmodeling/abacus-*`,devcontainer.yml 也推 `dp-harbor-registry.us-east-1.cr.aliyuncs.com`。runner 在国内,却让它们从 ghcr.io 拉。 + +```yaml + container: + image: registry.dp.tech/deepmodeling/abacus-cuda # 原: ghcr.io/deepmodeling/abacus-cuda +``` + +一行改动。先验证匿名可拉(找台机器 `docker pull registry.dp.tech/deepmodeling/abacus-gnu:latest` 试试),拉不通就用 dp-harbor 地址(需配 docker login,workflow 里用 secrets)。 + +--- + +## 方案 E:toolchain 时间归零(随 PR-3 的镜像一起改) + +**问题**:7/1 #7449 起每次运行跑完整 `install_stage4.sh`(dftd4/cereal/rapidjson/**libtorch ~200MB 下载**/libnpy/libri/libcomm/nep)。热节点 20 秒,冷节点数分钟;且 **cuda.yml 的构建根本没开 MLALGO,libtorch 白下载**。 + +**改法**: +1. 把 stage4 产物预装进 deps 镜像(Dockerfile 里加一段 stage4 安装,libtorch gnu 镜像已有); +2. CI 的 toolchain 步骤保留(幂等,检测到已装会秒过)——热路径不变,冷路径归零; +3. cuda.yml 如确认不需要 libtorch/LibRI,toolchain 步骤可改为只装必需项(`install_dftd4.sh` 等单独调用),或 configure 时显式关掉对应 feature 避免误装。 + +--- + +## 方案 D:测试提速(PR-4,改动最大放最后) + +### D1. test.yml:26 个串行步骤 → 分组并行 + +**现状**:单 job 里 16 个 Module_* 单测 + 10 个集成套件串行跑 ~18min(实测:01_PW 4.4、08_EXX 3.0、03_NAO_multik 2.6、09_DeePKS 2.1…)。 + +**轻量版(diff 小,推荐先上)**:16 个 Module_* 步骤合并成一个并行调用: + +```yaml + - name: All Module Unittests + env: + GTEST_COLOR: 'yes' + OMP_NUM_THREADS: '2' + run: | + ctest --test-dir build -j $(nproc) --timeout 1700 \ + -R 'MODULE_' -E PERF_MODULE_HSOLVER_KERNELS +``` + +集成套件(每个内部 Autotest.sh 串行跑用例)保持分步即可。**收益:单测 ~4.5min → ~1min。** + +**完整版(参照你 cuda.yml 的 matrix 改法)**:build job 产出 `install/` artifact → 10 个集成套件 + 1 个单测 job 并行下载运行。18min → ~5min,但 yml 改动大,建议等 PR-1~3 合入后再提。 + +### D2. 01_PW GPU 套件瘦身(cuda.yml) + +**问题**:#7690 加的 73 个用例 `-n 1` 串行,+4.8min/次,且用例是为 CPU 写的。 + +**改法**: +1. `-n 1` → `-n 2`(与其他 GPU 套件一致); +2. `CASES_GPU.txt` 拆两份:`CASES_GPU_SMOKE.txt`(~20 个代表用例,PR 跑)+ 全量留给定时任务: + +```yaml +on: + pull_request: + schedule: + - cron: '0 18 * * *' # 每晚全量(UTC) + workflow_dispatch: + +jobs: + gpu-test: + steps: + - name: Select case list + run: | + if [ "${{ github.event_name }}" = "pull_request" ]; then + echo "CASES=CASES_GPU_SMOKE.txt" >> "$GITHUB_ENV" + else + echo "CASES=CASES_GPU.txt" >> "$GITHUB_ENV" + fi + - name: Test 01_PW on GPU + run: | + cd tests/01_PW && bash ../integrate/Autotest.sh -n 2 -a abacus -f "$CASES" +``` + +--- + +## 落地路线图(给导师汇报版) + +| 批次 | 内容 | 文件 | 预期收益 | 过审难度 | +|---|---|---|---|---| +| **PR-1** | B1 架构裁剪 + B2 nproc + D3 浅克隆 | cuda.yml、test.yml | CUDA Build 33→8~15min;冷节点 checkout 省 12min | ★ 极易(几行配置) | +| **PR-2** | C1 ccache 容量+统计;C3 cache-warmer | 2 个 yml + 1 个新 yml | 消除 2.7 vs 37.5min 波动;风暴后自动回暖 | ★★ 易(纯新增步骤) | +| **PR-3** | A1 Dockerfile 删 cache-bust+镜像内编译(+E stage4 预装 +A2 国内源) | 2 个 Dockerfile、devcontainer.yml、2 个 yml | init 8~78→~1min;toolchain 归零 | ★★★ 中(改镜像定位,需共识) | +| **PR-4** | D1 单测合并并行 + D2 GPU 套件瘦身(之后视情况上完整 matrix) | test.yml、cuda.yml、CASES_GPU | CPU 测试 18→~6min;GPU 测试省 ~3min | ★★ 易~中 | + +**验证方法(每个 PR 必做,写进 PR 描述,符合 AGENTS.md「报告确切验证」要求)**: +1. 从 fork 提交后观察本 PR 的 CI 运行时长对比(注意:fork PR 改 workflow 文件首次需维护者 approve,先发 PR 后 @ 维护者); +2. 贴 `ccache -s` 的 hit rate 前后对比; +3. 贴 Initialize containers / Build 步骤耗时的前后截图; +4. 确认 GPU 测试全部通过(架构裁剪后二进制必须能在 CI 卡上跑——这也是 nvidia-smi 要先看的原因)。 + +**风险提示(写进 PR)**: +- B1 若填错架构 → GPU 测试全挂(立即可见,回退一行即可); +- B2 不做 B1 直接加并行 → nvcc 内存 ×7 架构可能 OOM; +- A1 影响"用镜像体验 abacus"的用户路径,PR 里给维护者两个选项; +- C3 cache-warmer 占用 runner 时间,建议 `concurrency` 防堆积。 + +--- + +## 附:为什么这些方案能成立(事实锚点) + +| 方案依赖的事实 | 出处 | +|---|---| +| ccache 自动接入(含 CUDA) | CMakeLists.txt:299-303 | +| 构建只需 git log -1 | CMakeLists.txt:163-174 | +| 7 架构默认列表 | CMakeLists.txt:445-476 | +| -j4 写死 | 上游 cuda.yml `cmake --build build -j4` | +| 镜像每次合并重建 + cache-bust + 镜像内编译 | devcontainer.yml;Dockerfile.gnu:31-41;Dockerfile.cuda:27-36 | +| 国内镜像源已存在 | Dockerfile.gnu:4 注释;devcontainer.yml:39 | +| 01_PW GPU -n1 串行 73 例 | #7690(4b6ce1c63) | +| init 8~78min、Build 2.7 vs 37.5min 实测 | GitHub API runs/jobs 数据(见分析文档) | diff --git a/source/source_esolver/esolver_ks_lcao.cpp b/source/source_esolver/esolver_ks_lcao.cpp index 422e2f9fef..d85e6e0f27 100644 --- a/source/source_esolver/esolver_ks_lcao.cpp +++ b/source/source_esolver/esolver_ks_lcao.cpp @@ -450,7 +450,9 @@ void ESolver_KS_LCAO::hamilt2rho_single(UnitCell& ucell, int istep, int PARAM.inp.ks_solver, PARAM.globalv.kpar_lcao, PARAM.globalv.nlocal, - PARAM.inp.nelec); + PARAM.inp.nbands, + PARAM.inp.nelec, + PARAM.inp.device == "gpu"); hsolver_lcao_obj.solve(static_cast*>(this->p_hamilt), this->psi[0], this->pelec, *this->dmat.dm, this->chr, PARAM.inp.nspin, skip_charge); } diff --git a/source/source_esolver/esolver_ks_lcao_tddft.cpp b/source/source_esolver/esolver_ks_lcao_tddft.cpp index 154a3a65e2..f0b0e6de78 100644 --- a/source/source_esolver/esolver_ks_lcao_tddft.cpp +++ b/source/source_esolver/esolver_ks_lcao_tddft.cpp @@ -351,7 +351,9 @@ void ESolver_KS_LCAO_TDDFT::hamilt2rho_single(UnitCell& ucell, PARAM.inp.ks_solver, PARAM.globalv.kpar_lcao, PARAM.globalv.nlocal, - PARAM.inp.nelec); + PARAM.inp.nbands, + PARAM.inp.nelec, + PARAM.inp.device == "gpu"); hsolver_lcao_obj.solve(static_cast>*>(this->p_hamilt), this->psi[0], this->pelec, diff --git a/source/source_esolver/esolver_ks_lcaopw.cpp b/source/source_esolver/esolver_ks_lcaopw.cpp index 036648394b..95165d262b 100644 --- a/source/source_esolver/esolver_ks_lcaopw.cpp +++ b/source/source_esolver/esolver_ks_lcaopw.cpp @@ -135,7 +135,10 @@ namespace ModuleESolver hsolver::DiagoIterAssist::PW_DIAG_NMAX = PARAM.inp.pw_diag_nmax; bool skip_charge = PARAM.inp.calculation == "nscf" ? true : false; - hsolver::HSolverLIP hsolver_lip_obj(this->pw_wfc, PARAM.globalv.use_uspp); + hsolver::HSolverLIP hsolver_lip_obj(this->pw_wfc, + PARAM.globalv.use_uspp, + PARAM.inp.basis_type, + PARAM.inp.calculation); hsolver_lip_obj.solve(static_cast*>(this->p_hamilt), *this->stp.template get_psi_t(), this->pelec, *this->psi_local, skip_charge,ucell.tpiba,ucell.nat); diff --git a/source/source_hsolver/diago_cusolver.cpp b/source/source_hsolver/diago_cusolver.cpp index b02c413257..34ced44cb8 100644 --- a/source/source_hsolver/diago_cusolver.cpp +++ b/source/source_hsolver/diago_cusolver.cpp @@ -6,7 +6,6 @@ #include "source_base/module_external/scalapack_connector.h" #include "source_base/tool_title.h" #include "source_base/timer.h" -#include "source_io/module_parameter/parameter.h" #include #include @@ -22,7 +21,7 @@ template int DiagoCusolver::DecomposedState = 0; template -DiagoCusolver::DiagoCusolver() +DiagoCusolver::DiagoCusolver(const int nlocal_in, const int nbands_in) : nlocal(nlocal_in), nbands(nbands_in) { } @@ -42,13 +41,13 @@ void DiagoCusolver::diag( ModuleBase::TITLE("DiagoCusolver", "diag"); ModuleBase::timer::start("DiagoCusolver", "cusolver"); // Allocate memory for eigenvalues - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); std::vector eigenvectors(h_mat.row * h_mat.col); this->dc.Dngvd(h_mat.row, h_mat.col, h_mat.p, s_mat.p, eigen.data(), eigenvectors.data()); const int size = psi.get_nbands() * psi.get_nbasis(); BlasConnector::copy(size, eigenvectors.data(), 1, psi.get_pointer(), 1); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); ModuleBase::timer::end("DiagoCusolver", "cusolver"); } diff --git a/source/source_hsolver/diago_cusolver.h b/source/source_hsolver/diago_cusolver.h index 28ebdc852c..50d9a393c6 100644 --- a/source/source_hsolver/diago_cusolver.h +++ b/source/source_hsolver/diago_cusolver.h @@ -19,9 +19,11 @@ class DiagoCusolver public: - DiagoCusolver(); + /// @param nlocal_in global dimension of the NAO Hamiltonian + /// @param nbands_in number of lowest eigenpairs to compute + DiagoCusolver(const int nlocal_in, const int nbands_in); ~DiagoCusolver(); - + // Override the diag function for CUSOLVER diagonalization void diag( hamilt::MatrixBlock& h_mat, @@ -40,6 +42,9 @@ class DiagoCusolver // Function to check if ELPA handle needs to be created or reused in MPI settings bool ifElpaHandle(const bool& newIteration, const bool& ifNSCF) const; #endif + + const int nlocal; + const int nbands; }; } // namespace hsolver diff --git a/source/source_hsolver/diago_cusolvermp.cpp b/source/source_hsolver/diago_cusolvermp.cpp index b5bb80eb5f..395add83d2 100644 --- a/source/source_hsolver/diago_cusolvermp.cpp +++ b/source/source_hsolver/diago_cusolvermp.cpp @@ -1,6 +1,5 @@ #ifdef __CUSOLVERMP -#include "source_io/module_parameter/parameter.h" #include "diago_cusolvermp.h" #include "source_base/module_external/blas_connector.h" @@ -18,7 +17,7 @@ void DiagoCusolverMP::diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* hamilt::MatrixBlock h_mat, s_mat; phm_in->matrix(h_mat, s_mat); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); std::vector eigenvectors(h_mat.row * h_mat.col); MPI_Comm COMM_DIAG = MPI_COMM_WORLD; // use all processes @@ -30,7 +29,7 @@ void DiagoCusolverMP::diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* ModuleBase::timer::end("DiagoCusolverMP", "Diag_CusolverMP_gvd"); } const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); const int size = psi.get_nbands() * psi.get_nbasis(); BlasConnector::copy(size, eigenvectors.data(), inc, psi.get_pointer(), inc); } diff --git a/source/source_hsolver/diago_cusolvermp.h b/source/source_hsolver/diago_cusolvermp.h index d173315e31..155a4bf94d 100644 --- a/source/source_hsolver/diago_cusolvermp.h +++ b/source/source_hsolver/diago_cusolvermp.h @@ -16,11 +16,17 @@ class DiagoCusolverMP using Real = typename GetTypeReal::type; public: - DiagoCusolverMP() + /// @param nlocal_in global dimension of the NAO Hamiltonian + /// @param nbands_in number of lowest eigenpairs to compute + DiagoCusolverMP(const int nlocal_in, const int nbands_in) : nlocal(nlocal_in), nbands(nbands_in) { } // the diag function for CUSOLVERMP diagonalization void diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* eigenvalue_in); + + private: + const int nlocal; + const int nbands; }; } // namespace hsolver #endif // __CUSOLVERMP diff --git a/source/source_hsolver/diago_elpa.cpp b/source/source_hsolver/diago_elpa.cpp index a264a41b9c..55ca982539 100644 --- a/source/source_hsolver/diago_elpa.cpp +++ b/source/source_hsolver/diago_elpa.cpp @@ -2,7 +2,6 @@ #include "source_base/global_function.h" #include "source_base/module_external/blas_connector.h" -#include "source_io/module_parameter/parameter.h" #include "module_genelpa/elpa_solver.h" #include "source_base/module_external/blacs_connector.h" #include "source_base/global_variable.h" @@ -75,13 +74,13 @@ void DiagoElpa>::diag( matcd h_mat, s_mat; phm_in->matrix(h_mat, s_mat); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); bool isReal = false; MPI_Comm COMM_DIAG = setmpicomm(); // set mpi_comm needed ELPA_Solver es((const bool)isReal, COMM_DIAG, - (const int)PARAM.inp.nbands, + (const int)this->nbands, (const int)h_mat.row, (const int)h_mat.col, (const int*)h_mat.desc); @@ -97,7 +96,7 @@ void DiagoElpa>::diag( es.exit(); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); #else ModuleBase::WARNING_QUIT("DiagoElpa", "DiagoElpa only can be used with macro __MPI"); @@ -113,15 +112,13 @@ void DiagoElpa::diag(hamilt::Hamilt* phm_in, matd h_mat, s_mat; phm_in->matrix(h_mat, s_mat); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); bool isReal = true; MPI_Comm COMM_DIAG = setmpicomm(); // set mpi_comm needed - // ELPA_Solver es(isReal, COMM_DIAG, PARAM.inp.nbands, h_mat.row, h_mat.col, - // h_mat.desc); ELPA_Solver es((const bool)isReal, COMM_DIAG, - (const int)PARAM.inp.nbands, + (const int)this->nbands, (const int)h_mat.row, (const int)h_mat.col, (const int*)h_mat.desc); @@ -135,7 +132,7 @@ void DiagoElpa::diag(hamilt::Hamilt* phm_in, es.exit(); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); #else ModuleBase::WARNING_QUIT("DiagoElpa", "DiagoElpa only can be used with macro __MPI"); @@ -151,11 +148,11 @@ void DiagoElpa>::diag_pool(hamilt::MatrixBlock eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); bool isReal = false; ELPA_Solver es((const bool)isReal, comm, - (const int)PARAM.inp.nbands, + (const int)this->nbands, (const int)h_mat.row, (const int)h_mat.col, (const int*)h_mat.desc); @@ -170,7 +167,7 @@ void DiagoElpa>::diag_pool(hamilt::MatrixBlocknbands, eigen.data(), inc, eigenvalue_in, inc); } template <> @@ -180,14 +177,12 @@ void DiagoElpa::diag_pool(hamilt::MatrixBlock& h_mat, Real* eigenvalue_in, MPI_Comm& comm) { - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); bool isReal = true; - // ELPA_Solver es(isReal, COMM_DIAG, PARAM.inp.nbands, h_mat.row, h_mat.col, - // h_mat.desc); ELPA_Solver es((const bool)isReal, comm, - (const int)PARAM.inp.nbands, + (const int)this->nbands, (const int)h_mat.row, (const int)h_mat.col, (const int*)h_mat.desc); @@ -203,7 +198,7 @@ void DiagoElpa::diag_pool(hamilt::MatrixBlock& h_mat, const int inc = 1; ModuleBase::GlobalFunc::OUT(GlobalV::ofs_running, "K-S equation was solved by genelpa2"); - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); ModuleBase::GlobalFunc::OUT(GlobalV::ofs_running, "eigenvalues were copied to ekb"); } diff --git a/source/source_hsolver/diago_elpa.h b/source/source_hsolver/diago_elpa.h index c6050af224..2bee7a14fa 100644 --- a/source/source_hsolver/diago_elpa.h +++ b/source/source_hsolver/diago_elpa.h @@ -15,6 +15,10 @@ class DiagoElpa using Real = typename GetTypeReal::type; public: + /// @param nlocal_in global dimension of the NAO Hamiltonian + /// @param nbands_in number of lowest eigenpairs to compute + DiagoElpa(const int nlocal_in, const int nbands_in) : nlocal(nlocal_in), nbands(nbands_in) {}; + void diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* eigenvalue_in); #ifdef __MPI // diagnolization used in parallel-k case @@ -30,6 +34,9 @@ class DiagoElpa bool ifElpaHandle(const bool& newIteration, const bool& ifNSCF) const; static int lastmpinum; // last using mpi; #endif + + const int nlocal; + const int nbands; }; template diff --git a/source/source_hsolver/diago_elpa_native.cpp b/source/source_hsolver/diago_elpa_native.cpp index ee003a08e0..fb4cfbc448 100644 --- a/source/source_hsolver/diago_elpa_native.cpp +++ b/source/source_hsolver/diago_elpa_native.cpp @@ -4,7 +4,6 @@ #include "source_base/module_external/blas_connector.h" #include "source_base/module_external/blacs_connector.h" #include "source_base/global_variable.h" -#include "source_io/module_parameter/parameter.h" #include "source_base/timer.h" #include "source_base/tool_quit.h" #include "source_hsolver/module_genelpa/elpa_new.h" @@ -59,7 +58,7 @@ void DiagoElpaNative::diag_pool(hamilt::MatrixBlock& h_mat, ModuleBase::timer::start("DiagoElpaNative", "elpa_solve"); - int nev = PARAM.inp.nbands; + int nev = this->nbands; int narows = h_mat.row; int nacols = h_mat.col; @@ -70,7 +69,7 @@ void DiagoElpaNative::diag_pool(hamilt::MatrixBlock& h_mat, int nprows, npcols, myprow, mypcol; Cblacs_gridinfo(cblacs_ctxt, &nprows, &npcols, &myprow, &mypcol); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); std::vector eigenvectors(narows * nacols); if (elpa_init(20210430) != ELPA_OK) @@ -107,7 +106,7 @@ void DiagoElpaNative::diag_pool(hamilt::MatrixBlock& h_mat, #define ELPA_WITH_SYCL_GPU_VERSION 0 */ #if ELPA_WITH_NVIDIA_GPU_VERSION - if (PARAM.inp.device == "gpu") + if (this->use_gpu) { elpa_set(handle, "nvidia-gpu", 1, &success); elpa_set(handle, "real_kernel", ELPA_2STAGE_REAL_NVIDIA_GPU, &success); @@ -138,7 +137,7 @@ void DiagoElpaNative::diag_pool(hamilt::MatrixBlock& h_mat, } const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); const int size = psi.get_nbands() * psi.get_nbasis(); BlasConnector::copy(size, eigenvectors.data(), inc, psi.get_pointer(), inc); } diff --git a/source/source_hsolver/diago_elpa_native.h b/source/source_hsolver/diago_elpa_native.h index 2c556254e8..2cbd2c27d6 100644 --- a/source/source_hsolver/diago_elpa_native.h +++ b/source/source_hsolver/diago_elpa_native.h @@ -15,6 +15,12 @@ class DiagoElpaNative using Real = typename GetTypeReal::type; public: + /// @param nlocal_in global dimension of the NAO Hamiltonian + /// @param nbands_in number of lowest eigenpairs to compute + /// @param use_gpu_in offload to the NVIDIA-GPU ELPA kernels when ELPA was built with GPU support + DiagoElpaNative(const int nlocal_in, const int nbands_in, const bool use_gpu_in) + : nlocal(nlocal_in), nbands(nbands_in), use_gpu(use_gpu_in) {}; + void diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* eigenvalue_in); #ifdef __MPI // diagnolization used in parallel-k case @@ -27,6 +33,10 @@ class DiagoElpaNative static int DecomposedState; + private: + const int nlocal; + const int nbands; + const bool use_gpu; }; template diff --git a/source/source_hsolver/diago_iter_assist.cpp b/source/source_hsolver/diago_iter_assist.cpp index b547febefe..8436e4a26b 100644 --- a/source/source_hsolver/diago_iter_assist.cpp +++ b/source/source_hsolver/diago_iter_assist.cpp @@ -1,5 +1,4 @@ #include "diago_iter_assist.h" -#include "source_io/module_parameter/parameter.h" #include "source_base/complexmatrix.h" #include "source_base/constants.h" #include "source_base/global_function.h" @@ -177,6 +176,8 @@ void DiagoIterAssist::diag_subspace_init(hamilt::Hamilt* p int psi_nc, psi::Psi& evc, Real* en, + const std::string& basis_type, + const std::string& calculation, const std::function& add_to_hcc, const std::function& export_vcc) { @@ -330,13 +331,13 @@ void DiagoIterAssist::diag_subspace_init(hamilt::Hamilt* p //======================= // diagonize the H-matrix //======================= - if ((PARAM.inp.basis_type == "lcao" || PARAM.inp.basis_type == "lcao_in_pw") && PARAM.inp.calculation == "nscf") + if ((basis_type == "lcao" || basis_type == "lcao_in_pw") && calculation == "nscf") { GlobalV::ofs_running << " Not do zgemm to get evc." << std::endl; } - else if ((PARAM.inp.basis_type == "lcao" || PARAM.inp.basis_type == "lcao_in_pw" || PARAM.inp.basis_type == "pw") - && (PARAM.inp.calculation == "scf" || PARAM.inp.calculation == "md" - || PARAM.inp.calculation == "relax")) // pengfei 2014-10-13 + else if ((basis_type == "lcao" || basis_type == "lcao_in_pw" || basis_type == "pw") + && (calculation == "scf" || calculation == "md" + || calculation == "relax")) // pengfei 2014-10-13 { // because psi and evc are different here, // I think if psi and evc are the same, @@ -637,31 +638,6 @@ void DiagoIterAssist::diag_subspace_psi(const T* hcc, ModuleBase::timer::end("DiagoIterAssist", "diag_subspace_psi"); } -template -bool DiagoIterAssist::test_exit_cond(const int& ntry, const int& notconv) -{ - //================================================================ - // If this logical function is true, need to do diag_subspace - // and cg again. - //================================================================ - - bool scf = true; - if (PARAM.inp.calculation == "nscf") { - scf = false; -} - - // If ntry <=5, try to do it better, if ntry > 5, exit. - const bool f1 = (ntry <= 5); - - // In non-self consistent calculation, do until totally converged. - const bool f2 = ((!scf && (notconv > 0))); - - // if self consistent calculation, if not converged > 5, - // using diag_subspace and cg method again. ntry++ - const bool f3 = ((scf && (notconv > 5))); - return (f1 && (f2 || f3)); -} - template class DiagoIterAssist, base_device::DEVICE_CPU>; template class DiagoIterAssist, base_device::DEVICE_CPU>; #if ((defined __CUDA) || (defined __ROCM)) diff --git a/source/source_hsolver/diago_iter_assist.h b/source/source_hsolver/diago_iter_assist.h index e9c4c00520..8302933840 100644 --- a/source/source_hsolver/diago_iter_assist.h +++ b/source/source_hsolver/diago_iter_assist.h @@ -7,6 +7,7 @@ #include "source_psi/psi.h" #include +#include namespace hsolver { @@ -61,7 +62,10 @@ class DiagoIterAssist /// @param psi_nc number of columns (nbasis) /// @param evc new wavefunction /// @param en eigenenergies - /// @note exception handle: if there is no operator initialized in Hamilt, will directly copy value from psi to evc, + /// @param basis_type "lcao", "lcao_in_pw" or "pw"; together with calculation it selects + /// how the rotation matrix is applied to psi + /// @param calculation "scf", "nscf", "md", "relax", ... + /// @note exception handle: if there is no operator initialized in Hamilt, will directly copy value from psi to evc, /// and return all - zero eigenenergies. static void diag_subspace_init( hamilt::Hamilt* pHamilt, @@ -70,6 +74,8 @@ class DiagoIterAssist int psi_nc, psi::Psi &evc, Real* en, + const std::string& basis_type, + const std::string& calculation, const std::function& add_to_hcc = [](T* null, const int n) {}, const std::function& export_vcc = [](const T* null, const int n, const int m) {}); @@ -120,8 +126,6 @@ class DiagoIterAssist psi::Psi& evc, Real* en); - static bool test_exit_cond(const int& ntry, const int& notconv); - private: constexpr static const Device* ctx = {}; diff --git a/source/source_hsolver/diago_lapack.cpp b/source/source_hsolver/diago_lapack.cpp index 6bdba41ae9..0020efc1e8 100644 --- a/source/source_hsolver/diago_lapack.cpp +++ b/source/source_hsolver/diago_lapack.cpp @@ -1,7 +1,5 @@ // Refactored according to diago_scalapack // This code will be futher refactored to remove the dependency of psi and hamilt -#include "source_io/module_parameter/parameter.h" - #include "diago_lapack.h" #include "source_base/global_variable.h" @@ -42,14 +40,14 @@ void DiagoLapack::diag(hamilt::Hamilt* phm_in, psi::Psi& phm_in->matrix(h_mat, s_mat); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); check_lapack_layout(h_mat, s_mat, eigen.size()); // Diag this->dsygvx_diag(h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); // Copy result const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } template <> @@ -61,11 +59,11 @@ void DiagoLapack>::diag(hamilt::Hamilt matcd h_mat, s_mat; phm_in->matrix(h_mat, s_mat); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); check_lapack_layout(h_mat, s_mat, eigen.size()); this->zhegvx_diag(h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } #ifdef __MPI @@ -78,11 +76,11 @@ void DiagoLapack>::diag(hamilt::Hamilt { ModuleBase::TITLE("DiagoLapack", "diag_pool"); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); check_lapack_layout(h_mat, s_mat, eigen.size()); this->dsygvx_diag(h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } template<> void DiagoLapack>::diag_pool(hamilt::MatrixBlock>& h_mat, @@ -93,11 +91,11 @@ void DiagoLapack>::diag(hamilt::Hamilt { ModuleBase::TITLE("DiagoLapack", "diag_pool"); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); check_lapack_layout(h_mat, s_mat, eigen.size()); this->zhegvx_diag(h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } #endif @@ -115,20 +113,20 @@ std::pair> DiagoLapack::dsygvx_once(const int ncol, memcpy(s_tmp.c, s_mat, sizeof(double) * ncol * nrow); const char jobz = 'V', range = 'I', uplo = 'U'; - const int itype = 1, il = 1, iu = PARAM.inp.nbands, one = 1; + const int itype = 1, il = 1, iu = this->nbands, one = 1; int M = 0, NZ = 0, lwork = -1, liwork = -1, info = 0; double vl = 0, vu = 0; const double abstol = LAPACK_ABSTOL, orfac = LAPACK_ORFAC; std::vector work(3, 0); std::vector iwork(1, 0); - std::vector ifail(PARAM.globalv.nlocal, 0); + std::vector ifail(this->nlocal, 0); std::vector iclustr(2 * GlobalV::DSIZE); std::vector gap(GlobalV::DSIZE); // LAPACK dsygvx signature: // (ITYPE, JOBZ, RANGE, UPLO, N, A, LDA, B, LDB, VL, VU, IL, IU, // ABSTOL, M, W, Z, LDZ, WORK, LWORK, IWORK, IFAIL, INFO) - int n = PARAM.globalv.nlocal; + int n = this->nlocal; int lda = n, ldb = n, ldz = n; dsygvx_(&itype, &jobz, @@ -224,7 +222,7 @@ std::pair> DiagoLapack::zhegvx_once(const int ncol, memcpy(s_tmp.c, s_mat, sizeof(std::complex) * ncol * nrow); const char jobz = 'V', range = 'I', uplo = 'U'; - const int itype = 1, il = 1, iu = PARAM.inp.nbands, one = 1; + const int itype = 1, il = 1, iu = this->nbands, one = 1; int M = 0, NZ = 0, lwork = -1, lrwork = -1, liwork = -1, info = 0; const double abstol = LAPACK_ABSTOL, orfac = LAPACK_ORFAC; @@ -232,14 +230,14 @@ std::pair> DiagoLapack::zhegvx_once(const int ncol, std::vector> work(1, 0); std::vector rwork(3, 0); std::vector iwork(1, 0); - std::vector ifail(PARAM.globalv.nlocal, 0); + std::vector ifail(this->nlocal, 0); std::vector iclustr(2 * GlobalV::DSIZE); std::vector gap(GlobalV::DSIZE); // LAPACK zhegvx signature: // (ITYPE, JOBZ, RANGE, UPLO, N, A, LDA, B, LDB, VL, VU, IL, IU, // ABSTOL, M, W, Z, LDZ, WORK, LWORK, RWORK, IWORK, IFAIL, INFO) - int n = PARAM.globalv.nlocal; + int n = this->nlocal; int lda = n, ldb = n, ldz = n; zhegvx_(&itype, &jobz, @@ -416,8 +414,7 @@ void DiagoLapack::post_processing(const int info, const std::vector& vec { const std::string str_M = "M = " + ModuleBase::GlobalFunc::TO_STRING(vec[0]) + ".\n"; const std::string str_NZ = "NZ = " + ModuleBase::GlobalFunc::TO_STRING(vec[1]) + ".\n"; - const std::string str_NBANDS - = "PARAM.inp.nbands = " + ModuleBase::GlobalFunc::TO_STRING(PARAM.inp.nbands) + ".\n"; + const std::string str_NBANDS = "nbands = " + ModuleBase::GlobalFunc::TO_STRING(this->nbands) + ".\n"; throw std::runtime_error(str_info_FILE + str_M + str_NZ + str_NBANDS); } else if (info / 16 % 2) diff --git a/source/source_hsolver/diago_lapack.h b/source/source_hsolver/diago_lapack.h index bfdf78ac34..9f674fc20f 100644 --- a/source/source_hsolver/diago_lapack.h +++ b/source/source_hsolver/diago_lapack.h @@ -26,6 +26,10 @@ class DiagoLapack using Real = typename GetTypeReal::type; public: + /// @param nlocal_in global dimension of the NAO Hamiltonian + /// @param nbands_in number of lowest eigenpairs to compute + DiagoLapack(const int nlocal_in, const int nbands_in) : nlocal(nlocal_in), nbands(nbands_in) {}; + void diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* eigenvalue_in); #ifdef __MPI // diagnolization used in parallel-k case @@ -61,6 +65,10 @@ class DiagoLapack int degeneracy_max = 12; // For reorthogonalized memory. 12 followes siesta. void post_processing(const int info, const std::vector& vec); + + private: + const int nlocal; + const int nbands; }; } // namespace hsolver diff --git a/source/source_hsolver/diago_scalapack.cpp b/source/source_hsolver/diago_scalapack.cpp index 366478fdc2..f2ddb8289b 100644 --- a/source/source_hsolver/diago_scalapack.cpp +++ b/source/source_hsolver/diago_scalapack.cpp @@ -1,6 +1,5 @@ //===================== // AUTHOR : Peize Lin -#include "source_io/module_parameter/parameter.h" // DATE : 2021-11-02 // REFACTORING AUTHOR : Daye Zheng // DATE : 2022-04-14 @@ -28,10 +27,10 @@ namespace hsolver matd h_mat, s_mat; phm_in->matrix(h_mat, s_mat); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); this->pdsygvx_diag(h_mat.desc, h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } template<> void DiagoScalapack>::diag(hamilt::Hamilt>* phm_in, psi::Psi>& psi, Real* eigenvalue_in) @@ -40,10 +39,10 @@ namespace hsolver matcd h_mat, s_mat; phm_in->matrix(h_mat, s_mat); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); this->pzhegvx_diag(h_mat.desc, h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } #ifdef __MPI @@ -56,10 +55,10 @@ namespace hsolver { ModuleBase::TITLE("DiagoScalapack", "diag_pool"); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); this->pdsygvx_diag(h_mat.desc, h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } template<> void DiagoScalapack>::diag_pool(hamilt::MatrixBlock>& h_mat, @@ -70,10 +69,10 @@ namespace hsolver { ModuleBase::TITLE("DiagoScalapack", "diag_pool"); assert(h_mat.col == s_mat.col && h_mat.row == s_mat.row && h_mat.desc == s_mat.desc); - std::vector eigen(PARAM.globalv.nlocal, 0.0); + std::vector eigen(this->nlocal, 0.0); this->pzhegvx_diag(h_mat.desc, h_mat.col, h_mat.row, h_mat.p, s_mat.p, eigen.data(), psi); const int inc = 1; - BlasConnector::copy(PARAM.inp.nbands, eigen.data(), inc, eigenvalue_in, inc); + BlasConnector::copy(this->nbands, eigen.data(), inc, eigenvalue_in, inc); } #endif @@ -92,13 +91,13 @@ namespace hsolver memcpy(s_tmp.c, s_mat, sizeof(double) * ncol * nrow); const char jobz = 'V', range = 'I', uplo = 'U'; - const int itype = 1, il = 1, iu = PARAM.inp.nbands, one = 1; + const int itype = 1, il = 1, iu = this->nbands, one = 1; int M = 0, NZ = 0, lwork = -1, liwork = -1, info = 0; double vl = 0, vu = 0; const double abstol = SCALAPACK_ABSTOL, orfac = SCALAPACK_ORFAC; std::vector work(3, 0); std::vector iwork(1, 0); - std::vector ifail(PARAM.globalv.nlocal, 0); + std::vector ifail(this->nlocal, 0); std::vector iclustr(2 * GlobalV::DSIZE); std::vector gap(GlobalV::DSIZE); @@ -106,7 +105,7 @@ namespace hsolver &jobz, &range, &uplo, - &PARAM.globalv.nlocal, + &this->nlocal, h_tmp.c, &one, &one, @@ -152,7 +151,7 @@ namespace hsolver &jobz, &range, &uplo, - &PARAM.globalv.nlocal, + &this->nlocal, h_tmp.c, &one, &one, @@ -217,7 +216,7 @@ namespace hsolver memcpy(s_tmp.c, s_mat, sizeof(std::complex) * ncol * nrow); const char jobz = 'V', range = 'I', uplo = 'U'; - const int itype = 1, il = 1, iu = PARAM.inp.nbands, one = 1; + const int itype = 1, il = 1, iu = this->nbands, one = 1; int M = 0, NZ = 0, lwork = -1, lrwork = -1, liwork = -1, info = 0; const double abstol = SCALAPACK_ABSTOL, orfac = SCALAPACK_ORFAC; //Note: pzhegvx_ has a bug @@ -227,7 +226,7 @@ namespace hsolver std::vector> work(1, 0); std::vector rwork(3, 0); std::vector iwork(1, 0); - std::vector ifail(PARAM.globalv.nlocal, 0); + std::vector ifail(this->nlocal, 0); std::vector iclustr(2 * GlobalV::DSIZE); std::vector gap(GlobalV::DSIZE); @@ -235,7 +234,7 @@ namespace hsolver &jobz, &range, &uplo, - &PARAM.globalv.nlocal, + &this->nlocal, h_tmp.c, &one, &one, @@ -276,7 +275,7 @@ namespace hsolver // GlobalV::ofs_running<<"lwork="<degeneracy_max * PARAM.globalv.nlocal; + lrwork = rwork[0] + this->degeneracy_max * this->nlocal; int maxlrwork = std::max(lrwork,3); rwork.resize(maxlrwork, 0); liwork = iwork[0]; @@ -286,7 +285,7 @@ namespace hsolver &jobz, &range, &uplo, - &PARAM.globalv.nlocal, + &this->nlocal, h_tmp.c, &one, &one, @@ -431,7 +430,7 @@ namespace hsolver const std::string str_M = "M = " + ModuleBase::GlobalFunc::TO_STRING(vec[0]) + ".\n"; const std::string str_NZ = "NZ = " + ModuleBase::GlobalFunc::TO_STRING(vec[1]) + ".\n"; const std::string str_NBANDS - = "PARAM.inp.nbands = " + ModuleBase::GlobalFunc::TO_STRING(PARAM.inp.nbands) + ".\n"; + = "nbands = " + ModuleBase::GlobalFunc::TO_STRING(this->nbands) + ".\n"; throw std::runtime_error(str_info_FILE + str_M + str_NZ + str_NBANDS); } else if (info / 16 % 2) diff --git a/source/source_hsolver/diago_scalapack.h b/source/source_hsolver/diago_scalapack.h index 05b1144836..7dce030616 100644 --- a/source/source_hsolver/diago_scalapack.h +++ b/source/source_hsolver/diago_scalapack.h @@ -27,6 +27,10 @@ namespace hsolver private: using Real = typename GetTypeReal::type; public: + /// @param nlocal_in global dimension of the NAO Hamiltonian + /// @param nbands_in number of lowest eigenpairs to compute + DiagoScalapack(const int nlocal_in, const int nbands_in) : nlocal(nlocal_in), nbands(nbands_in) {}; + void diag(hamilt::Hamilt* phm_in, psi::Psi& psi, Real* eigenvalue_in); #ifdef __MPI // diagnolization used in parallel-k case @@ -67,6 +71,9 @@ namespace hsolver int degeneracy_max = 12; // For reorthogonalized memory. 12 followes siesta. void post_processing(const int info, const std::vector &vec); + + const int nlocal; + const int nbands; }; } // namespace hsolver diff --git a/source/source_hsolver/hsolver_lcao.cpp b/source/source_hsolver/hsolver_lcao.cpp index d31658d8ad..3c8bcc3ff0 100644 --- a/source/source_hsolver/hsolver_lcao.cpp +++ b/source/source_hsolver/hsolver_lcao.cpp @@ -141,19 +141,19 @@ void HSolverLCAO::hamiltSolvePsiK(hamilt::Hamilt* hm, psi::Psi& if (this->method == "scalapack_gvx") { #ifdef __MPI - DiagoScalapack sa; + DiagoScalapack sa(this->nlocal, this->nbands); sa.diag(hm, psi, eigenvalue); #endif } #ifdef __ELPA else if (this->method == "genelpa") { - DiagoElpa el; + DiagoElpa el(this->nlocal, this->nbands); el.diag(hm, psi, eigenvalue); } else if (this->method == "elpa") { - DiagoElpaNative el; + DiagoElpaNative el(this->nlocal, this->nbands, this->use_gpu); el.diag(hm, psi, eigenvalue); } #endif @@ -161,7 +161,7 @@ void HSolverLCAO::hamiltSolvePsiK(hamilt::Hamilt* hm, psi::Psi& else if (this->method == "cusolver") { // Note: This branch will only be executed in the single-process case - DiagoCusolver cu; + DiagoCusolver cu(this->nlocal, this->nbands); hamilt::MatrixBlock hk, sk; hm->matrix(hk, sk); cu.diag(hk, sk, psi, eigenvalue); @@ -169,14 +169,14 @@ void HSolverLCAO::hamiltSolvePsiK(hamilt::Hamilt* hm, psi::Psi& #ifdef __CUSOLVERMP else if (this->method == "cusolvermp") { - DiagoCusolverMP cm; + DiagoCusolverMP cm(this->nlocal, this->nbands); cm.diag(hm, psi, eigenvalue); } #endif #endif else if (this->method == "lapack") // only for single core { - DiagoLapack la; + DiagoLapack la(this->nlocal, this->nbands); la.diag(hm, psi, eigenvalue); } else @@ -253,23 +253,23 @@ void HSolverLCAO::parakSolve(hamilt::Hamilt* pHamilt, /// solve eigenvector and eigenvalue for H(k) if (this->method == "scalapack_gvx") { - DiagoScalapack sa; + DiagoScalapack sa(this->nlocal, this->nbands); sa.diag_pool(hk_pool, sk_pool, psi_pool, &(pes->ekb(ik_global, 0)), k2d.POOL_WORLD_K2D); } else if (this->method == "lapack") { - DiagoLapack la; + DiagoLapack la(this->nlocal, this->nbands); la.diag_pool(hk_pool, sk_pool, psi_pool, &(pes->ekb(ik_global, 0)), k2d.POOL_WORLD_K2D); } #ifdef __ELPA else if (this->method == "genelpa") { - DiagoElpa el; + DiagoElpa el(this->nlocal, this->nbands); el.diag_pool(hk_pool, sk_pool, psi_pool, &(pes->ekb(ik_global, 0)), k2d.POOL_WORLD_K2D); } else if (this->method == "elpa") { - DiagoElpaNative el; + DiagoElpaNative el(this->nlocal, this->nbands, this->use_gpu); el.diag_pool(hk_pool, sk_pool, psi_pool, &(pes->ekb(ik_global, 0)), k2d.POOL_WORLD_K2D); } #endif @@ -427,7 +427,7 @@ void HSolverLCAO::parakSolve_cusolver(hamilt::Hamilt* pHamilt, if(kpt_assigned != -1) { psi_local.resize(1, ncol, nrow); - DiagoCusolver cu{}; + DiagoCusolver cu(this->nlocal, this->nbands); hamilt::MatrixBlock hk_local = hamilt::MatrixBlock{ hk_mat.data(), (size_t)nrow, (size_t)ncol, mat_para_local.desc}; diff --git a/source/source_hsolver/hsolver_lcao.h b/source/source_hsolver/hsolver_lcao.h index 60a777e3a4..3b54d92d1f 100644 --- a/source/source_hsolver/hsolver_lcao.h +++ b/source/source_hsolver/hsolver_lcao.h @@ -19,8 +19,11 @@ class HSolverLCAO const std::string method_in, const int kpar_lcao_in, const int nlocal_in, - const double nelec_in) - : ParaV(ParaV_in), method(method_in), kpar_lcao(kpar_lcao_in), nlocal(nlocal_in), nelec(nelec_in) {}; + const int nbands_in, + const double nelec_in, + const bool use_gpu_in) + : ParaV(ParaV_in), method(method_in), kpar_lcao(kpar_lcao_in), nlocal(nlocal_in), nbands(nbands_in), + nelec(nelec_in), use_gpu(use_gpu_in) {}; void solve(hamilt::Hamilt* pHamilt, psi::Psi& psi, @@ -49,8 +52,10 @@ class HSolverLCAO const std::string method; const int kpar_lcao; // number of pools for LCAO diagonalization - const int nlocal; // global dimension of the NAO Hamiltonian, only used by the pexsi branch + const int nlocal; // global dimension of the NAO Hamiltonian + const int nbands; // number of bands to be solved for const double nelec; // total number of electrons, only used by the pexsi branch + const bool use_gpu; // true if running on GPU, only used by the native-ELPA branch }; } // namespace hsolver diff --git a/source/source_hsolver/hsolver_lcaopw.cpp b/source/source_hsolver/hsolver_lcaopw.cpp index f61035ef6a..66a14263ee 100644 --- a/source/source_hsolver/hsolver_lcaopw.cpp +++ b/source/source_hsolver/hsolver_lcaopw.cpp @@ -71,8 +71,10 @@ void HSolverLIP::solve(hamilt::Hamilt* pHamilt, // ESolver_KS_PW::p_hamilt transform.get_pointer(), // transform matrix between lcao and pw transform.get_nbands(), transform.get_nbasis(), - psi, // psi in pw basis - eigenvalues.data() + ik * pes->ekb.nc // eigenvalues + psi, // psi in pw basis + eigenvalues.data() + ik * pes->ekb.nc, // eigenvalues + this->basis_type, + this->calculation #ifdef __EXX , add_exx_to_subspace_hamilt, diff --git a/source/source_hsolver/hsolver_lcaopw.h b/source/source_hsolver/hsolver_lcaopw.h index 49bbe63cb9..c226376b0c 100644 --- a/source/source_hsolver/hsolver_lcaopw.h +++ b/source/source_hsolver/hsolver_lcaopw.h @@ -18,8 +18,11 @@ class HSolverLIP using Real = typename GetTypeReal::type; public: - HSolverLIP(ModulePW::PW_Basis_K* wfc_basis_in, const bool use_uspp_in) - : wfc_basis(wfc_basis_in), use_uspp(use_uspp_in) {}; + HSolverLIP(ModulePW::PW_Basis_K* wfc_basis_in, + const bool use_uspp_in, + const std::string basis_type_in, + const std::string calculation_in) + : wfc_basis(wfc_basis_in), use_uspp(use_uspp_in), basis_type(basis_type_in), calculation(calculation_in) {}; /// @brief solve function for lcao_in_pw /// @param pHamilt interface to hamilt @@ -39,6 +42,9 @@ class HSolverLIP ModulePW::PW_Basis_K* wfc_basis = nullptr; const bool use_uspp; // true if ultrasoft pseudopotentials are in use + + const std::string basis_type; // "lcao_in_pw" for this solver + const std::string calculation; // "scf", "nscf", "md", "relax", ... }; } // namespace hsolver diff --git a/source/source_hsolver/test/diago_lapack_test.cpp b/source/source_hsolver/test/diago_lapack_test.cpp index 8620eaafa4..e3d358052e 100644 --- a/source/source_hsolver/test/diago_lapack_test.cpp +++ b/source/source_hsolver/test/diago_lapack_test.cpp @@ -1,8 +1,5 @@ // Author: Zhang Xiaoyang // A modified version of diago_lcao_test.cpp -#define private public -#include "source_io/module_parameter/parameter.h" -#undef private // Remove some useless functions and dependencies. Serialized the full code // and refactored some function. @@ -184,21 +181,14 @@ class DiagoLapackPrepare hmtest.ncol = nlocal; } - void set_env() - { - PARAM.sys.nlocal = nlocal; - PARAM.input.nbands = nbands; - } - void diago() { this->pb2d(); this->print_hs(); - this->set_env(); for (int i = 0; i < REPEATRUN; i++) { - hsolver::DiagoLapack dh; + hsolver::DiagoLapack dh(nlocal, nbands); dh.diag(&hmtest, psi, e_solver.data()); // dh->diag(&hmtest, psi, e_solver.data()); } diff --git a/source/source_hsolver/test/diago_lcao_cusolver_test.cpp b/source/source_hsolver/test/diago_lcao_cusolver_test.cpp index 9f340419b7..3a5154af10 100644 --- a/source/source_hsolver/test/diago_lcao_cusolver_test.cpp +++ b/source/source_hsolver/test/diago_lcao_cusolver_test.cpp @@ -1,8 +1,5 @@ #include "source_hsolver/diago_scalapack.h" #include "source_hsolver/test/diago_elpa_utils.h" -#define private public -#include "source_io/module_parameter/parameter.h" -#undef private #include "mpi.h" #include "string.h" @@ -204,8 +201,6 @@ class DiagoPrepare void set_env() { - PARAM.sys.nlocal = nlocal; - PARAM.input.nbands = nbands; GlobalV::DSIZE = dsize; } @@ -225,13 +220,13 @@ class DiagoPrepare hmtest.s_local = this->s_local; if (ks_solver == "scalapack_gvx") { - hsolver::DiagoScalapack dh; + hsolver::DiagoScalapack dh(nlocal, nbands); dh.diag(&hmtest, psi, e_solver.data()); } #ifdef __CUDA else if (ks_solver == "cusolver") { - hsolver::DiagoCusolver dh; + hsolver::DiagoCusolver dh(nlocal, nbands); hamilt::MatrixBlock h_mat, s_mat; hmtest.matrix(h_mat, s_mat); dh.diag(h_mat, s_mat, psi, e_solver.data()); diff --git a/source/source_hsolver/test/diago_lcao_test.cpp b/source/source_hsolver/test/diago_lcao_test.cpp index d0ed0fdae6..6d66269415 100644 --- a/source/source_hsolver/test/diago_lcao_test.cpp +++ b/source/source_hsolver/test/diago_lcao_test.cpp @@ -1,9 +1,6 @@ #include "source_hsolver/diago_scalapack.h" #include "source_hsolver/diago_lapack.h" #include "source_hsolver/test/diago_elpa_utils.h" -#define private public -#include "source_io/module_parameter/parameter.h" -#undef private #include "mpi.h" #include "string.h" @@ -205,8 +202,6 @@ class DiagoPrepare void set_env() { - PARAM.sys.nlocal = nlocal; - PARAM.input.nbands = nbands; GlobalV::DSIZE = dsize; } @@ -226,18 +221,18 @@ class DiagoPrepare hmtest.s_local = this->s_local; if (ks_solver == "scalapack_gvx") { - hsolver::DiagoScalapack dh; + hsolver::DiagoScalapack dh(nlocal, nbands); dh.diag(&hmtest, psi, e_solver.data()); } else if (ks_solver == "lapack") { - hsolver::DiagoLapack la; + hsolver::DiagoLapack la(nlocal, nbands); la.diag(&hmtest, psi, e_solver.data()); } #ifdef __ELPA else if (ks_solver == "genelpa") { - hsolver::DiagoElpa dh; + hsolver::DiagoElpa dh(nlocal, nbands); dh.diag(&hmtest, psi, e_solver.data()); } #endif diff --git a/source/source_hsolver/test/test_hsolver_pw.cpp b/source/source_hsolver/test/test_hsolver_pw.cpp index d484a5ed92..2e6a07fffb 100644 --- a/source/source_hsolver/test/test_hsolver_pw.cpp +++ b/source/source_hsolver/test/test_hsolver_pw.cpp @@ -375,9 +375,15 @@ TEST_F(TestHSolverPW, SolveLcaoInPW) { elecstate_test.ekb.c[1] = 2.0; hsolver::HSolverLIP> hs_f_lip - = hsolver::HSolverLIP>(&pwbk, PARAM.sys.use_uspp); + = hsolver::HSolverLIP>(&pwbk, + PARAM.sys.use_uspp, + PARAM.input.basis_type, + PARAM.input.calculation); hsolver::HSolverLIP> hs_d_lip - = hsolver::HSolverLIP>(&pwbk, PARAM.sys.use_uspp); + = hsolver::HSolverLIP>(&pwbk, + PARAM.sys.use_uspp, + PARAM.input.basis_type, + PARAM.input.calculation); hs_f_lip.solve(&hamilt_test_f, psi_test_cf, &elecstate_test,transform_test_cf, true,0.0,0); EXPECT_DOUBLE_EQ(hsolver::DiagoIterAssist>::avg_iter, 0.0); for (int i = 0; i < psi_test_cf.size(); i++) diff --git a/source/source_lcao/LCAO_set.cpp b/source/source_lcao/LCAO_set.cpp index c27b4485f6..ad5345585e 100644 --- a/source/source_lcao/LCAO_set.cpp +++ b/source/source_lcao/LCAO_set.cpp @@ -234,7 +234,9 @@ void LCAO_domain::init_chg_hr( ks_solver, PARAM.globalv.kpar_lcao, PARAM.globalv.nlocal, - PARAM.inp.nelec); + PARAM.inp.nbands, + PARAM.inp.nelec, + PARAM.inp.device == "gpu"); hsolver_lcao_obj.solve(p_hamilt, psi, pelec, dm, chr, nspin, 0); } diff --git a/source/source_lcao/module_deltaspin/cal_mw_from_lambda.cpp b/source/source_lcao/module_deltaspin/cal_mw_from_lambda.cpp index 65cbd88608..fc2fdc06b3 100644 --- a/source/source_lcao/module_deltaspin/cal_mw_from_lambda.cpp +++ b/source/source_lcao/module_deltaspin/cal_mw_from_lambda.cpp @@ -525,7 +525,9 @@ void spinconstrain::SpinConstrain>::cal_mw_from_lambda( PARAM.inp.ks_solver, PARAM.globalv.kpar_lcao, PARAM.globalv.nlocal, - PARAM.inp.nelec); + PARAM.inp.nbands, + PARAM.inp.nelec, + PARAM.inp.device == "gpu"); if (this->nspin_ == 2) { dynamic_cast, double>>*>(this->p_operator) diff --git a/source/source_psi/psi_prepare.cpp b/source/source_psi/psi_prepare.cpp index eada5f13d1..5458d58992 100644 --- a/source/source_psi/psi_prepare.cpp +++ b/source/source_psi/psi_prepare.cpp @@ -198,7 +198,9 @@ void PSIPrepare::initialize_psi(Psi>* psi, nbands_start, nbasis, *(kspw_psi), - etatom.data()); + etatom.data(), + this->basis_type, + PARAM.inp.calculation); } else {