Skip to content

chore(rocm): ROCm 7.14 / TheRock packaging support for AMD builds (base bump + build compat)#10978

Open
walcz-de wants to merge 3 commits into
mudler:masterfrom
walcz-de:feat/rocm-7.14-bump
Open

chore(rocm): ROCm 7.14 / TheRock packaging support for AMD builds (base bump + build compat)#10978
walcz-de wants to merge 3 commits into
mudler:masterfrom
walcz-de:feat/rocm-7.14-bump

Conversation

@walcz-de

@walcz-de walcz-de commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

Brings LocalAI's ROCm / hipblas build up to ROCm 7.14 and makes it compatible with AMD's new "TheRock" packaging.

This is a platform-wide change for AMD, not a single-GPU one. The base image, package layout and library paths it fixes apply to every architecture in AMDGPU_TARGETS (gfx908 / gfx90a / gfx942 / gfx1030 / gfx1100 / gfx1101 / gfx1102 / gfx1151 / gfx1200 / gfx1201) — the ROCm/hipblas images simply do not build (and then do not load) on ROCm 7.14 until these are fixed, regardless of the target card. gfx1151 / Strix Halo is only the AMD hardware I had available to runtime-validate on.

  • Bumps the ROCm base rocm/dev-ubuntu-24.04:7.2.1 → :7.14.0-full across backend-matrix.yml, base-images.yml, image.yml/image-pr.yml, the .agents/* docs and the Dockerfile.base-grpc-builder comment.
  • Fixes the ROCm 7.x / TheRock build breakages (below).

Follow-up to #10883, companion to the docs PR #9229.

Why — ROCm 7.x switched to "TheRock" packaging

See AMD's TheRock transition guide. A naive base bump breaks the AMD build in three ways (all fixed here, all architecture-agnostic):

  1. Legacy -dev metapackages removed. apt-get install hipblas-dev hipblaslt-dev rocblas-dev fails (Unable to locate package) — they are consolidated + arch-split (amdrocm-blas<ver>-gfx*, amdrocm-blas-dev, …). The -full base already ships them, so the install is dropped in both places that used the legacy names (Dockerfile requirements-drivers + .docker/install-base-deps.sh). The amd capability marker + ldconfig are kept.

  2. ROCm libs are scattered and not on the linker path. TheRock places libs under /opt/rocm/lib, /opt/rocm/lib/rocm_sysdeps/lib and /opt/rocm/llvm/lib with no ld.so.conf.d entry. The ldd-based GPU-lib packaging (scripts/build/package-gpu-libs.sh) then silently skips librocm_kpack, librocm_sysdeps_*, rocBLAS, hipBLASLt, etc., and the built backend fails to load at runtime on the GPU (... cannot open shared object file). CI does not catch this — it only surfaces when the backend loads on an AMD GPU. Fixed by registering the three ROCm lib dirs in /etc/ld.so.conf.d/rocm.conf before ldconfig, in both blocks.

  3. The base bump itself. base-images.yml rebuilds the base-grpc-rocm-amd64 gRPC cache on 7.14 so it stays ABI-compatible with the new runtime base.

Testing

The build changes are architecture-agnostic — they fix the ROCm/hipblas image for all AMD architectures. I runtime-validated on the AMD hardware I have, gfx1151 / Strix Halo (Ryzen AI Max+ 395 / Radeon 8060S): the core image and the llama-cpp backend build, all ROCm libraries bundle (verified ldd resolves everything), and inference runs on the GPU — model loaded, tokens generated, no missing-lib errors.

Since I only have one AMD GPU to test on, a maintainer or CI with other AMD cards (CDNA gfx908/90a/942, RDNA gfx1030/1100/…) should confirm — especially fix #2, which is a runtime-only failure.

(Aside, out of scope: GGML_HIP_ROCWMMA_FATTN is not enabled in the llama-cpp build flags — a potential RDNA3 flash-attention perf win.)

🤖 Generated with Claude Code

https://claude.ai/code/session_01NK35FML8epTCLBGwCor9et

@walcz-de walcz-de mentioned this pull request Jul 20, 2026
Open
@walcz-de walcz-de changed the title chore(rocm): ROCm 7.14 / gfx1151 (Strix Halo) support — base bump + TheRock build compat chore(rocm): ROCm 7.14 / TheRock packaging support for AMD builds (base bump + build compat) Jul 20, 2026
walcz-de added 2 commits July 20, 2026 16:37
@walcz-de
chore(rocm): bump ROCm base image 7.2.1 -> 7.14.0-full (native gfx115…
c3618f2 
…1 / Strix Halo)

ROCm 7.14 adds native gfx1151 (Strix Halo / RDNA 3.5) runtime support. The
prebuilt ROCm images pinned rocm/dev-ubuntu-24.04:7.2.1, which predates gfx1151
support, so the published -gpu-rocm-hipblas-* images do not run on that hardware
even though gfx1151 is already in AMDGPU_TARGETS.

Bump the pin to rocm/dev-ubuntu-24.04:7.14.0-full everywhere it is referenced:
- .github/backend-matrix.yml  (all hipblas backend build entries)
- .github/workflows/base-images.yml  (base-grpc-rocm-amd64 gRPC cache is rebuilt
  on 7.14 so it stays ABI-compatible with the new runtime base)
- .github/workflows/image.yml + image-pr.yml  (core hipblas image)
- .agents/*.md examples + backend/Dockerfile.base-grpc-builder comment

Verified on gfx1151 / Strix Halo: a full ROCm 7.14 stack (LocalAI + llama.cpp
hipblas, vLLM, embeddings/reranker) runs natively, no HSA_OVERRIDE_GFX_VERSION
needed. Other AMD architectures (gfx908/90a/942/1030/1100/1200/1201) build on
the same base and are covered by CI; I do not have that hardware to test at runtime.

Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
@walcz-de
fix(rocm): ROCm 7.x / TheRock build compat (drop legacy -dev pkgs, re…
cd4e5f4 
…gister lib paths)

ROCm 7.x moved to 'TheRock' packaging. Bumping the base to
rocm/dev-ubuntu-24.04:7.14.0-full surfaces two build breakages, fixed here in
ALL THREE spots that installed the legacy ROCm dev metapackages:

1) Legacy hipblas-dev / hipblaslt-dev / rocblas-dev metapackages were removed
   (consolidated + arch-split: amdrocm-blas<ver>-gfx*, amdrocm-blas-dev, ...).
   'apt-get install' of the old names fails. The -full base already ships the
   BLAS dev libs+headers, so the install is dropped in:
     - Dockerfile (requirements-drivers)          -> core image
     - .docker/install-base-deps.sh (section 6)   -> C++ backend builder
     - backend/Dockerfile.python (hipblas block)  -> Python backends (vllm,
       sglang, transformers, diffusers, kokoro, ...)

2) TheRock scatters the ROCm libs across /opt/rocm/lib,
   /opt/rocm/lib/rocm_sysdeps/lib and /opt/rocm/llvm/lib with no ld.so.conf.d
   entry, so the dynamic linker can't resolve them and the built backends fail
   at RUNTIME ('... cannot open shared object file') — a failure CI never sees
   because it only surfaces when a backend loads on an AMD GPU. Fixed by
   registering all three lib dirs in /etc/ld.so.conf.d/rocm.conf before ldconfig
   in all three blocks.

Validated end-to-end on gfx1151 / Strix Halo (build + GPU inference, llama-cpp).
Ref: https://rocm.docs.amd.com/en/latest/about/transition-guide-TheRock.html

Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
@walcz-de
walcz-de force-pushed the feat/rocm-7.14-bump branch from a21c8a5 to cd4e5f4 Compare July 20, 2026 14:38
@localai-org-maint-bot

localai-org-maint-bot commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

I reviewed this and found one missed ROCm 7.14 build path: backend/Dockerfile.golang still installed the removed hipblas-dev, hipblaslt-dev, and rocblas-dev packages. I pushed the matching TheRock linker-path migration as 8c08804a.

The content checks pass locally (git diff --check, shell syntax, no remaining obsolete ROCm package installs, and no stale 7.2.1 build pins), but DCO correctly rejects my bot-authored commit. @walcz-de, please amend/recreate that commit under your identity with your own Signed-off-by certification and force-push the branch. Once DCO and the substantive CI checks are green, I can finish the review and flag @mudler.

@walcz-de @localai-org-maint-bot
fix(rocm): update Go backend Dockerfile for TheRock packaging
23b2a98 
Apply the same ROCm 7.14 dependency + linker-path migration to
backend/Dockerfile.golang: the removed legacy hipblas-dev / hipblaslt-dev /
rocblas-dev metapackages are no longer installed (the rocm/dev-ubuntu-*:*-full
base already ships them), and TheRock's scattered ROCm lib dirs are registered
in /etc/ld.so.conf.d/rocm.conf before ldconfig so cgo-based Go backends resolve
the ROCm runtime libraries at build/link and runtime.

Identified by the LocalAI maintenance bot; re-authored under my identity per DCO.

Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
@walcz-de
walcz-de force-pushed the feat/rocm-7.14-bump branch from 8c08804 to 23b2a98 Compare July 20, 2026 15:32
@walcz-de

walcz-de commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the catch — backend/Dockerfile.golang was indeed a fourth spot with the removed legacy metapackages. I've re-authored your fix under my identity with my own Signed-off-by (kept you as Co-authored-by) and force-pushed (23b2a98c), so DCO should be green now.

That makes it four ROCm build paths migrated to TheRock: the core Dockerfile, .docker/install-base-deps.sh, backend/Dockerfile.python, and now backend/Dockerfile.golang.

@mudler
mudler enabled auto-merge (squash) July 20, 2026 16:57
This was referenced Jul 20, 2026
Draft
Draft
walcz-de added a commit to walcz-de/LocalAI that referenced this pull request Jul 21, 2026
@walcz-de
feat(backend/python): build vllm from source for AMD consumer/RDNA GPUs
251972b 
vllm's ROCm install pulls a prebuilt wheel from wheels.vllm.ai/rocm, but that
wheel exists only for CDNA data-center arches (gfx942/gfx950) and is pinned to a
rocm7.2.3 torch that can't enumerate consumer/RDNA GPUs (device_count == 0 on
e.g. gfx1151). No prebuilt vllm wheel exists for these GPUs anywhere, so build it
from source against AMD's ROCm 7.14 torch -- reproducing the recipe AMD's own
rocm/vllm:*_rdna_* image uses.

- torch[device-gfx<arch>] + rocm-sdk-devel (hipcc + the HIP CMake packages the
  runtime SDK lacks) from AMD's multi-arch index; ROCM_PATH / CMAKE_PREFIX_PATH
  resolved from the SDK's own `rocm-sdk path` CLI -- self-contained, no system
  /opt/rocm needed.
- vLLM detects ROCm at runtime via `import amdsmi`; register the SDK's in-place
  amd_smi with a .pth. A pip copy breaks its relative libamd_smi.so lookup, and
  forcing the lib via LD_LIBRARY_PATH shadows torch's own ROCm runtime and zeroes
  device_count.
- run.sh exposes the bundled amdclang as CC/CXX: Triton JIT-compiles kernels at
  first inference and the runtime image ships hipcc but not cc/gcc, so without it
  inference dies with "Failed to find C compiler". Mirrors the CPU toolchain
  block just above it.
- arch from AMDGPU_TARGETS (added in the multi-arch-torch change this stacks on),
  defaulting to gfx1151.

Validated on gfx1151: builds from a single `make`, loads natively via LocalAI
(no external gRPC) and generates. Only gfx1151 was hardware-tested; other AMD
arches use the same mechanism but are unverified.

Depends on mudler#10978 (ROCm 7.14 base) and the multi-arch-torch change it stacks on.

Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
(cherry picked from commit 836fe95)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

windows compatibility?

2 participants

@walcz-de @localai-org-maint-bot