Build gpu-driver-container images for Flatcar by robinschneider · Pull Request #1 · flatcar/gpu-driver-container

robinschneider · 2026-07-15T14:21:51Z

This PR adds support for Flatcar for the NVIDIA GPU Operator.
It features both pre-compiled images as well as a build container that compiles the driver modules at runtime for arbitrary Flatcar releases.
Both image variants ship the NVIDIA Fabric Manager, required for multi-GPU nodes such as HGX H100 with 4 or 8 GPUs on a single board.

The repository is standalone (no longer a fork of NVIDIA's gpu-driver-container that needs to be kept in sync) and intentionally minimal: a single Dockerfile builds both variants — when the FLATCAR_VERSION build arg is set, the kernel modules for that Flatcar release are compiled and packed into the image at build time; when unset, a generic image is built that compiles the modules at runtime.

All images are built using GitHub Actions and published to ghcr.io/flatcar/driver:

Pushes to any branch publish dev images with a commit-based prefix (dev-<sha8>-<driver_version>-flatcar[<VERSION_ID>]).
GitHub Releases publish clean tags (<driver_version>-flatcar for the build container, <driver_version>-flatcar<VERSION_ID> for pre-compiled images) matching the GPU Operator's default image reference scheme.

Fixes flatcar/Flatcar#1928 flatcar/Flatcar#1962

Testing

build container (driver compiled at runtime)

Deploy the GPU Operator with the base image tag (the operator appends the
-flatcar<VERSION_ID> suffix derived from the node labels):

export BASE_TAG=595.71.05-flatcar
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.repository=ghcr.io/flatcar \
  --set driver.image=driver \
  --set driver.version="${BASE_TAG}" \
  --set driver.imagePullPolicy=Always \
  --set toolkit.installDir=/opt/nvidia-runtime

Note: toolkit.installDir must point to a writable path — the default /usr/local is read-only on Flatcar.

The driver container downloads the Flatcar kernel sources and compiles the kernel
modules on the node. This is visible in the driver pod logs (takes a few minutes):

$ kubectl -n gpu-operator logs -l app=nvidia-driver-daemonset
Checking NVIDIA driver packages...
Installing the Flatcar kernel sources into the development environment...
...
Building kernel modules:
  [##############################] 100%
Loading NVIDIA driver kernel modules...
Done, now waiting for signal

pre-compiled image

The pre-compiled image ships kernel modules built for one specific Flatcar release
(tag suffix flatcar<VERSION_ID>, e.g. flatcar4593.2.0 / kernel 6.12.81-flatcar).
Deploy it by using the version prefix without the -flatcar suffix:

export BASE_TAG=595.71.05
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.repository=ghcr.io/flatcar \
  --set driver.image=driver \
  --set driver.version="${BASE_TAG}" \
  --set driver.imagePullPolicy=Always \
  --set toolkit.installDir=/opt/nvidia-runtime

That the pre-compiled path is actually used (no download, no compilation) can be
verified in the driver pod logs — the packaged kernel interface is matched against
the running kernel's /proc/version and installed directly, the pod becomes Ready
within seconds:

$ kubectl -n gpu-operator logs -l app=nvidia-driver-daemonset
Checking NVIDIA driver packages...
kernel interface matches.
Found NVIDIA driver package nvidia-modules-6.12.81
Installing NVIDIA driver kernel modules...
Loading NVIDIA driver kernel modules...
Done, now waiting for signal

If the node runs a Flatcar release that does not match the image, the container
falls back to building the modules at runtime (same behavior as the build container).

fabric manager (NVSwitch systems, e.g. HGX H100 8-GPU)

On NVSwitch based nodes the driver container detects the switch devices
(/proc/driver/nvidia-nvswitch/devices) and starts the NVIDIA Fabric Manager
automatically — with both image types:

$ kubectl -n gpu-operator logs -l app=nvidia-driver-daemonset
...
Starting NVIDIA persistence daemon...
Starting NVIDIA fabric manager daemon...
Done, now waiting for signal

A successfully initialized fabric is reported by nvidia-smi:

$ kubectl -n gpu-operator exec ds/nvidia-driver-daemonset -- nvidia-smi -q | grep -A2 "^    Fabric"
    Fabric
        State                                          : Completed
        Status                                         : Success

Verified on an HGX H100 8-GPU node (Flatcar 4593.2.0): all 8 GPUs visible via
nvidia-smi -L, CUDA validator pods complete, cuda-vectoradd sample passes.
On nodes without NVSwitch (e.g. L40S) the fabric manager start is skipped.

Example workload

Applies to both image types — schedule a CUDA sample on a GPU node:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: Never
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
      resources:
        limits:
          nvidia.com/gpu: 1
EOF

$ kubectl logs cuda-vectoradd
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Copilot

Pull request overview

This PR adds Flatcar Container Linux support to the GPU driver container build/publish flow, including support for both “build-at-runtime” and precompiled kernel-module images, plus NVSwitch/Fabric Manager enablement for multi-GPU systems.

Changes:

Adds a Flatcar-aware nvidia-driver entrypoint script that can fetch Flatcar kernel sources, pack precompiled interfaces, and start Fabric Manager on NVSwitch systems.
Introduces a Makefile and GitHub Actions workflow to build/push generic and precompiled Flatcar driver images to GHCR using GPU Operator-compatible tags.
Adds Dockerfile and documentation updates/scripts to build and run the images.

Reviewed changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`.github/workflows/image.yaml`	CI build/push pipeline for generic + precompiled Flatcar image variants and tagging.
`Makefile`	Local build targets and tag conventions for generic vs precompiled Flatcar images.
`Dockerfile`	Container image build, driver/userspace install, and optional precompile at build time via `FLATCAR_VERSION`.
`nvidia-driver`	Runtime logic for fetching Flatcar artifacts, precompile matching/packing, driver install/load, and Fabric Manager start.
`README.md`	Flatcar-focused usage and GPU Operator deployment documentation.
`build.sh` / `run.sh` / `save.sh` / `update.sh`	Helper scripts for local image build/run/update workflows.
`.idea/.gitignore`	IDE metadata ignore rules.
`empty`	Placeholder file used for `PUBLIC_KEY` build arg default.

Files not reviewed (1)

.idea/.gitignore: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+          BASE_TAG="${VERSION:+${VERSION}-}${{ matrix.driver_version }}-flatcar"
+          for FLATCAR_VERSION in ${{ needs.setup.outputs.flatcar_versions_plain }}; do
+            docker buildx imagetools create \
+              -t "${IMAGE_NAME}:${BASE_TAG}-flatcar${FLATCAR_VERSION}" \


+
+Commands:
+  init   [-a | --accept-license]
+  update [-k | --kernel VERSION] [-s | --sign KEYID] [-t | --tag TAG]


+command=$1; shift
+case "${command}" in
+    init) options=$(getopt -l accept-license -o a -- "$@") ;;
+    update) options=$(getopt -l kernel:,sign:,tag: -o f:k:s:t: -- "$@") ;;


+for opt in ${options}; do
+    case "$opt" in
+    -a | --accept-license) ACCEPT_LICENSE="yes"; shift 1 ;;
+    -k | --kernel) KERNEL_VERSION=$2; shift 2 ;;
+    -f | --flatcar) FLATCAR_VERSION=$2; shift 2 ;;
+    -s | --sign) PRIVATE_KEY=$2; shift 2 ;;
+    -t | --tag) PACKAGE_TAG=$2; shift 2 ;;
+    --) shift; break ;;
+    esac
+done


+RUN if [ "$TARGETARCH" = "amd64" ]; then \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy main universe" > /etc/apt/sources.list && \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy-updates main universe" >> /etc/apt/sources.list && \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy-security main universe" >> /etc/apt/sources.list && \
+        usermod -o -u 0 -g 0 _apt ; \
+    elif [ "$TARGETARCH" = "arm64" ]; then \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy main universe" > /etc/apt/sources.list && \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-updates main universe" >> /etc/apt/sources.list && \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-security main universe" >> /etc/apt/sources.list && \
+        usermod -o -u 0 -g 0 _apt ; \
+    else \
+        echo "TARGETARCH doesn't match a known arch target" ; \
+        exit 1 ; \
+    fi


+set -x
+
+FLATCAR=3815.2.0
+docker rm -f nvidia-driver
+docker run -it --privileged --pid=host \
+    --entrypoint nvidia-driver \
+    -v /run/nvidia:/run/nvidia:shared \
+    -v /var/log:/var/log \
+    -v /dev/log:/dev/log \
+    --name nvidia-driver \
+    nvidia/nvidia-driver-flatcar:latest update -f ${FLATCAR}


+$ git clone https://gitlab.com/nvidia/container-images/driver.git \
+    && cd driver/flatcar


Copilot

Pull request overview

Copilot reviewed 9 out of 14 changed files in this pull request and generated 13 comments.

Files not reviewed (4)

.idea/.gitignore: Generated file
.idea/gpu-driver-container.iml: Generated file
.idea/modules.xml: Generated file
.idea/vcs.xml: Generated file

+RUN if [ "$TARGETARCH" = "amd64" ]; then \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy main universe" > /etc/apt/sources.list && \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy-updates main universe" >> /etc/apt/sources.list && \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy-security main universe" >> /etc/apt/sources.list && \
+        usermod -o -u 0 -g 0 _apt ; \
+    elif [ "$TARGETARCH" = "arm64" ]; then \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy main universe" > /etc/apt/sources.list && \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-updates main universe" >> /etc/apt/sources.list && \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-security main universe" >> /etc/apt/sources.list && \
+        usermod -o -u 0 -g 0 _apt ; \
+    else \
+        echo "TARGETARCH doesn't match a known arch target" ; \
+        exit 1 ; \
+    fi


+RUN if [ "$TARGETARCH" = "amd64" ]; then dpkg --add-architecture i386 ; fi && \
+    apt-get update && apt-get install -y --no-install-recommends \
+        apt-transport-https \
+        apt-utils \
+        bc \
+        binutils \
+        build-essential \
+        ca-certificates \
+        curl \
+        gcc-14 \
+        gnupg2 \
+        jq \
+        kmod \
+        libelf-dev \
+        libssl-dev \
+        patchelf \
+        pahole \
+        file \
+        fdisk \
+        software-properties-common && \
+    rm -rf /var/lib/apt/lists/*


+# Add NGC DL license from the CUDA image
+RUN mkdir /licenses && mv /NGC-DL-CONTAINER-LICENSE /licenses/NGC-DL-CONTAINER-LICENSE
+
+ENV CC=gcc-14


+RUN curl -fsSL --retry 5 --retry-delay 5 --retry-all-errors -o /usr/local/bin/donkey https://github.com/3XX0/donkey/releases/download/v1.1.0/donkey && \
+    curl -fsSL --retry 5 --retry-delay 5 --retry-all-errors -o /usr/local/bin/extract-vmlinux https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux && \
+    chmod +x /usr/local/bin/donkey /usr/local/bin/extract-vmlinux


+          BASE_TAG="${VERSION:+${VERSION}-}${{ matrix.driver_version }}-flatcar"
+          for FLATCAR_VERSION in ${{ needs.setup.outputs.flatcar_versions_plain }}; do
+            docker buildx imagetools create \
+              -t "${IMAGE_NAME}:${BASE_TAG}-flatcar${FLATCAR_VERSION}" \
+              "${IMAGE_NAME}:${BASE_TAG}"
+          done


+#!/bin/bash
+
+set -xeuo pipefail
+
+: ${DRIVER_VERSION:="$(sed -n "s/^ARG DRIVER_VERSION=//p" Dockerfile)"}
+
+docker build --pull \
+    --tag nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} \
+    --file Dockerfile .
+docker tag nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} nvidia/nvidia-driver-flatcar:latest


+docker run -d --privileged --pid=host \
+     -v /run/nvidia:/run/nvidia:shared \
+     -v /var/log:/var/log \
+     -v /dev/log:/dev/log \
+     -v /etc/os-release:/host-etc/os-release \
+     --name nvidia-driver \
+     nvidia/nvidia-flatcar-${VERSION_ID}:${DRIVER_VERSION}


+docker commit \
+    --change='ENTRYPOINT ["nvidia-driver", "init"]' \
+    nvidia-driver nvidia/nvidia-flatcar-${VERSION_ID}:${DRIVER_VERSION}


+Clone this repository and build a driver container image using the following as an example:
+
+```bash
+$ git clone https://gitlab.com/nvidia/container-images/driver.git \
+    && cd driver/flatcar
+```
+
+```bash
+$ DRIVER_VERSION=460.32.03 
+$ docker build --pull \
+    --build-arg DRIVER_VERSION=$DRIVER_VERSION \
+    --tag nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} \
+    --file Dockerfile .
+```


+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="VcsDirectoryMappings">
+    <mapping directory="" vcs="Git" />
+    <mapping directory="$PROJECT_DIR$" vcs="Git" />
+  </component>
+</project>


Signed-off-by: Robin Schneider <mail@robinschneider.dev>

Copilot

Pull request overview

Copilot reviewed 9 out of 14 changed files in this pull request and generated 15 comments.

Files not reviewed (4)

.idea/.gitignore: Generated file
.idea/gpu-driver-container.iml: Generated file
.idea/modules.xml: Generated file
.idea/vcs.xml: Generated file

+RUN if [ "$TARGETARCH" = "amd64" ]; then \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy main universe" > /etc/apt/sources.list && \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy-updates main universe" >> /etc/apt/sources.list && \
+        echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu/ jammy-security main universe" >> /etc/apt/sources.list && \
+        usermod -o -u 0 -g 0 _apt ; \
+    elif [ "$TARGETARCH" = "arm64" ]; then \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy main universe" > /etc/apt/sources.list && \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-updates main universe" >> /etc/apt/sources.list && \
+        echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-security main universe" >> /etc/apt/sources.list && \
+        usermod -o -u 0 -g 0 _apt ; \
+    else \
+        echo "TARGETARCH doesn't match a known arch target" ; \
+        exit 1 ; \
+    fi


+ARG BASE_URL=https://us.download.nvidia.com/tesla
+ARG DRIVER_VERSION=580.95.05
+ENV DRIVER_VERSION=$DRIVER_VERSION


+usage() {
+    cat >&2 <<EOF
+Usage: $0 COMMAND [ARG...]
+
+Commands:
+  init   [-a | --accept-license]
+  update [-k | --kernel VERSION] [-s | --sign KEYID] [-t | --tag TAG]
+EOF
+    exit 1
+}


+case "${command}" in
+    init) options=$(getopt -l accept-license -o a -- "$@") ;;
+    update) options=$(getopt -l kernel:,sign:,tag: -o f:k:s:t: -- "$@") ;;
+    *) usage ;;
+esac


+    case "$versionid" in
+      *.2.*)
+        channel=stable;;
+      *.1.*)
+        channel=beta;;
+      *.0.*)
+        channel=alpha;;
+    esac
+
+    case "$(uname -m)" in
+      x86_64)
+        board=amd64-usr;;
+      aarch64)
+        board=arm64-usr;;
+    esac


+$ DRIVER_VERSION=460.32.03 
+$ docker build --pull \
+    --build-arg DRIVER_VERSION=$DRIVER_VERSION \
+    --tag nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} \
+    --file Dockerfile .
+```


+$ docker run -d --privileged --pid=host \
+    -v /run/nvidia:/run/nvidia:shared \
+    -v /tmp/nvidia:/var/log \
+    -v /usr/lib64/modules:/usr/lib64/modules \
+    --name nvidia-driver \
+    nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} update
+```


+docker build --pull \
+    --tag nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} \
+    --file Dockerfile .
+docker tag nvidia/nvidia-driver-flatcar:${DRIVER_VERSION} nvidia/nvidia-driver-flatcar:latest


+     --name nvidia-driver \
+     nvidia/nvidia-flatcar-${VERSION_ID}:${DRIVER_VERSION}


+docker commit \
+    --change='ENTRYPOINT ["nvidia-driver", "init"]' \
+    nvidia-driver nvidia/nvidia-flatcar-${VERSION_ID}:${DRIVER_VERSION}


Copilot AI review requested due to automatic review settings July 15, 2026 14:21

Copilot started reviewing on behalf of robinschneider July 15, 2026 14:22 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 15, 2026 14:36

robinschneider force-pushed the container branch from 1814d7b to 9fd7b0a Compare July 15, 2026 14:36

Copilot started reviewing on behalf of robinschneider July 15, 2026 14:37 View session

robinschneider mentioned this pull request Jul 15, 2026

Build images for Flatcar flatcar-hub/gpu-driver-container#1

Open

Copilot AI reviewed Jul 15, 2026

View reviewed changes

build gpu-driver-container

bd2aa17

Signed-off-by: Robin Schneider <mail@robinschneider.dev>

Copilot AI review requested due to automatic review settings July 15, 2026 14:50

robinschneider force-pushed the container branch from 9fd7b0a to bd2aa17 Compare July 15, 2026 14:50

Copilot started reviewing on behalf of robinschneider July 15, 2026 14:51 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Build gpu-driver-container images for Flatcar#1

Build gpu-driver-container images for Flatcar#1
robinschneider wants to merge 1 commit into
flatcar:mainfrom
robinschneider:container

robinschneider commented Jul 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		$ git clone https://gitlab.com/nvidia/container-images/driver.git \
		&& cd driver/flatcar

		--name nvidia-driver \
		nvidia/nvidia-flatcar-${VERSION_ID}:${DRIVER_VERSION}

Uh oh!

Conversation

robinschneider commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

build container (driver compiled at runtime)

pre-compiled image

fabric manager (NVSwitch systems, e.g. HGX H100 8-GPU)

Example workload

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robinschneider commented Jul 15, 2026 •

edited

Loading