Skip to content

09-AI-Server.md

Latest commit

 

History

History
436 lines (357 loc) · 16.3 KB

09-AI-Server.md

File metadata and controls

436 lines (357 loc) · 16.3 KB
Raw

AI Server Setup with Incus & Lemonade

This guide covers setting up a high-performance local AI server using Lemonade inside an Incus container. The critical factor for performance is successfully passing through the GPU and NPU to the container.

Update (March 2026): Linux NPU support is now available for AMD XDNA 2 devices through FastFlowLM. On current Ryzen AI Linux systems, the practical setup is to keep Lemonade + llama.cpp for GPU-backed serving and add FastFlowLM when you want to run supported models directly on the NPU. See the FastFlowLM Linux guides for the current support matrix and package sources.

1. Container Configuration

NPU Passthrough

First, check if the NPU is recognized on the host system:

ls -l /dev/accel/accel*

Create a new container or update an existing one to add the NPU device. Replace gipDLVmAIServ with your container name:

sudo incus config device add gipDLVmAIServ xdna-npu unix-char path=/dev/accel/accel0

Grant necessary permissions (ensure the GID matches the render group, typically 992 or similar):

getent group render
sudo incus config device set gipDLVmAIServ xdna-npu gid=992 mode=0660
sudo incus config device add gipDLVmAIServ dev_kfd unix-char source=/dev/kfd path=/dev/kfd gid=992 mode=0660

GPU Passthrough

Add the GPU device. On Ubuntu 24.04 and newer Incus versions, the container often requires privileged status and specific syscall intercepts to access hardware accelerators correctly.

sudo incus config device add gipDLVmAIServ mygpu gpu
sudo incus config set gipDLVmAIServ security.privileged true
sudo incus config set gipDLVmAIServ security.syscalls.intercept.mknod true
sudo incus config set gipDLVmAIServ security.syscalls.intercept.setxattr true
sudo incus config set gipDLVmAIServ limits.kernel.memlock unlimited

Resource Limits (CPU/RAM)

Configure resources via the Incus UI or CLI.

  • CPU: Primarily used for model loading. During inference, the GPU handles the workload. Testing suggests that assigning ~6 cores is sufficient; allocating all cores does not significantly improve performance as Lemonade may not utilize them all.
  • Memory: Allocate sufficient RAM (e.g., 54 GiB) depending on the models you plan to run.

Example Profile Configuration (YAML):

architecture: x86_64
config:
  image.os: Ubuntu
  image.release: noble
  limits.cpu: '6'
  limits.kernel.memlock: unlimited
  limits.memory: 54GiB
  security.privileged: 'true'
  security.syscalls.intercept.mknod: 'true'
  security.syscalls.intercept.setxattr: 'true'
devices:
  dev_kfd:
    gid: '992'
    mode: '0660'
    path: /dev/kfd
    source: /dev/kfd
    type: unix-char
  mygpu:
    type: gpu
  xdna-npu:
    gid: '992'
    mode: '0660'
    path: /dev/accel/accel0
    type: unix-char
  root:
    path: /
    pool: default
    size: 120GiB
    type: disk

2. Server Installation

Start the container and open a shell session inside it.

FastFlowLM NPU Runtime

FastFlowLM can now use the AMD XDNA 2 NPU directly on Linux. This is separate from the ROCm/Vulkan path used by Lemonade with llama.cpp.

References:

Host prerequisites

Install the AMD XRT and XDNA kernel driver packages on the host so the NPU device is available and passed through to the container:

sudo add-apt-repository ppa:amd-team/xrt
sudo apt update
sudo apt install libxrt-npu2 amdxdna-dkms
sudo reboot

Container prerequisites

Install the NPU runtime package inside the Incus container as well:

sudo add-apt-repository ppa:amd-team/xrt
sudo apt update
sudo apt install libxrt-npu2
sudo reboot

Install FastFlowLM

Download the current Ubuntu package from the FastFlowLM releases page. For example:

wget https://github.com/FastFlowLM/FastFlowLM/releases/download/v0.9.35/fastflowlm_0.9.35_ubuntu24.04_amd64.deb
sudo apt install ./fastflowlm*.deb

Required memlock configuration

FastFlowLM validation will fail if the container cannot lock enough memory for NPU execution.

Check the current limit:

ulimit -l

If it is not unlimited, configure all of the following inside the container:

  1. Ensure the Incus container has the kernel memlock limit enabled:
sudo incus config set gipDLVmAIServ limits.kernel.memlock unlimited
  1. Edit /etc/security/limits.conf and add:
* soft memlock unlimited
* hard memlock unlimited
  1. Edit /etc/systemd/system.conf and /etc/systemd/user.conf, then set:
DefaultLimitMEMLOCK=infinity
  1. Reboot the container after changing the limits.

Validate NPU access

After reboot, verify that the NPU, firmware, driver, and memlock settings are all detected:

flm validate

Example output:

[Linux]  Kernel: 6.17.0-108014-tuxedo
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.6
[Linux]  Memlock Limit: infinity

If flm validate does not report the NPU or shows a finite memlock limit, re-check the host driver installation, the Incus device passthrough, and the systemd/limits configuration above before troubleshooting FastFlowLM itself.

ROCm Drivers

Install the AMD ROCm drivers (essential for GPU acceleration). Reference: ROCm Installation on Linux

apt-get update && apt-get install wget
mkdir --parents --mode=0755 /etc/apt/keyrings
wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null

sudo tee /etc/apt/sources.list.d/rocm.list << EOF
deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/7.1.1 noble main
deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/graphics/7.1.1/ubuntu noble main
EOF

sudo tee /etc/apt/preferences.d/rocm-pin-600 << EOF
Package: *
Pin: release o=repo.radeon.com
Pin-Priority: 600
EOF

sudo apt update
apt install rocm rocm-hip-runtime rocm-hip-libraries

Environment Variables Configure the library path and force the graphics version (required for Radeon 890M / gfx1150 support). Add these lines to your ~/.bashrc to make them persistent:

export LD_LIBRARY_PATH=/opt/rocm/lib:$LD_LIBRARY_PATH
export HSA_OVERRIDE_GFX_VERSION=11.5.0

Apply changes:

source ~/.bashrc

Install Lemonade

Download and install the minimal Lemonade server. Reference: Lemonade Server Docs

sudo add-apt-repository ppa:lemonade-team/stable
sudo apt install lemonade-server
sudo update-pciids

Arrows fixed, async Dialog, TimeSeries with OxyPlot, FillLevel still bugy

3. BIOS & System Tuning (Optional / Troubleshooting)

VRAM Configuration

On AMD Ryzen AI 9 laptops, you can manually set the VRAM size in the BIOS (up to 16GB). Usually, "Auto" is sufficient. I did not observe performance gains by maximizing it manually.

Troubleshooting: LVM Activation Error

Changing the VRAM size in BIOS might cause the container to fail on boot with LVM errors (e.g., exit status 5, Activation of logical volume... is prohibited). This occurs because the system attempts to activate LVM before the kernel finishes reallocating memory addresses for the VRAM.

Fix 1: LVM Configuration Edit /etc/lvm/lvm.conf on the host:

# Locate thin_check_options and uncomment the line to skip mappings
thin_check_options = [ "-q", "--skip-mappings" ]

Fix 2: GRUB Delay Edit /etc/default/grub (or /boot/grub/grub.cfg) to add a boot delay:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash rootdelay=5"

Then update initramfs and reboot:

sudo update-initramfs -u
reboot

After reboot, verify VRAM size:

rocm-smi --showmeminfo vram

4. Usage

Configuration and model locations when lemonade is started from shell/CLI:

~/.cache/lemonade/recipe_options.json
~/.cache/lemonade/user_models.json
~/.cache/lemonade/config.json
~/.cache/huggingface/hub
~/.config/flm/models

Configuration and model locations when lemonade is started with systemd:

/var/lib/lemonade/.cache/lemonade/recipe_options.json
/var/lib/lemonade/.cache/lemonade/user_models.json
/var/lib/lemonade/.cache/lemonade/config.json
/var/lib/lemonade/.cache/huggingface/hub
/var/lib/lemonade/.config/flm/models

When you download models using the Lemonade application, they are downloaded either to your home directory—if you launch Lemonade from the shell—or to /var/lib/lemonade/* if it is started as a systemd service. However, to use a shared directory, you can specify the paths within the config.json file using the models_dir and extra_models_dir parameters. Please note that there are two separate config.json files: one located in your home directory, and another in /var/lib/lemonade for the systemd service.

Ensure you expose the server to the network with "host": "0.0.0.0", in config.json file.

To ensure that the GPU is utilized when Lemonade is launched via systemd, you must provide the necessary environment variables to the service:

sudo systemctl edit lemond

Copy this text and save:

[Service]
User=root
Environment="LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/local/lib:/opt/rocm/lib:/usr/local/share/lemonade-server/llama/roc:/root/.cache/lemonade/bin/llamacpp/vulkan"
Environment="HSA_OVERRIDE_GFX_VERSION=11.5.0"
Environment="HIP_VISIBLE_DEVICES=0"
LimitMEMLOCK=infinity

Restart:

sudo systemctl daemon-reload
sudo systemctl restart lemond.service

Lemonade can run in Service Mode (awaits client requests to load models - recommended) or Run Mode (pre-loads a specific model).

Starting Server

# As service
systemctl start lemond.service
# Or from shell
lemond --host 0.0.0.0 --port 8080

Optional: Examples manual run of an explicit model

1. Nemotron (Vulkan Backend) ROCm may be unstable with certain models like Nemotron. Use Vulkan in these cases. Note that "Reasoning" features are currently disabled (--reasoning-budget 0) for stability.

lemonade-server pull Nemotron-3-Nano-30B-A3B-GGUF
lemonade-server run Nemotron-3-Nano-30B-A3B-GGUF --host 0.0.0.0 --port 8080 --log-level debug --llamacpp vulkan --llamacpp-args "-c 16384 --reasoning-budget 0"

2. Qwen (ROCm Backend) Qwen 3 works well with ROCm and supports reasoning features.

lemonade-server pull Qwen3-30B-A3B-Instruct-2507-GGUF
lemonade-server run Qwen3-30B-A3B-Instruct-2507-GGUF --host 0.0.0.0 --port 8080 --log-level debug --llamacpp rocm --llamacpp-args "-c 16384 --reasoning-budget -1"

3. Directly usage of llama.cpp To load a model directly with llama.cpp find the snapshot location where the model is downloaded from huggingface:

find ~/.cache/huggingface/hub -name "*.gguf"

Then decide if you want to use the vulkan or ROCm backend, which are installed in different locations of lemonade.
With Vulkan (replace the snapshot id with yours!):

/usr/local/share/lemonade-server/llama/vulkan/build/bin/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Nemotron-3-Nano-30B-A3B-GGUF/snapshots/9ad8b366c308f931b2a96b9306f0b41aef9cd405/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf --ctx-size 32768 --temp 0.6 --top-p 0.9 --host 0.0.0.0

With ROCm (replace the snapshot id with yours!):

/usr/local/share/lemonade-server/llama/rocm/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Nemotron-3-Nano-30B-A3B-GGUF/snapshots/9ad8b366c308f931b2a96b9306f0b41aef9cd405/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf --ctx-size 32768 --temp 0.6 --top-p 0.9 --host 0.0.0.0

The Server may not start if the ROCm libraries are not found. Either you switch to the '/usr/local/share/lemonade-server/llama/rocm/' directory and run the llama-server command there.
Or you temprary set environment variables of the library path to:

export LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/share/lemonade-server/llama/rocm:

4. glm-4.7-flash from hugging face

lemonade-server pull user.GLM-4.7-Flash-GGUF --checkpoint unsloth/GLM-4.7-Flash-GGUF:UD-Q4_K_XL --recipe llamacpp
lemonade-server run user.GLM-4.7-Flash-GGUF --host 0.0.0.0 --port 8080 --log-level debug --llamacpp rocm --llamacpp-args "-c 32768 --temp 0.7 --min-p 0.01 --top-p 1.00 --dry-multiplier 1.1 --fit on"

For function calling --jinja has to be used. Unfortunately this is currently not supported by the lemonade-server. This will not work:

lemonade-server run user.GLM-4.7-Flash-GGUF --host 0.0.0.0 --port 8080 --log-level debug --llamacpp rocm --llamacpp-args "-c 32768 --chat-template-file /root/glm4_template.jinja  --jinja --temp 0.7 --min-p 0.01 --top-p 1.00 --reasoning-budget -1 --dry-multiplier 1.1 --fit on"

You only can run it directly with llama-server at the moment:

/usr/local/share/lemonade-server/llama/vulkan/build/bin/llama-server -m /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-GGUF/snapshots/218bcb725e428c5b8c4153bcf5bf7ead738a9799/GLM-4.7-Flash-UD-Q4_K_XL.gguf --jinja --threads -1 --ctx-size 32768 --temp 0.7 --min-p 0.01 --top-p 1.00 --dry-multiplier 1.1 --fit on --host 0.0.0.0
/usr/local/share/lemonade-server/llama/rocm/llama-server -m /root/.cache/huggingface/hub/models--unsloth--GLM-4.7-Flash-GGUF/snapshots/218bcb725e428c5b8c4153bcf5bf7ead738a9799/GLM-4.7-Flash-UD-Q4_K_XL.gguf --jinja --threads -1 --ctx-size 32768 --temp 0.7 --min-p 0.01 --top-p 1.00 --dry-multiplier 1.1 --fit on --host 0.0.0.0

Client Integration

Check Status and open lemonade app: http://<container-ip>:8080

Tools:

  • VS Code:
    • Use the VS Code Insiders Version and Copilot with a custom endpoint Model. Unfortunately the native Copilot provider sends a richer request with additional fields that lemonades llama.cpp's server doesn't recognize for "apiType": "chat-completions". Maybe the future vllm backed will support it. Example setting chatLanguageModels.json: 
      	{
	"name": "lemonade-local",
	"vendor": "customendpoint",
	"apiType": "chat-completions",
	"models": [
		{
			"id": "Qwen3.6-35B-A3B-MTP-GGUF",
			"name": "lemonade-local-Qwen3.6",
			"url": "http://myhost:8080/api/v1/chat/completions",
			"toolCalling": true,
			"vision": false,
			"maxInputTokens": 131072,
			"maxOutputTokens": 16000,
			"settings": {
				"temperature": 0.6,
				"top_p": 0.95,
				"top_k": 20,
				"frequency_penalty": 1.0,
				"presence_penalty": 0.0
			}
		}
	]
},
    • The Standard VS Code Version doesn't support custom models with Copilot but there are two alternatives:
    • Use the unify chat provider extension. Example setting settings.json: 
      "unifyChatProvider.endpoints": [
    {
        "models": [
            {
                "id": "Qwen3.6-35B-A3B-MTP-GGUF",
                "capabilities": {
                    "toolCalling": true
                },
                "name": "lemonade-local-Qwen3.6",
                "maxInputTokens": 131072,
                "settings": {
                    "temperature": 0.6,
                    "top_p": 0.95,
                    "top_k": 20,
                    "frequency_penalty": 1.0,
                    "presence_penalty": 0.0
                }                   
            }
        ],
        "type": "openai-chat-completion",
        "baseUrl": "http://myhost:8080/v1",
        "name": "lemonade-local"
    }
],
    • Or the lemonade extension for copilot.
  • iPlus Framework:
    • Endpoint: http://myhost:8080/api/v1/chat/completions

Accessing the Server

Once configured, access your AI server via HTTPS using the following URL:

https://gipDLVmAIServ.incus:8081
https://<yourhostname>:8081

OpenAI compatible API:

https://gipDLVmAIServ.incus:8081/api/v1
https://<yourhostname>:8081/api/v1