RK3588 NPU with Rocket + llama.cpp

A practical setup for running GGUF language models on the Divine D. using its RK3588 NPU.

This setup targets DawnOS running the mainline Linux 7.1 kernel with the mainline Rocket NPU driver, rocket-userspace, ggml-rocket, and llama.cpp.

Hardware:     Divine D.
SoC:          Rockchip RK3588S
OS:           DawnOS 1.0
Kernel:       Mainline Linux 7.1
NPU driver:   Rocket
Inference:    llama.cpp + ggml-rocket

Architecture

GGUF model
    ↓
llama.cpp
    ↓
libggml-rocket.so
    ↓
rocket-userspace
    ↓
/dev/accel/accel0
    ↓
Linux accel/rocket
    ↓
RK3588 NPU

Rocket mainly accelerates prompt processing / prefill and supported matrix multiplications on the RK3588 NPU. Token-by-token generation remains substantially CPU-bound.

Rocket mainly accelerates prompt processing / prefill. Token-by-token generation remains mostly CPU-bound.

Requirements

Verify Rocket:

ls -l /dev/accel/accel0
dmesg | grep -i rocket

Install build tools:

sudo apt update
sudo apt install -y git cmake build-essential pkg-config wget curl

Build rocket-userspace

cd /home/mobian
git clone https://github.com/gregordinary/rocket-userspace.git
cd rocket-userspace

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"
sudo cmake --install build

Build llama.cpp

cd /home/mobian
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

cmake -S . -B build \
    -DGGML_BACKEND_DL=ON \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_CPU_REPACK=OFF \
    -DCMAKE_BUILD_TYPE=Release

cmake --build build -j"$(nproc)"

GGML_CPU_REPACK=OFF is important for quantized GGUF models used with Rocket.

Build ggml-rocket

Build it against the same llama.cpp checkout:

cd /home/mobian
git clone https://github.com/gregordinary/ggml-rocket.git
cd ggml-rocket

cmake -S . -B build-dl \
    -DGGML_ROCKET_DL=ON \
    -DHOST_DIR=/home/mobian/llama.cpp \
    -DCMAKE_BUILD_TYPE=Release

cmake --build build-dl -j"$(nproc)"

Verify:

ls -lh build-dl/libggml-rocket.so

Recommended Models

Qwen3.5-0.8B Q4_0

Small general-purpose model and useful Rocket smoke test.

mkdir -p /home/mobian/models/qwen3.5-0.8b
cd /home/mobian/models/qwen3.5-0.8b

wget -O Qwen3.5-0.8B-Q4_0.gguf \
"https://huggingface.co/ggml-org/Qwen3.5-0.8B-GGUF/resolve/main/Qwen3.5-0.8B-Q4_0.gguf"

Qwen2.5-Coder-0.5B-Instruct Q4_0

Very small coding model.

mkdir -p /home/mobian/models/qwen2.5-coder-0.5b
cd /home/mobian/models/qwen2.5-coder-0.5b

wget -O Qwen2.5-Coder-0.5B-Instruct-Q4_0.gguf \
"https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-0.5b-instruct-q4_0.gguf"

Qwen2.5-Coder-1.5B-Instruct Q4_0

Better coding quality while remaining practical on RK3588.

mkdir -p /home/mobian/models/qwen2.5-coder-1.5b
cd /home/mobian/models/qwen2.5-coder-1.5b

wget -O Qwen2.5-Coder-1.5B-Instruct-Q4_0.gguf \
"https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF/resolve/main/qwen2.5-coder-1.5b-instruct-q4_0.gguf"

Benchmark

Example with Qwen3.5-0.8B:

MODEL=/home/mobian/models/qwen3.5-0.8b/Qwen3.5-0.8B-Q4_0.gguf

GGML_BACKEND_PATH=/home/mobian/ggml-rocket/build-dl/libggml-rocket.so \
ROCKET_MM_PROFILE=1 \
sudo -E /home/mobian/llama.cpp/build/bin/llama-bench \
    -m "$MODEL" \
    -ngl 0 \
    -p 2048 \
    -n 128 \
    -b 2048 \
    -ub 2048

For prefill-only testing:

-n 0

pp results measure prompt processing and are the main indicator of Rocket/NPU acceleration. tg measures token generation and is mostly CPU-bound.

Interactive Use

MODEL=/home/mobian/models/qwen3.5-0.8b/Qwen3.5-0.8B-Q4_0.gguf

GGML_BACKEND_PATH=/home/mobian/ggml-rocket/build-dl/libggml-rocket.so \
ROCKET_MM_PROFILE=1 \
sudo -E /home/mobian/llama.cpp/build/bin/llama-cli \
    -m "$MODEL" \
    -ngl 0 \
    -c 8192 \
    -b 2048 \
    -ub 2048 \
    -n 2048 \
    --reasoning off

Short prompts may show little NPU activity because Rocket mainly benefits larger prefill workloads.

Verify NPU Activity

Monitor runtime state:

watch -n 0.1 '
for D in fdab0000.npu fdac0000.npu fdad0000.npu; do
    printf "%s: " "$D"
    cat /sys/bus/platform/devices/$D/power/runtime_status 2>/dev/null
done
'

Check the NPU clock:

grep scmi_clk_npu /sys/kernel/debug/clk/clk_summary

Keep ROCKET_MM_PROFILE=1 enabled to confirm that supported matrix multiplications are actually being offloaded.

Notes

• -ngl 0 is correct for Rocket.
• Rocket is a GGML accelerator backend, not normal GPU layer offload.
• Large prefill workloads benefit most from the NPU.
• Autoregressive token generation remains mostly CPU-bound.
• Seeing loaded ROCKET backend confirms the backend loaded, but not necessarily that the NPU executed work.