Technologies · Artificial intelligence
Local LLMs on the AMD Radeon AI PRO R9700 GPU
We run our own language model server on a GPU at our company headquarters. We tune it for working with Claude Code and AI agents — source code and data never leave our infrastructure.
Why a local model
We use Claude Code for requirements analysis, design and implementation of our systems. To keep source code and data inside the company, we connect it to our own llama.cpp LLM server in our infrastructure — the server provides an Anthropic API compatible interface, so Claude Code works with it just as it does with a cloud model. We switch the model Claude Code works with simply via a launch script.
The same server also serves autonomous AI agents (Hermes Agent) on our internal network.
- confidentiality — source code and data stay in the company
- cost — no fees for processed tokens
- independence — operation regardless of cloud availability
- control — we can measure, tune and fix the whole chain
Hardware and software
One workstation, one graphics card.
GPU
AMD Radeon AI PRO R9700 — RDNA4 architecture, 32 GB VRAM.
Operating system and runtime
Windows 11 Pro, AMD ROCm 10 (HIP) — no Linux or WSL.
llama.cpp
The stew675 “llama-cpp-rdna-boosts” fork with optimisations for RDNA GPUs, extended with our own changes on top of the current llama.cpp version.
Our own build
Our repository and CI build for ROCm on Windows on GitHub — every new version goes through performance measurements and an output-identity check.
Models in use
Production model: Qwen3.8-27B
- Unsloth UD-Q5_K_XL quantisation (about 21 GB)
- hybrid SSM + attention architecture, support for images and reasoning
- context of 262,144 tokens — the model's maximum — in two parallel slots
- KV cache quantised to q8_0
- MTP speculative decoding (multi-token prediction)
Alternative and earlier models
- Swift-1.5-Qwen3.8-27B (Q5_K_M, Q6_K_S)
- Qwen3.6-35B-A3B (Mixture of Experts), Qwen3.6-27B
Our latest results
As of October 2026, measured on real prompts from our agents.
| Period | Change | Prompt processing | Generation |
|---|---|---|---|
| August 2026 | switch to ROCm and port of the rdna-boosts optimisations | 312 → 574 t/s (+84%) | — |
| September 2026 | WMMA flash attention kernels for RDNA4 | 660 → 752 t/s (+14%) | −12.5% time per token |
| September 2026 | further decoding improvements | 743 t/s | −5.4% time per token |
| October 2026 | Qwen3.8-27B V2, MTP, 262k context | 780–835 t/s | 45–47 t/s |
At the start (August 2026, Vulkan backend, Qwen3.6-27B model) we reached about 530 t/s in prompt processing and 21 t/s in generation. The rows are not strictly comparable — the model, quantisation and decoding method changed along the way.
Quality with a quantised KV cache
At a context of 126k tokens with a q8_0 KV cache, the model correctly retrieved all 120 of 120 test facts, and its output was token-for-token identical to the output at full precision — with considerably lower VRAM use.
What helped and what did not
We measure every change on real tasks and deploy only what holds up.
Helped
- port of the rdna-boosts optimisations (+84% in prompt processing)
- skipping masked blocks in flash attention (+8 to 24%)
- WMMA kernels for RDNA4
- q8_0 KV cache (about 2 GB less VRAM)
- one-time allocation of the flash attention staging buffer — our change, +73k tokens of context
- MTP speculative decoding (+56% in generation)
- prompt cache fixes — returning to a conversation in 1.2 s instead of 41 s
Did not help
- larger micro-batch — VRAM overflow
- HIP graphs — no gain, unstable with long prompts
- native bf16 prefill — 47% slower
- hipBLASLt library — non-deterministic kernel selection, so disabled
Lessons from operation
- after a long server uptime, VRAM measurements are unreliable — we restart before measuring
- the Adrenalin 26.7 driver was unstable; we use the OEM driver branch
- most repeated prompt processing comes from the client, not the server
Interested in deploying local LLMs?
We are glad to share our experience of running language models on our own hardware.