My new Linux workstation has an AMD Ryzen AI 9 HX 470, including an XDNA2 neural processing unit (NPU). Under CachyOS, an Arch-based distribution, the kernel already detected it. The amdxdna driver was loaded and /dev/accel/accel0 existed. What was missing was the userspace stack needed to run a model.
I wanted to find out whether AMD’s Ryzen AI tooling could use the functioning in-tree driver without replacing it with a separately managed kernel module. That worked, but getting the software installed was only the first part of the exercise. The more useful finding came when a model compiled and ran successfully, then returned invalid results.
Keep the driver that already works
The XDNA project included Arch packaging, but its standard path brought in an external kernel driver alongside the userspace plugin. On this machine, that meant building a replacement for a driver the kernel already supplied.
A full build tried to compile amdxdna.ko and exposed a compiler mismatch: CachyOS had built the kernel with Clang 22, while the XDNA build invoked GCC 16. GCC rejected the Clang-specific kernel flags. Forcing Clang across the whole build got past that problem, then caused the userspace build to fail on newer warnings treated as errors.
The simpler answer was to reduce the build scope. AMD’s build script has a mode that omits the kernel module:
./build.sh -release -nokmodI packaged the userspace-only result as an Arch package and retained the in-tree amdxdna driver. This removed the kernel-module build from the installation path.
Establish that the runtime can use the NPU
With XRT 2.26 and the XDNA userspace plugin installed, xrt-smi examine identified the device as NPU Gorgon Point 1, with an aie2p architecture, a 6×8 topology and firmware 1.1.2.64. All three xrt-smi validate tests passed.
GEMM 51.0 TOPS PASSED
Latency 57 μs PASSED
Throughput 94,922 ops/s PASSEDThese synthetic tests confirmed that the driver and userspace components could execute NPU workloads. The next step was to install the model runtime.
Adapt the userspace environment
Ryzen AI Software 1.8 used Python 3.12, while the Arch installation on this machine had moved to Python 3.14. I used uv to install an isolated Python 3.12 environment and left the system Python alone.
AMD’s Ubuntu-targeted binaries then exposed three compatibility problems. An ONNX Runtime shared object was marked as requiring an executable stack, which this system rejected. Clearing that flag with patchelf --clear-execstack resolved the import failure.
The Vitis AI provider expected libncurses.so.6, while Arch supplied libncursesw.so.6. I kept a compatibility link in a private library directory. FlexML also needed libpython3.12.so.1.0, which lived under uv’s managed Python installation. Adding that directory to the environment’s LD_LIBRARY_PATH resolved the missing library.
After these changes, ONNX Runtime listed VitisAIExecutionProvider alongside CPUExecutionProvider, and AMD’s bundled quicktest passed. The software could now compile and execute the supplied test model through the NPU path.
From ONNX model to XDNA2
The ONNX model uses ONNX Runtime 1.27.0 and its Vitis AI execution provider from Ryzen AI Software 1.8. The NPU path connects through XRT 2.26.0 and the XDNA userspace plugin 2.26.0 to the existing CachyOS amdxdna kernel driver and the AMD XDNA2 NPU. ONNX Runtime also executes CPU-assigned operators on the CPU.
Run a model beyond the quicktest
I chose Laya, a 421-million-parameter ModernBERT-based decision model, for the next test. Its published ONNX export has five inputs and two outputs: decision logits and action probabilities. It provided a concrete workload to compare between CPU and NPU execution.
The exported graph used dynamic dimensions, and the Vitis compilation failed on those shapes. ONNX Runtime’s free-dimension overrides let me fix the NPU build to a batch size of 1, a sequence length of 512 tokens and 20 option slots. Unused options were padded and masked. With those dimensions fixed, compilation progressed.
Successful execution, invalid output
The two-output graph compiled, loaded and ran through Vitis AI. Both outputs contained NaNs: invalid numerical values.
The same model and input produced finite results on the CPU. The test described a web service that had stopped accepting requests, with monitoring showing a full database disk. The choices were to restart the web server, free space on the database disk, or wait. The CPU assigned approximately 98.8% to freeing disk space; the NPU result could not be used.
A report in AMD’s RyzenAI-SW issue tracker described similar failures in some multi-output ONNX graphs on a different hardware and software combination. It suggested an experiment: change the outputs exposed by the graph. The report included working multi-output models too, so this was a lead to investigate, not a confirmed diagnosis.
Expose a single output and compare again
I created a second ONNX graph exposing only Laya’s logits output, retained the same external weights and compiled it with a fresh Vitis cache. For the test input, the returned logits were finite and matched the CPU reference.
CPU
[-1.7870014, 3.3337939, -1.7052455]
NPU
[-1.7870014, 3.3337939, -1.7052455]
Maximum observed difference: 0.0
Finite NPU output: TrueThe compiler marked 1,598 of 1,611 operators and 99.998% of the graph’s estimated operation count as supported by VAIML, its NPU compilation path. After partitioning, the execution-provider report counted 1,461 VAIML assignments and 119 CPU assignments. This confirmed that work had been assigned to the NPU path while some operators remained on the CPU.
The single-output graph now matched the CPU reference for this input. It no longer exposed the separate action-probability output. Preserving both logical outputs through one physical ONNX output remained a further experiment.
Measure the workload, not just the hardware
I then benchmarked the single-output model at batch sizes 1, 8 and 16, using five warm-up iterations and 20 measured iterations per provider. Sequence length remained fixed at 512 tokens, with 20 option slots. The larger batches repeated the same decision input, so this tested compute throughput rather than a broader range of decision problems.
Both sessions remained alive for each benchmark. Timings covered the inference call, excluding input preparation, compilation and session creation. Every measured NPU result was checked for finite values and compared with the CPU reference. The reported maximum output difference was zero at all three batch sizes.
The table shows median batch latency divided by batch size. A decision in a batch still waits for the whole batch to finish.
| Batch size | CPU | NPU |
|---|---|---|
| 1 | 1,196.04 | 1,156.77 |
| 8 | 1,118.98 | 1,124.19 |
| 16 | 1,356.27 | 1,359.55 |
At batch 1, the NPU reduced median latency by 3.3%, equivalent to a 1.034× speedup. At batch 8, both providers delivered approximately 0.89 decisions per second; at batch 16, both fell to approximately 0.74. Increasing the batch beyond 8 made this workload slower per decision on both providers.
Similar medians did not mean identical timing distributions. In the 20 measured batch-8 runs, NPU P95 batch latency was 11.00 seconds, compared with 9.10 seconds on the CPU.
For this Laya workload on the tested Ryzen AI 9 HX 470 stack, the NPU provided no substantial latency or throughput advantage. Almost all of the estimated computation was supported by the NPU compiler, but that did not translate into a substantial speedup.
Reloading the compiled model
The model compiled and ran in the Python process that created the NPU session, but a second process could not reliably reload the raw Vitis cache. I reproduced the failure without rebooting or changing the kernel, XRT or firmware.
For the measurements above, I compiled a fresh cache and kept the session alive in the same process through warm-up and timing. An ONNX Runtime EP Context Cache experiment also failed to produce a reusable compiled model in this configuration.
A long-running application could keep its session alive across requests. Reusing the compiled model after a process restart remains unresolved on this setup.
What I would test next
With Laya running and checked against the CPU reference, I can use this setup to investigate other local inference workloads, including classification, embeddings and decision models.
For Laya itself, the next useful measurement is host CPU utilisation during sustained inference. Comparable latency could still be useful if NPU execution leaves more CPU capacity available for other work. I have not measured that yet.