🛠 Exploiter le NPU 6 TOPS avec le SDK RKNN

Objectives

Deploy a vision model on the RK3588's 6 TOPS NPU (mixed INT4/INT8/INT16/FP16) using Rockchip's RKNN SDK: convert a model (ONNX/TF/PyTorch) to .rknn on an x86_64 PC, then run it on the board using the rknn-lite runtime. Be aware of the vendor/mainline split: the NPU (like MPP/RGA and the closed Mali blob) is available on vendor BSP 5.10 images (Debian 11 / Ubuntu 22.04 / Orange Pi OS); under Armbian mainline, the NPU is not usable — switch to the vendor image for this tutorial.

Hardware

• Orange Pi 5 Plus running a vendor image (the wiki documents the installation; the NPU is listed on the product page)
• x86_64 Linux PC for conversion (rknn-toolkit2: github.com/airockchip/rknn-toolkit2)
• 5 V / 4 A power supply; MIPI CSI camera or test images

Complete Code

# --- Sur le PC x86_64 : conversion du modèle (Python 3.8-3.10 requis) ---
# git clone https://github.com/airockchip/rknn-toolkit2
# pip install rknn-toolkit2/rknn-toolkit2/packages/rknn_toolkit2-*.whl   # wheel selon version
python3 convert.py   # exemple : ONNX -> RKNN, quantification INT8 sur jeu d'images

# --- Sur l'Orange Pi 5 Plus (image vendor ARM64) ---
sudo apt install -y python3-pip
pip3 install rknn_toolkit_lite2/rknn_toolkit_lite2/packages/rknn_toolkit_lite2-*.whl
scp model.rknn orangepi@<ip-de-la-carte>:~/

#!/usr/bin/env python3
"""Inférence NPU avec rknn-lite2 (API du SDK airockchip/rknn-toolkit2)."""
import numpy as np
from rknnlite.api import RKNNLite

rknn = RKNNLite
assert rknn.load_rknn("model.rknn") == 0
# Utiliser les 3 cœurs NPU ; CORE_0 seul pour laisser du marge
assert rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) == 0

input_name = rknn.get_input_name(0)[0]
img = np.zeros((224, 224, 3), dtype=np.uint8)   # remplacer par la vraie image
outputs = rknn.inference(inputs=[img])
print("Sortie :", np.asarray(outputs[0]).shape)
rknn.release

Expected Verification

The conversion on the PC produces model.rknn without errors; on the board, init_runtime returns 0 (the runtime has opened /dev/rga and the NPU device) and inference prints the shape of the output tensor. Examples from the rknn-toolkit2 repository (e.g., mobilenet, yolov5) display classes and scores; the NPU appears in ls /dev/ | grep rknpu on vendor images.

Common Errors

• Trying under Armbian mainline: no NPU driver or blob; revert to the vendor BSP 5.10 image for RKNN (panfrost/stateless codecs do not cover the NPU).
• Version incompatibility: the rknn-toolkit2 wheel (PC conversion) and rknn_toolkit_lite2 (board) must correspond to the same SDK version.
• Python incompatibility: the wheels target specific versions (3.8–3.10 depending on the SDK); check python3 --version before pip install.
• > on CPU using the 4 to 32 GB of RAM.