RKLLM
A local LLM toolkit runs language and vision models on Rockchip NPUs, with PC model conversion, quantization and C/C++ application interfaces.
RKLLM is a toolkit for developers building local AI applications on Rockchip boards. It runs language and multimodal models on the chip's neural processing unit (NPU).
- Model preparation: Convert trained models into RKLLM format and quantize them on a PC, or choose converted models from the model zoo.
- Language models: Run supported families including LLAMA, Qwen2/Qwen2.5/Qwen3/Qwen3.5, Phi2/Phi3, Gemma, DeepSeek-R1-Distill and RWKV7.
- Image questions: The multimodal demo lets you ask questions about your own images; supported models include Qwen2-VL/Qwen3-VL, MiniCPM-V-2_6 and InternVL2-1B/InternVL3-1B.
- Application integration: C/C++ interfaces let developers add model inference to their applications, with examples for API use and an API server.
- Performance assessment: Benchmarks and measurement tools help assess inference performance, memory use and CPU utilization for a board and model combination.
The runtime targets RK3588, RK3576, RK3562 and RV1126B series hardware and uses RKLLM-format models. Model preparation runs on a PC, while inference runs on the development board. The vision demo runs entirely on the local device, combining RKNN for vision with RKLLM for language.
