---
library_name: rkllm
pipeline_tag: text-generation
license: llama3.2
language:
- en
base_model:
- meta-llama/Llama-3.2-3B-Instruct
tags:
  - rkllm
  - rk3588
  - rockchip
  - edge-ai
  - llm
  - llama
  - text-generation-inference
---
# Llama-3.2-3B-Instruct — RKLLM build for RK3588 boards

#### Built with Llama 3.2 (Meta Platforms, Inc.)

**Author:** @jamescallander  
**Source model:** [meta-llama/Llama-3.2-3B-Instruct · Hugging Face](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)

**Target:** Rockchip RK3588 NPU via **RKNN-LLM Runtime**

> This repository hosts a **conversion** of `Llama-3.2-3B-Instruct` for use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the [RKNN-LLM toolkit](https://github.com/airockchip/rknn-llm?utm_source=chatgpt.com)

#### Conversion details

- **RKLLM-Toolkit version:** v1.2.1
- **NPU driver:** v0.9.8
- **Python:** 3.12
- **Quantization:** `w8a8_g128`
- **Output:** single-file `.rkllm` artifact
- **Modifications:** quantization (w8a8_g128), export to `.rkllm` format for RK3588 SBCs
- ****Tokenizer:**** not required at runtime (UI handles prompt I/O)

## Intended use

- On-device chat and instruction following on RK3588 SBCs.
- Llama-3.2-3B-Instruct is tuned for general conversational tasks, Q&A, and reasoning, making it suitable for **edge inference deployments** where low power and privacy matter.

## Limitations

- Requires 4GB free memory
- Quantized build (`w8a8_g128`) may show small quality differences vs. full-precision upstream.
- Tested on a Radxa Rock 5B+; other devices may require different drivers/toolkit versions.

## Quick start (RK3588)

### 1) Install runtime

The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from [airockchip's github page](https://github.com/airockchip).

Download and install the required packages as per the toolkit's instructions.

### 2) Simple Flask server deployment

The simplest way the deploy the `.rkllm` converted model is using an example script provided in the toolkit in this directory: `rknn-llm/examples/rkllm_server_demo`

```bash
python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
  --rkllm_model_path <MODEL_PATH>/Llama-3.2-3B-Instruct_w8a8_g128_rk3588.rkllm \
  --target_platform rk3588
```

### 3) Sending a request

A basic format for message request is:

```json
{
    "model":"Llama-3.2-3B",
    "messages":[{
        "role":"user",
        "content":"<YOUR_PROMPT_HERE>"}],
    "stream":false
}
```

Example request using `curl`:

```bash
curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
    -H 'Content-Type: application/json' \
    -d '{"model":"Llama-3.2-3B","messages":[{"role":"user","content":"Explain who Napoleon Bonaparte is in two or three sentences."}],"stream":false}'
```

The response is formated in the following way:

```json
{
    "choices":[{
        "finish_reason":"stop",
        "index":0,
        "logprobs":null,
        "message":{
            "content":"<MODEL_REPLY_HERE">,
            "role":"assistant"}}],
        "created":null,
        "id":"rkllm_chat",
        "object":"rkllm_chat",
        "usage":{
            "completion_tokens":null,
            "prompt_tokens":null,
            "total_tokens":null}
}
```

Example response:

```json
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Napoleon Bonaparte was a French military leader and statesman who rose to power during the French Revolution, becoming Emperor of France from 1804 to 1815. He is known for his military conquests, strategic genius, and domestic reforms that transformed Europe, but his ambition and authoritarian rule ultimately led to his downfall. His legacy has been debated among historians, with some viewing him as a brilliant leader who modernized France, while others see him as a power-hungry dictator responsible for devastating wars and loss of life.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}
```

# License

This conversion follows the license of the source model: [LICENSE · meta-llama/Llama-3.2-3B-Instruct at main](https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/LICENSE.txt)
- -**Required notice:** see [`NOTICE`](NOTICE)