jamescallander/txgemma-9b-chat_w8a8_g128_rk3588.rkllm

TxGemma-9B-Chat — RKLLM build for RK3588 boards

Built with TxGemma (Google Health AI Developer Foundations).

Author: @jamescallander
Source model: google/txgemma-9b-chat

Target: Rockchip RK3588 NPU via RKNN-LLM Runtime

This repository hosts a conversion of txGemma-9B-Chat for use on Rockchip RK3588 single-board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using the RKNN-LLM toolkit

Conversion details

• RKLLM-Toolkit version: v1.2.1
• NPU driver: v0.9.8
• Python: 3.11
• Quantization: w8a8_g128
• Output: single-file .rkllm artifact
• Modifications: quantization (w8a8_g128), export to .rkllm format for RK3588 SBCs.
• Tokenizer: not required at runtime (UI handles prompt I/O)

⚠️ Safety disclaimer

🛑 This model is not a substitute for professional advice, diagnosis, or treatment.

• It is intended for research, educational, and experimental purposes only.
• Do not rely on its outputs for decisions related to health, safety, or legal/financial matters.
• Always consult a qualified professional for real-world guidance in these domains.
• Use responsibly and in compliance with the source model’s license and Acceptable Use Policy.

Intended use

• On-device chat/inference on RK3588 SBCs, particularly for therapeutics-oriented R&D workflows where private, low-power inference is useful. The txGemma-Chat variants are instruction-tuned for conversational use. (See Google’s txGemma release and model docs for positioning and scope.)
  developers.googleblog.comGoogle for Developers

Limitations

• Requires 12GB free memory
• Quantized build (w8a8_g128) may show small quality differences vs. full-precision upstream.
• Tested on Orange Pi 5 Plus and Radxa Rock 5B+; other devices may require different drivers/toolkit versions.
• Follow Google’s TxGemma/Gemma usage policies and AUP; additional domain restrictions may apply. Google AI for Developers

Quick start (RK3588)

1) Install runtime

The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from airockchip's github page.

Download and install the required packages as per the toolkit's instructions.

2) Simple Flask server deployment

The simplest way the deploy the .rkllm converted model is using an example script provided in the toolkit in this directory: rknn-llm/examples/rkllm_server_demo

python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
  --rkllm_model_path <MODEL_PATH>/txgemma-9b-chat_w8a8_g128_rk3588.rkllm \
  --target_platform rk3588

3) Sending a request

A basic format for message request is:

{
    "model":"txgemma-9b",
    "messages":[{
        "role":"user",
        "content":"<YOUR_PROMPT_HERE>"}],
    "stream":false
}

Example request using curl:

curl -s -X POST <SERVER_IP_ADDRESS>:8080/rkllm_chat \
    -H 'Content-Type: application/json' \
    -d '{"model":"txgemma-9b","messages":[{"role":"user","content":"Explain the role of protein folding in disease."}],"stream":false}'

The response is formated in the following way:

{
    "choices":[{
        "finish_reason":"stop",
        "index":0,
        "logprobs":null,
        "message":{
            "content":"<MODEL_REPLY_HERE">,
            "role":"assistant"}}],
        "created":null,
        "id":"rkllm_chat",
        "object":"rkllm_chat",
        "usage":{
            "completion_tokens":null,
            "prompt_tokens":null,
            "total_tokens":null}
}

Example response:

{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"## The Devastating Impact of Misfolded Proteins: A Role in Many Diseases\n\nProteins are the workhorses of our cells, carrying out essential tasks from catalyzing reactions to transporting molecules. For a protein to function correctly, it must fold into a specific three-dimensional shape. However, errors in this folding process can lead to misfolded proteins, which can disrupt cellular processes and ultimately contribute to a wide range of diseases.\n\nHere's how misfolded proteins contribute to disease:\n\n**1. Loss of Function:**\n\n* **Direct effect:** When a protein misfolds, it loses its intended shape and, consequently, its function. This can be seen in diseases like cystic fibrosis, where a mutated CFTR protein misfolds and fails to regulate chloride ion transport, leading to thick mucus buildup in the lungs.\n* **Indirect effect:** Even if a protein retains some of its structure, its misfolded state might interfere with the function of other proteins, leading to downstream problems.\n\n**2. Gain of Toxic Function:**\n\n* **Aggregation:** Misfolded proteins are often sticky and prone to clumping together, forming aggregates. These aggregates can disrupt cellular processes and even trigger cell death. This is a key feature of diseases like Alzheimer's (amyloid plaques) and Parkinson's (Lewy bodies).\n* **Aberrant Activity:**  A misfolded protein might gain new, potentially harmful activities, disrupting normal cellular pathways. For example, in Huntington's disease, a mutated huntingtin protein gains toxic properties that lead to neuronal dysfunction and death.\n\n**3. Stress on the Cellular Machinery:**\n\n* The accumulation of misfolded proteins overwhelms the cell's protein quality control mechanisms, leading to stress and ultimately cell death. This is a major contributor to neurodegenerative diseases like Alzheimer's and Parkinson's.\n\n**Examples of Diseases Linked to Protein Misfolding:**\n\n* **Neurodegenerative diseases:** Alzheimer's disease, Parkinson's disease, Huntington's disease, Amyotrophic Lateral Sclerosis (ALS), Prion diseases\n* **Cystic fibrosis:**  A mutation in the CFTR gene leads to a misfolded protein that disrupts chloride ion transport.\n* **Alpha-1 antitrypsin deficiency:** A mutation causes the alpha-1 antitrypsin protein to misfold and accumulate in the liver, leading to emphysema.\n* **Cancer:**  Misfolded proteins can contribute to cancer development by disrupting cell signaling pathways and promoting uncontrolled cell growth.\n\n**Research and Treatment:**\n\nUnderstanding the intricacies of protein folding and misfolding is crucial for developing effective treatments for these devastating diseases. Ongoing research focuses on:\n\n* **Developing drugs that can prevent or correct misfolding.**\n* **Identifying ways to clear toxic protein aggregates.**\n* **Boosting the capacity of cells to handle misfolded proteins.**\n\nBy unraveling the complexities of protein folding and its role in disease, researchers are striving to develop therapies that can alleviate suffering and improve the lives of those affected by these conditions.\n","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}

4) UI compatibility

This server exposes an OpenAI-compatible Chat Completions API.

You can connect it to any OpenAI-compatible client or UI (for example: Open WebUI)

• Configure your client with the API base: http://<SERVER_IP_ADDRESS>:8080 and use the endpoint: /rkllm_chat
• Make sure the model field matches the converted model’s name, for example:

{
 "model": "TxGemma-9B-Chat",
 "messages": [{"role":"user","content":"Hello!"}],
 "stream": false
}

License

This conversion follows the license of the source model: Health AI Developer Foundations Terms of Use  |  Google for Developers. Health AI Developer Foundations Prohibited Use Policy

For more information on the deployment and use of .rkllm models on RK3588 platforms, please refer to the RKNN-LLM toolkit documentation.