Instructions to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Use Docker
docker model run hf.co/SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
- SGLang
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with Ollama:
ollama run hf.co/SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with Docker Model Runner:
docker model run hf.co/SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
- Lemonade
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.NVIDIA-Nemotron-Nano-9B-v2-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Quantized NVIDIA-Nemotron-Nano-9B-v2
This repository contains the Q4 and Q5 quantized varients of NVIDIA-Nemotron-Nano-9B-v2 is an LLM trained from scratch by NVIDIA for both reasoning and non-reasoning tasks. It can generate reasoning traces before providing a final answer, improving accuracy for complex prompts, though this can be disabled for faster responses. The model uses a hybrid architecture of Mamba-2, MLP, and four Attention layers, and was trained with Megatron-LM and NeMo-RL. It supports English, German, Spanish, French, Italian, and Japanese.
Model Overview
- Original Model: NVIDIA-Nemotron-Nano-9B
- Base Model: NVIDIA-Nemotron-Nano-12B-v2-Base
- Variants: Instruct-tuned multimodal model
- Architecture: Mamba2-Transformer Hybrid
- Quantized Versions:
- Q4_K_M (4-bit quantization)
- Q5_K_M (5-bit quantization)
- Modalities: Text
- Developer: nvidia
- License: nvidia-open-model-license
- Languages: English, German, Spanish, French, Italian, and Japanese.
Quantization Details
Q4_K_M Version
- Approx. ~64% size reduction
- Lower memory footprint (~6.08 GB)
- Well-suited for deployment on edge devices or low-resource GPUs
- Minor performance degradation in highly complex reasoning scenarios
Q5_K_M Version
- Approx. ~63% size reduction
- Lower memory footprint (~6.58 GB)
- Better performance retention, recommended when quality is a priority
Key Features
- High inference throughput: up to ~6ร faster reasoning performance on long-context tasks (e.g., 8k input + 16k output) compared to similar 9B models.
- Unified Reasoning & Completion: Designed for both reasoning and non-reasoning tasks, generating reasoning traces before final responses.
- Hybrid Architecture: Combines Mamba-2 and MLP layers with just four Attention layers for efficient processing.
- Extended Context Length: Supports up to 128K tokens, enabling long-context reasoning.
- Reasoning Budget Control: Allows specification of token limits during inference to manage reasoning depth.
Model Data
- The model is pretrained and instruction-tuned on a large and diverse mixture of text and code datasets, designed for efficiency, reasoning, and general-purpose language understanding, including:
- Massive web-scale text corpora covering diverse domains such as news, literature, technical writing, and dialogue.
- High-quality instruction-tuning data curated for reasoning, step-by-step problem solving, and chat-style interactions.
- Code and programming datasets spanning multiple languages to enhance logical and structured reasoning capabilities.
- Synthetic reasoning traces distilled from larger Nemotron and Minitron models to improve long-context performance.
- English-centric pretraining with selective multilingual coverage for generalization and robustness.
- ~20 trillion training tokens processed using NVIDIAโs FP8 training pipeline for optimal efficiency and scalability.
Recommended Use Cases
- Instruction following and chatbots: builds responsive conversational agents capable of multi-turn dialogue and contextual understanding.
- Complex reasoning and problem solving: excels at step-by-step logical reasoning, explanations, and analytical tasks.
- Code generation and understanding: supports multiple programming languages, aiding in code synthesis, debugging, and explanation.
- Educational and tutoring assistants: provides structured, step-by-step reasoning for math, science, and technical subjects.
- Research and experimentation: ideal for exploring hybrid Mamba-Transformer architectures and efficient inference design.
- Edge and production deployment: optimized for high throughput and reduced memory usage on single or multi-GPU setups.
Usage Example
Using llama.cpp for inference:
./llama-cli -hf SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2 -p "Provide me script to print stars in triangle shape in python"
Acknowledgments
These quantized models are based on the original work by the NVIDIA development team.
Special thanks to:
The NVIDIA team for developing and releasing the NVIDIA-Nemotron-Nano-9B-v2 model.
Georgi Gerganov and the entire
llama.cppopen-source community for enabling efficient model quantization and inference via the GGUF format.
Contact
For any inquiries or support, please contact us at support@sandlogic.com or visit our Website.
- Downloads last month
- 31
4-bit
5-bit
ollama run hf.co/SandLogicTechnologies/NVIDIA-Nemotron-Nano-9B-v2-GGUF:Q4_K_M