Instructions to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: llama cli -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Use Docker
docker model run hf.co/IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "IsValorum/Axion-1.0-35B-Q8_0-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "IsValorum/Axion-1.0-35B-Q8_0-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
- Ollama
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with Ollama:
ollama run hf.co/IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
- Unsloth Desktop
- Pi
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with Docker Model Runner:
docker model run hf.co/IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
- Lemonade
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Run and chat with the model
lemonade run user.Axion-1.0-35B-Q8_0-GGUF-Q8_0
List all available models
lemonade list
- Hermes Agent
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use IsValorum/Axion-1.0-35B-Q8_0-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "IsValorum/Axion-1.0-35B-Q8_0-GGUF:Q8_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Axion-1.0 35B - Q8_0
APEX Quantization Series by IsValorum
Independently computed using dedicated cloud compute clusters, funded 100% out-of-pocket.
If you consider these releases to be of high quality and they help you in your day-to-day workflow, consider supporting on Ko-fi. Every contribution directly sustains future cloud compute hours and helps support my university studies.
Family cross-links
| Tier | Footprint | Base | Abliterated |
|---|---|---|---|
| Q8_0 | 36.9 GB | Axion-1.0-35B | โ |
Architecture and precision
- Base model:
Axion1-0/Axion-1.0-35B - Architecture: MoE โ 40 layers, 30 linear-attention/SSM + 10 full-attention; 256 routed experts + 1 shared, 8 active; hidden 2048; 262,144-token context; 248,320 vocab
- iMatrix: not used โ plain Q8_0; this file is the calibration source for the family I-Matrix
- Recurrent shielding: 1D convs, state params, routers and RMSNorms in
F32
Empirical evaluation metrics
| Variant | Perplexity | Delta vs. BF16 | Fidelity retention |
|---|---|---|---|
| Base BF16 | pending | pending | pending |
| Q8_0 | pending | pending | pending |
Audited tensor map
| Tensor group | Layers | Count | Precision |
|---|---|---|---|
blk.*.attn_gate.weight |
SSM layers except 0 | 29 | Q8_0 |
blk.*.attn_k.weight |
full attention 3,7,...,39 | 10 | Q8_0 |
blk.*.attn_k_norm.weight |
full attention 3,7,...,39 | 10 | F32 |
blk.*.attn_norm.weight, blk.*.post_attention_norm.weight |
all 40 | 80 | F32 |
blk.*.attn_output.weight |
full attention 3,7,...,39 | 10 | Q8_0 |
blk.*.attn_q.weight |
full attention 3,7,...,39 | 10 | Q8_0 |
blk.*.attn_q_norm.weight |
full attention 3,7,...,39 | 10 | F32 |
blk.*.attn_qkv.weight |
SSM layers except 0 | 29 | Q8_0 |
blk.*.attn_v.weight |
full attention 3,7,...,39 | 10 | Q8_0 |
blk.*.ffn_down_exps.weight |
0-9 | 10 | Q8_0 |
blk.*.ffn_down_exps.weight |
10-29 | 20 | Q8_0 |
blk.*.ffn_down_exps.weight |
30-39 | 10 | Q8_0 |
blk.*.ffn_down_shexp.weight |
all 40 | 40 | Q8_0 |
blk.*.ffn_gate_exps.weight |
0-9 | 10 | Q8_0 |
blk.*.ffn_gate_exps.weight |
10-29 | 20 | Q8_0 |
blk.*.ffn_gate_exps.weight |
30-39 | 10 | Q8_0 |
blk.*.ffn_gate_inp.weight |
all 40 | 40 | F32 |
blk.*.ffn_gate_inp_shexp.weight |
all 40 | 40 | F32 |
blk.*.ffn_gate_shexp.weight |
all 40 | 40 | Q8_0 |
blk.*.ffn_up_exps.weight |
0-9 | 10 | Q8_0 |
blk.*.ffn_up_exps.weight |
10-29 | 20 | Q8_0 |
blk.*.ffn_up_exps.weight |
30-39 | 10 | Q8_0 |
blk.*.ffn_up_shexp.weight |
all 40 | 40 | Q8_0 |
blk.*.ssm_a |
SSM layers | 30 | F32 |
blk.*.ssm_alpha.weight |
SSM layers | 30 | Q8_0 |
blk.*.ssm_beta.weight |
SSM layers except 0 | 29 | Q8_0 |
blk.*.ssm_conv1d.weight |
SSM layers | 30 | F32 |
blk.*.ssm_dt.bias |
SSM layers | 30 | F32 |
blk.*.ssm_norm.weight |
SSM layers | 30 | F32 |
blk.*.ssm_out.weight |
SSM layers except 0 | 29 | Q8_0 |
blk.0.attn_gate.weight |
layer 0 | 1 | Q8_0 |
blk.0.attn_qkv.weight |
layer 0 | 1 | Q8_0 |
blk.0.ssm_beta.weight |
layer 0 | 1 | Q8_0 |
blk.0.ssm_out.weight |
layer 0 | 1 | Q8_0 |
output.weight |
global | 1 | Q8_0 |
output_norm.weight |
global | 1 | F32 |
token_embd.weight |
global | 1 | Q8_0 |
The complete audited list is tensor-map.tsv. |
- Downloads last month
- 168
8-bit
Model tree for IsValorum/Axion-1.0-35B-Q8_0-GGUF
Base model
Axion1-0/Axion-1.0-35B