---
license: other
license_name: open-aleph-license
license_link: LICENSE
language:
- en
---
# LLaMA 11.7B - 4-bit Quantized

📚 [Paper](https://arxiv.org/html/2602.15563) • 🏠 [GitHub](https://github.com/Aleph-Alpha-Research/1-Bit-Wonder/tree/main)

This is one of the checkpoints supplementing the paper [1-Bit-Wonder: Improving QAT Performance in the Low-Bit Regime through K-Means Quantization](https://arxiv.org/html/2602.15563). Instructions on how to use the model for inference can be found in the corresponding [repository](https://github.com/Aleph-Alpha-Research/1-Bit-Wonder/tree/main).

⚠️ IMPORTANT: This model is intended for research purposes only. It is provided as-is without warranties for production use.

## Model Details
- **Architecture**: LLaMA
- **Size**: 11.7B (11,701,129,216 parameters)
- **Quantization**: 4-bit k-means with per-64 block absmax scaling
- **Centroids**: Defined per Layer

## Directory Structure
```
.
├── config.json                  # HuggingFace model config
├── quantization_config.json     # Quantization parameters
├── generation_config.json       # Default generation settings
├── tokenizer.json              # Tokenizer files
└── model.safetensors           # Weights (quantized linear layers + bf16 embeddings/norms)
```

## Weight Format
Linear layers are quantized:
- `*.weight_packed`: Packed uint8 indices (2 elements per byte)
- `*.scales`: Per-block scale factors (bfloat16)
- `*.centroids`: K-means centroids for this layer (float32)

Embeddings and norms are stored in bfloat16.

## Citation

If you find this work helpful, please cite:

```bibtex
@inproceedings{
maskey2026bit,
title={1-Bit Wonder: Improving {QAT} Performance in the Low-Bit Regime through K-Means Quantization},
author={Sohir Maskey and Constantin Eichenberg and Johannes Messner and Douglas Orr},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=6EGpWhH2yx}
}
```