Update README.md
Browse files
README.md
CHANGED
|
@@ -7,9 +7,15 @@ pipeline_tag: text-generation
|
|
| 7 |
## Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ
|
| 8 |
This is a version of the Mixtral-8x7B-v0.1 model (https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) quantized to 4-bit via Half-Quadratic Quantization (HQQ).
|
| 9 |
### Basic Usage
|
| 10 |
-
To run the model, install the HQQ library
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
``` Python
|
| 12 |
-
model_id = 'mobiuslabsgmbh/Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ
|
| 13 |
|
| 14 |
#Load the model
|
| 15 |
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
|
|
|
|
| 7 |
## Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ
|
| 8 |
This is a version of the Mixtral-8x7B-v0.1 model (https://huggingface.co/mistralai/Mixtral-8x7B-v0.1) quantized to 4-bit via Half-Quadratic Quantization (HQQ).
|
| 9 |
### Basic Usage
|
| 10 |
+
To run the model, install the HQQ library:
|
| 11 |
+
```
|
| 12 |
+
pip install hqq==0.1.8
|
| 13 |
+
pip install transformers==4.46.0
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
and use it as follows:
|
| 17 |
``` Python
|
| 18 |
+
model_id = 'mobiuslabsgmbh/Mixtral-8x7B-v0.1-hf-4bit_g64-HQQ'
|
| 19 |
|
| 20 |
#Load the model
|
| 21 |
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
|