Instructions to use mradermacher/llama-33B-instructed-i1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mradermacher/llama-33B-instructed-i1-GGUF with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mradermacher/llama-33B-instructed-i1-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mradermacher/llama-33B-instructed-i1-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
Use Docker
docker model run hf.co/mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use mradermacher/llama-33B-instructed-i1-GGUF with Ollama:
ollama run hf.co/mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use mradermacher/llama-33B-instructed-i1-GGUF with Docker Model Runner:
docker model run hf.co/mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
- Lemonade
How to use mradermacher/llama-33B-instructed-i1-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mradermacher/llama-33B-instructed-i1-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.llama-33B-instructed-i1-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Is there slop in this?
The original model creator doesn't seem too active on HF.
That said—I'm looking for the last good model before slop took over all open-source AND paid LLMs (chatgpt, claude3, etc.).
Someone on Reddit said to check out LLama 33b or LLaMa 65B.
I'm also on an M4 Macbook Pro with only 48GB of RAM. At this point, I just downloaded the i1-Q4_K_M from here (and am hoping for the best): https://huggingface.co/mradermacher/llama-65b-instruct-i1-GGUF
I would prefer to use your i-quants because you (guys) are really good at what you do.
You were told to check out llama 33b or 65b because these were leaked before practically anything else was available that was worth talking to. The question is whether these specific models are the original llama or not - from reading the original model cards, I would be doubtful.
It seems this one is more likely the original: https://huggingface.co/mradermacher/llama-65b-hf-i1-GGUF
I have also queued https://huggingface.co/pinkmanlove/llama-33b-hf https://huggingface.co/pinkmanlove/llama-13b-hf and https://huggingface.co/pinkmanlove/llama-7b-hf which might be the original keaked models.
From judging the date (mid/early 2023), you have a good chance that none of these contain significant slop, though - you'd need to try it out.
Can you also put this in the queue? I left a message on your request board thingy: https://huggingface.co/soob3123/amoral-gemma3-27B-v2/tree/main. Would like the imatrix quants.
Thanks, and I will test those two models once they're finished.
Really appreciate the work you guys are doing!