Instructions to use EldanRing/Winnow-E4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use EldanRing/Winnow-E4B with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf EldanRing/Winnow-E4B:BF16 # Run inference directly in the terminal: llama cli -hf EldanRing/Winnow-E4B:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf EldanRing/Winnow-E4B:BF16 # Run inference directly in the terminal: llama cli -hf EldanRing/Winnow-E4B:BF16
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf EldanRing/Winnow-E4B:BF16 # Run inference directly in the terminal: ./llama-cli -hf EldanRing/Winnow-E4B:BF16
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf EldanRing/Winnow-E4B:BF16 # Run inference directly in the terminal: ./build/bin/llama-cli -hf EldanRing/Winnow-E4B:BF16
Use Docker
docker model run hf.co/EldanRing/Winnow-E4B:BF16
- LM Studio
- Jan
- vLLM
How to use EldanRing/Winnow-E4B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "EldanRing/Winnow-E4B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "EldanRing/Winnow-E4B", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/EldanRing/Winnow-E4B:BF16
- Ollama
How to use EldanRing/Winnow-E4B with Ollama:
ollama run hf.co/EldanRing/Winnow-E4B:BF16
- Unsloth Desktop
- Docker Model Runner
How to use EldanRing/Winnow-E4B with Docker Model Runner:
docker model run hf.co/EldanRing/Winnow-E4B:BF16
- Lemonade
How to use EldanRing/Winnow-E4B with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull EldanRing/Winnow-E4B:BF16
Run and chat with the model
lemonade run user.Winnow-E4B-BF16
List all available models
lemonade list
- Atomic Chat
Add verified benchmark figures to model card
Browse files- .gitattributes +3 -0
- README.md +12 -0
- assets/e4b-profiles.png +3 -0
- assets/e4b-quality.png +3 -0
- assets/e4b-speed-memory.png +3 -0
.gitattributes
CHANGED
|
@@ -36,3 +36,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 36 |
gguf/Winnow-E4B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
gguf/Winnow-E4B-BF16.gguf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
gguf/mmproj-Winnow-E4B.gguf filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
gguf/Winnow-E4B-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
gguf/Winnow-E4B-BF16.gguf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
gguf/mmproj-Winnow-E4B.gguf filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
assets/e4b-profiles.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
assets/e4b-quality.png filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
assets/e4b-speed-memory.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -79,8 +79,16 @@ These figures evaluate the **downloadable Q8 GGUF** against Winnow-12B Q8 on the
|
|
| 79 |
|
| 80 |
Winnow-12B leads on the broader public suites. E4B leads by 2.2 points on this local typed panel, which uses synthetic teacher targets rather than independently verified truth for every item. The JevBench figure is **public-subset accuracy**, not the official composite leaderboard score. Methods, uncertainty intervals, calibration measures, and evaluation boundaries are in the [benchmark report](docs/BENCHMARKS.md).
|
| 81 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
In a separate matched 8K text decision workload, E4B Q8 delivered **147.8 decisions/s** at 64 questions per request, versus **67.2 decisions/s** for Winnow-12B Q8. Those are warm request medians on an RTX 5070 Ti, not a guarantee for other hardware or workloads. The [timing table](docs/BENCHMARKS.md#same-gpu-latency-and-memory) includes smaller batches and memory use.
|
| 83 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
## Probabilities and calibration
|
| 85 |
|
| 86 |
The server normalizes candidate logits over the options in each question. The optional `winnow.temperature` setting scales those logits without changing which option wins. For **Q8 text decisions**, `1.2574172017327816` was fitted on 778 separate calibration questions. For **BF16**, the separately fitted value is `1.3331553765162731`; do not reuse Q8's temperature for BF16.
|
|
@@ -104,6 +112,10 @@ Download Q8 for the measured decision, chat, and vision setup; add the projector
|
|
| 104 |
|
| 105 |
Capacity suggestions come from one RTX 5070 Ti using Q8 KV cache, four decision branches, one chat slot, and exclusive memory scheduling. They have not been tested on every GPU in those tiers. The Q8 model and projector answered six questions over a 65,023-position state that included a synthetic image; repeated cached requests took 129–131 ms after an 11.59 s cold request. A separate 62,431-position multimodal chat prompt generated 512 tokens at 71.2 output tokens/s. The [benchmark report](docs/BENCHMARKS.md#64k-context-and-vision-smoke-test) describes these operational probes and their limits.
|
| 106 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
## Scope and limitations
|
| 108 |
|
| 109 |
- The decision path is intended for supplied answer options. Open-ended responses use the regular chat endpoint.
|
|
|
|
| 79 |
|
| 80 |
Winnow-12B leads on the broader public suites. E4B leads by 2.2 points on this local typed panel, which uses synthetic teacher targets rather than independently verified truth for every item. The JevBench figure is **public-subset accuracy**, not the official composite leaderboard score. Methods, uncertainty intervals, calibration measures, and evaluation boundaries are in the [benchmark report](docs/BENCHMARKS.md).
|
| 81 |
|
| 82 |
+

|
| 83 |
+
|
| 84 |
+
*Decision quality on the same questions. The local typed panel measures agreement with synthetic teacher labels.*
|
| 85 |
+
|
| 86 |
In a separate matched 8K text decision workload, E4B Q8 delivered **147.8 decisions/s** at 64 questions per request, versus **67.2 decisions/s** for Winnow-12B Q8. Those are warm request medians on an RTX 5070 Ti, not a guarantee for other hardware or workloads. The [timing table](docs/BENCHMARKS.md#same-gpu-latency-and-memory) includes smaller batches and memory use.
|
| 87 |
|
| 88 |
+

|
| 89 |
+
|
| 90 |
+
*Matched text-only workload on an RTX 5070 Ti with Q8 weights and KV cache. Timings are medians of ten warm requests.*
|
| 91 |
+
|
| 92 |
## Probabilities and calibration
|
| 93 |
|
| 94 |
The server normalizes candidate logits over the options in each question. The optional `winnow.temperature` setting scales those logits without changing which option wins. For **Q8 text decisions**, `1.2574172017327816` was fitted on 778 separate calibration questions. For **BF16**, the separately fitted value is `1.3331553765162731`; do not reuse Q8's temperature for BF16.
|
|
|
|
| 112 |
|
| 113 |
Capacity suggestions come from one RTX 5070 Ti using Q8 KV cache, four decision branches, one chat slot, and exclusive memory scheduling. They have not been tested on every GPU in those tiers. The Q8 model and projector answered six questions over a 65,023-position state that included a synthetic image; repeated cached requests took 129–131 ms after an 11.59 s cold request. A separate 62,431-position multimodal chat prompt generated 512 tokens at 71.2 output tokens/s. The [benchmark report](docs/BENCHMARKS.md#64k-context-and-vision-smoke-test) describes these operational probes and their limits.
|
| 114 |
|
| 115 |
+

|
| 116 |
+
|
| 117 |
+
*Q8 operational profiles on an RTX 5070 Ti. The 8K text workload was matched; the 64K vision and chat probes used different images and question counts. The 10 GB+ and 12 GB+ capacity suggestions are inferred from measured memory, not tests on those cards.*
|
| 118 |
+
|
| 119 |
## Scope and limitations
|
| 120 |
|
| 121 |
- The decision path is intended for supplied answer options. Open-ended responses use the regular chat endpoint.
|
assets/e4b-profiles.png
ADDED
|
Git LFS Details
|
assets/e4b-quality.png
ADDED
|
Git LFS Details
|
assets/e4b-speed-memory.png
ADDED
|
Git LFS Details
|