Card: official Jev Decision Index 0.3.2 results and serving performance

.gitattributes

@@ -37,3 +37,4 @@ assets/banner.png filter=lfs diff=lfs merge=lfs -text
 assets/index-areas.png filter=lfs diff=lfs merge=lfs -text
 assets/index-pareto.png filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/serving-throughput.png filter=lfs diff=lfs merge=lfs -text

MODEL_MANIFEST.json

@@ -582,12 +582,13 @@
  "files_sha256": {
   "LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
   "NOTICE": "0e7694dee1927d83b3637be7a8f989b941ebd884dd40767b995c7adee484789b",
-  "README.md": "8cbc6ffab18ce179f7ab4bd69201c3aeaf8efb2b6a17fa0dacef7cd468e03d70",
+  "README.md": "ec8a4da255507f7fdeb4022f3c4ecaff056915f112bb2e4f28a3385b0f4b0422",
   "assets/banner.png": "352353b5ac097e31e6447ac0db8185439894e9e6c66376565e30f33c39d27913",
   "assets/example-receipt.png": "b9dbb8b103bacd45f1de824d844a3c07b81d57cb1a7466712ef060ad7dcb3926",
   "assets/example-video.mp4": "f4968ef89d07e97f20df4fd8047510a7a8e9df5c2d07c35e363a97927f164ab9",
-  "assets/index-areas.png": "06be306799c515e90bea74994788239a53c01c893183879b18192cebe0158531",
-  "assets/index-pareto.png": "13ee761164b7fe0e99a8a48a942d783a759196a4a1eb466b82ada6bc8ce36daa",
+  "assets/index-areas.png": "b2659c49dd9b990ae61e9f26ff3dd2d1e76e0d9a993301b8707f06e89b0b32e5",
+  "assets/index-pareto.png": "70238ccc7b77efeb873007b6464b2d00295a2eea9fc0d59fc82a3d52d79cd4b5",
+  "assets/serving-throughput.png": "e324024e87c1d1538def9813bc64d6fd9117ad1f5f81f357771717cd678bd804",
   "chat_template.jinja": "273d8e0e683b885071fb17e08d71e5f2a5ddfb5309756181681de4f5a1822d80",
   "config.json": "5e529d9aea2f8d387750acf5b02cc32a47d7e87dc65d6d008fba98e5d15acc4d",
   "d3_engine.py": "086238942762df45047a32cd9a8e2bedd9b63167eb40be260c776de567979f53",
@@ -613,12 +614,13 @@
  "files_bytes": {
   "LICENSE": 11357,
   "NOTICE": 106,
-  "README.md": 10345,
+  "README.md": 16328,
   "assets/banner.png": 2235551,
   "assets/example-receipt.png": 65918,
   "assets/example-video.mp4": 26300,
-  "assets/index-areas.png": 123190,
-  "assets/index-pareto.png": 186971,
+  "assets/index-areas.png": 124151,
+  "assets/index-pareto.png": 209711,
+  "assets/serving-throughput.png": 175072,
   "chat_template.jinja": 7755,
   "config.json": 2572,
   "d3_engine.py": 4413,

README.md

@@ -29,11 +29,12 @@ tags:
 
 ## Highlights
 
-- **Jev Decision Index 0.3, public suite: 28.82**, measured with the official 0.3 kit on the released weights: all 140,178 public requests answered, none unsupported.
-- **+12.1 on the public suite over Decision 2.0** (its 0.6B model: 16.68 on the board), ahead in all five areas.
+- **Jev Decision Index 0.3.2: 20.2 on the official text board**, second among the entries of similar size; public suite 28.82, measured with the official 0.3 kit on the released weights: all 140,178 public requests answered, none unsupported.
+- **+6.6 over Decision 2.0** (its 0.6B model: 13.6 on the official board): +12.1 on the public suite, ahead in all five of its areas, and +7.4 and +1.1 on the private same-skill tests and new-domain tasks.
 - **Reads images:** multiple images per request (PNG, JPEG or WebP), given as paths, URLs, PIL images or base64 data URLs; every question of the request sees all of them.
 - **Reads videos:** multiple videos per request (MP4, WebM, MOV or MKV), given as paths, URLs, base64 data URLs or frame arrays, read at 2 frames per second; **Perception Test (multiple-choice video QA): 46.6** (internal evaluation).
-- **Speed:** a median of 6.7 ms for a text request, 41.9 ms for a request with an image and 308.3 ms for a request with a 10-second video, on one AMD Instinct MI325X GPU, one request at a time.
+- **Speed:** a median of 6.6 ms for a text request, 58.6 ms for a request with an image and 274 ms for a request with a 10-second video, on one AMD Instinct MI325X GPU, one request at a time with AutoModel.
+- **Serving:** 861 requests/s with p99 under 1 s on one AMD Instinct MI325X with vLLM Semantic Router, 6.1× AutoModel (one-question text, `batching` profile); the default `exact` profile gives AutoModel's answers byte for byte when both use the same kernel settings ([Serving performance](#serving-performance)).
 - **Many questions, one call:** Choice, Yes / No and Score questions about the same input are answered together, each from its own forward pass over the input, with a probability for every option.
 
 ## Quickstart
@@ -199,28 +200,79 @@ EOF
 
 `vllm-sr stop` stops it. See [Router and Engine modes](https://github.com/vllm-project/semantic-router/blob/main/src/vllm-sr/INSTANCE_MODES.md#ask-about-images-and-videos) for limits and options.
 
+### Serving performance
+
+**On one AMD Instinct MI325X, vLLM Semantic Router serves d3-edge at 861 requests/s with p99 under 1 s, 6.1× AutoModel** (one-question text, `batching` profile), and its default `exact` profile gives AutoModel's answers byte for byte when both use the same kernel settings.
+
+![Serving throughput of d3-edge on one AMD Instinct MI325X: AutoModel, the package server and vLLM Semantic Router Engine mode](assets/serving-throughput.png)
+
+AutoModel runs in your own process, one call at a time (the Quickstart code); the package server is its `python d3_server.py`; Engine mode is `vllm-sr serve -e` with its default `exact` profile, or with `--runtime-profile batching`.
+
+Where Engine mode helps, and what it costs:
+
+- **Throughput:** 6.1× AutoModel's requests per second on one-question text, 5.0× on 8-question images and 2.4× on 10-second videos, with the better Engine profile of each row.
+- **Start-up:** the first answer comes 11.6 s after launch with empty kernel caches, against 24.9 s for AutoModel, whose first image then takes another 1.13 s, and 36.5 s for the package server.
+- **Many questions and long inputs:** one request at a time, Engine mode (`exact`) answers 16 text questions in 21.4 ms (AutoModel: 22.4 ms), 8 questions about one image in 215 ms (267 ms) and a 32k-token text in 268 ms (281 ms); `batching` runs the 8 image questions together in 111 ms, 58 % less than AutoModel.
+- **GPU memory:** 15.9 GiB per replica at peak with `exact`, above the package server's 12.1 GiB, and 19.9 GiB with `batching`; AutoModel, serving one request at a time, peaked at 5.2 GiB. Engine mode keeps up to 2 GiB of image features cached.
+- **Answers:** Engine mode (`exact`) returned answers byte-identical to AutoModel's on all 7 request types (text with 1, 4 or 16 questions, an image with 1 or 8, the video and a 32k-token text) when AutoModel used the same kernel settings: 140 Engine replies and 21 AutoModel calls compared. With the kernel autotuning AutoModel runs in each new process, 6 of the 7 were byte-identical and no decision changed (largest probability difference 0.009). `batching` changed 0.151 % of decisions on the complete public suite (487 of 323,259 questions, near-ties), with gold accuracy 61.702 → 61.694 % and the index 28.82 → 28.82.
+
+A single short request is fastest in your own process: AutoModel answers one text question 1.3 ms sooner than Engine mode and one new image 26 ms sooner, the cost of going through the router to the model worker and its media path. Engine mode's gains come from serving many requests at once.
+
+<details>
+<summary>Single-request latency and the numbers behind the chart</summary>
+
+One request at a time, warm (p50 / p90, ms):
+
+| Request | AutoModel | Package server | Engine, `exact` | Engine, `batching` |
+| --- | ---: | ---: | ---: | ---: |
+| Text, 1 question | 6.6 / 7.2 | 7.0 / 7.4 | 7.9 / 8.4 | 9.9 / 10.8 |
+| Text, 16 questions | 22.4 / 24.1 | 22.1 / 22.4 | 21.4 / 22.1 | 20.6 / 21.0 |
+| Image (1.6 MP), 1 question | 58.6 / 59.7 | 61.8 / 63.4 | 84.9 / 87.0 | 87.7 / 93.0 |
+| Image (1.6 MP), 8 questions | 267 / 269 | 261 / 267 | 215 / 219 | 111 / 113 |
+| Video (10 s, 1080p), 1 question | 274 / 279 | 256 / 265 | 294 / 309 | 309 / 324 |
+| 32k-token text, 1 question | 281 / 285 | 285 / 308 | 268 / 269 | 274 / 282 |
+
+Requests per second on one GPU, closed loop with up to 64 clients (p99 under 1 s, except *):
+
+| | AutoModel † | Package server | Engine, `exact` | Engine, `batching` |
+| --- | ---: | ---: | ---: | ---: |
+| Text, 1 question | 142 | 137 | 127 | 861 |
+| Image (1.6 MP), 8 questions | 3.7 | 3.9 | 6.2 | 18.8 |
+| Video (10 s, 1080p), 1 question | 3.7 | 7.2 | 7.7 | 8.7 |
+| Text, 1 question, saturation | 142 | 137 | 127 | 861 |
+
+† Back-to-back calls in one process (1,000 / mean latency). * p99 over 1 s at every load; the highest rate is shown.
+
+</details>
+
+<sub>Measured on 2026-10-13, one AMD Instinct MI325X per replica and each side alone on the GPU: ROCm 7.2 with PyTorch 2.12; AutoModel and the package server with Transformers 5.17.0; vLLM Semantic Router Engine mode built from source, with serving optimizations that are newer than its published images. Requests: a support ticket of about 330 tokens with 1, 4 or 16 questions; a 1,456 × 1,092 PNG (1.6 MP) with 1 or 8 questions; a 10-second 1080p MP4, read at 2 frames per second; a 32k-token text; every image and video request carries new bytes. One request at a time: 3 warm-up calls, then up to 20 timed calls. Under load: 1, 2, 4, … 64 closed-loop clients, each level warmed up and then counted for 15 to 60 s; Engine mode admits 32 requests in flight per worker and answers the rest at once with 503. Start-up: model files on local disk; Engine mode from the start of its router process, without the CLI's image pull.</sub>
+
 ## Evaluation
 
-### Text: Jev Decision Index 0.3.1
+### Text: Jev Decision Index 0.3.2
 
 | Model | Jev Decision Index ↑ | Public ↑ | Same-skill tests ↑ | New-domain tasks ↑ |
 | --- | ---: | ---: | ---: | ---: |
-| **d3-edge** | **18.0** | **28.8** | **23.9** | **8.0** |
-| Bekko System One v0 400M | 15.2 | 22.5 | 21.6 | 7.2 |
-| Dinah-0 | 13.9 | 26.1 | 18.8 | 5.5 |
+| **d3-edge** | **20.2** | **28.8** | **24.3** | **12.9** |
+| jiwo 0.8B v2 | 25.6 | 29.4 | 30.6 | 17.9 |
 | Decision 2.0 (0.6B) | 13.6 | 16.7 | 16.9 | 11.8 |
-| LiquidAI d1-omni-600M | 9.5 | 17.9 | 13.6 | 5.1 |
+| vera-spark | 11.5 | 19.6 | 15.1 | 7.3 |
 | GLiNER2.5-Decide | 9.2 | 11.5 | 12.1 | 9.6 |
+| LiquidAI d1-omni-600M | 9.1 | 17.9 | 12.9 | 5.1 |
+
+d3-edge is not on the official text board's Pareto frontier: Crivo-150M scores 22.2 with 0.15 billion parameters. Among the eight entries within 0.75 to 1.33 times its size, d3-edge ranks second, behind jiwo 0.8B v2 at 25.6. The other five Decision 3.0 models are on the frontier.
 
 ![Jev Decision Index against model size](assets/index-pareto.png)
 
 ![Jev Decision Index by area: d3-edge and Decision 2.0 (0.6B)](assets/index-areas.png)
 
-### Images: Jev Decision Index vision board 0.3.1
+### Images: Jev Decision Index vision board 0.3.2
 
 | Model | Vision Index ↑ | Public ↑ | Private ↑ |
 | --- | ---: | ---: | ---: |
-| **d3-edge** | **35.6** | **37.5** | **33.6** |
+| **d3-edge** | **31.7** | **36.6** | **26.8** |
+
+On the official vision board, the six Decision 3.0 models form the whole Pareto frontier: no other entry is on it. d3-edge is the smallest entry on the board.
 
 d3-edge on the public vision benchmarks († approximate rebuild):
 
@@ -250,7 +302,7 @@ d3-edge on the public vision benchmarks († approximate rebuild):
 
 Multiple-choice video question answering (three options per question, chance 33.3): top-1 accuracy (%), videos read with the defaults above.
 
-<sub>d3-edge: internal evaluation. Others: live board data, text 2026-10-10, vision 2026-10-09.</sub>
+<sub>Jev Decision Index: official leaderboard, text and vision boards 0.3.2, 2026-10-12; the boards evaluated v3.1.0, which has the same weights as this version. Size bands and the Pareto frontier use the parameter counts listed by the boards. d3-edge on the public vision benchmarks and the Perception Test: internal evaluation.</sub>
 
 ## License