# Third-party projects, data and services

MIT (see `LICENSE`) covers this harness and the 72 original public decisions in
`datasets/public/original.jsonl`. Nothing else in this list is ours to license.

## Systems evaluated

Each was reached through the interface its author published. We used public
endpoints as ordinary clients, at one request at a time, and never sent a
provider's API key to anyone else's endpoint.

| System | Author | Source |
|---|---|---|
| Jev 1.13.0 | TypeSafe AI | <https://docs.typesafe.ai> |
| openjev-sglang | ekzhang | <https://github.com/ekzhang/openjev-sglang> |
| system-one-open | mithalouni | <https://github.com/mithalouni/system-one-open> |
| open-alternative-jev | IkerMoel | <https://github.com/ikermoel/open-alternative-jev>, <https://huggingface.co/spaces/IkerMoel/open-alternative-jev> |
| open-jev-deberta-v3-large | Kotoba Labs | <https://huggingface.co/com-kotobalabs/open-jev-deberta-v3-large>, <https://github.com/kotoba-lang/typed-decisions> |
| GPT-5.6 Luna | OpenAI | <https://platform.openai.com> |
| Gemini 3.1 Flash-Lite | Google | <https://ai.google.dev> |
| DeepSeek V4.1 Flash | DeepSeek | <https://api-docs.deepseek.com> |
| Qwen3.8 27B | Qwen, served by Chutes | <https://chutes.ai> |

Model weights, base models and each project's own code keep their own licences.
A permissive licence on a repository is not a licence for the base model it
fine-tunes, and we do not restate either.

## Wire format

The typed-decision request shape (`state`, `questions`, the `noul` / `choice` /
`score` primitives) is TypeSafe's public HTTP API, documented at
<https://docs.typesafe.ai/api>. Several of the open rebuilds implement it
deliberately, which is why one adapter reaches more than one of them. JevBench is
not affiliated with or endorsed by TypeSafe AI.

## Imported decisions

146 of the 242 decisions come from our own earlier auto-router experiment
(<https://github.com/fstandhartinger/auto-model-router>): 78 routing requests and
68 answer-adequacy judgements whose ground truth is a deterministic grader's
verdict on a saved answer. The upstream task text is **not** redistributed here,
because the datasets it was drawn from keep their own terms. `datasets/manifest.json`
pins the hashes; `scripts/import_router.py` shows exactly what was taken and what
was excluded.

## Held-out decisions

24 scenarios are written by us and deliberately unpublished, so the suite cannot
be trained on in full. Only their whole-split hash and aggregate results appear
here. They are sent to the services being evaluated, which is not the same thing
as being public - see the limits section of the README.