Praveenrajus/jevify-qwen3-vl-2b

Qwen3-VL-2B, Jevified (Tier 0)

A Jevify Tier 0 model: Qwen/Qwen3-VL-2B-Instruct used as a vision-language model: state may carry images (PIL, path, URL, data URI or bytes) and the same three typed questions are asked about them, with no training — one forward pass, the probability of every allowed answer read from a single position, and a calibration recipe fitted on validation splits only. This repo carries the recipe and a pointer to the backbone; nothing else is needed.

At a glance — macro accuracy 0.779, macro ECE 0.047 over POPE, A-OKVQA and AI2D — no training

jev-bench · leaderboard · findings · code

Use it

from jevify import load_jevified

model = load_jevified("Praveenrajus/jevify-qwen3-vl-2b")
model.ask({"image": "https://example.com/photo.jpg", "question": "Is there a cat?"},
          {"q": {"type": "noul", "instructions": "Is the answer to `question` yes, based on `image`?"}})

Serve it as a drop-in for the TypeSafe SDK: jevify-serve --model Praveenrajus/jevify-qwen3-vl-2b then TYPESAFE_BASE_URL=http://localhost:8000.

Results on jev-bench test splits

Macro accuracy 0.779, macro ECE 0.047 over 3 sources.

source	primitive	n	acc	ECE	Brier
ai2d	choice	1500	0.652	0.059	0.454
aokvqa	choice	744	0.793	0.037	0.299
pope	noul	2000	0.891	0.046	0.079

Recipe: mode index, permutations {'choice': 1, 'score': 1, 'noul': 1}, temperature {'choice': 1.9176, 'score': 1.0, 'noul': 2.6309}, Noul bias {'noul': 1.8}. Predictions, metrics and figures: results/qwen3vl-2b/ on Praveenrajus/jev-bench. Findings and method: docs/FINDINGS.md.

How it was built

No training. Each question is asked with its image in the prompt, and the probability of every allowed answer is read at the answer position in one forward pass. A recipe — one temperature per primitive and a Noul bias — is then fitted on validation splits only (POPE ships a single split, so a held-out 10% of it serves as validation and is excluded from the test numbers).

Related models

• Same backbone with a decoder LoRA trained on the readout (Tier 2)

Limitations

• Measured on three vision sources (POPE, A-OKVQA, AI2D); other image tasks need their own recipe fit.
• The raw readout is over-confident (fitted Choice temperature 1.92); always load it through load_jevified, which applies the recipe.
• English only.