# Benchmark report layout

Final-model measurements are pending. Tables below define what will be reported;
blank values are not measurements. Keep quality and runtime results separate.

## Model quality

Compare the selected Winnow checkpoint, its matching stock Gemma base and Jev
where equivalent saved evaluations are available. Identify the Jev version/date
and any difference in access method or prompting.

| Benchmark / split | Questions | Stock Gemma | Winnow BF16 | Winnow Q8 | Jev |
|---|---:|---:|---:|---:|---:|
| Final held-out typed decisions | Pending | Pending | Pending | Pending | Pending |
| Public decision benchmark (exact subset/version stated) | Pending | Pending | Pending | Pending | Pending |

Accompany accuracy with NLL, Brier score and ECE for probability-bearing tasks,
and score MAE for ordered outcomes, where those metrics are applicable. Report
raw and fitted-temperature results separately. Use calibration-only examples
to fit temperature; use the untouched test split to evaluate it. Document any
known training exposure or benchmark limitations.

Chat retention should identify the established benchmark/subset, full decoding
settings, 4,096-token allowance, stop reasons and judging method. Report vision
checks separately. Short samples support scoped comparisons, not broad parity claims.

## Hardware performance

Each row will identify model/projector SHA256, runtime/source revision, GPU/VRAM,
KV type, context capacity, actual text/image positions and parallel branches.

| GPU | Context / actual positions | Vision | KV / branches | Cold prefill tok/s | Generation tok/s | First-content latency | Peak VRAM |
|---|---|---|---|---:|---:|---:|---:|
| RTX 5070 Ti 16 GB | Pending final artifact | Pending | Pending | Pending | Pending | Pending | Pending |
| RTX PRO 5000 | Earlier candidate; identity required | Pending | Pending | Pending | Pending | Pending | Pending |

| GPU / artifact | Questions per shared state | Cold batch ms | Warm median ms | Warm questions/s | Samples |
|---|---:|---:|---:|---:|---:|
| Pending | 1 / 8 / 16 / 32 / 64 | Pending | Pending | Pending | Pending |

Use saved short-run measurements first. New testing should target the final
artifact and missing release claims, not repeat a broad parameter sweep.
Cloud speed data from a different checkpoint may be reported with that candidate
explicitly identified; do not relabel it as a measurement of the released model.

Publish the benchmark version, rendering, sample count, input/output token counts,
timing boundaries, cache state, repetitions, stop reason and launch command with
the results. Prefill, generation and end-to-end response time measure different
work. Synthetic context-filling probes establish capacity and basic retrieval.
Do not infer reliable tail latency from a handful of repetitions.

Private training examples, teacher responses and raw private benchmark responses
are excluded from the public report. Aggregate metrics and reproducible public
or synthetic fixtures can be shared independently.