# SafeRide v0.5.8 external standard benchmark execution plan

**Date:** August 13, 2026

**Status:** Ready for mentor selection and execution; no external-standard
scores are claimed in this document

**Artifact under review:** SafeRide Gemma 4 E2B v0.5.8 adapter and the exact
merged LiteRT-LM artifact with SHA-256
`8b73fd844464f220955eeedc474c30f39e621458c7a6b092de5afa2c3d027fcd`

## 1. Purpose

UNICEF checkpoint correspondence asks SafeRide to publish standard benchmarks
on Hugging Face alongside the released Gemma model. SafeRide has published a
standardized CSV/JSON export of its internal aggregate evidence, but those
results are correctly labeled `internal-custom-not-external-standard`.

This plan defines a reproducible external benchmark package so a technical
mentor can approve the suite before compute is spent. It does not reinterpret
the existing internal results as external benchmarks.

## 2. Proposed minimum external suite

| Benchmark | Proposed task | Reason for inclusion | Required slice |
| --- | --- | --- | --- |
| TruthfulQA | `truthfulqa_mc2` | Established truthfulness/misconception benchmark relevant to unsupported factual certainty. | Overall score and exact sample count |
| Belebele | `belebele_eng_Latn` | Parallel reading-comprehension evidence for English. | English |
| Belebele | `belebele_swh_Latn` | Parallel reading-comprehension evidence for Kiswahili. | Kiswahili |
| IFEval | `ifeval` | Established instruction-following benchmark that helps detect broad degradation after adaptation. | Overall strict and loose accuracy where supplied by the harness |

Authoritative references:

- EleutherAI `lm-evaluation-harness`:
  https://github.com/EleutherAI/lm-evaluation-harness
- Belebele paper and dataset description:
  https://arxiv.org/abs/2308.16884
- TruthfulQA paper:
  https://arxiv.org/abs/2109.07958

The mentor may replace or add tasks, but the approved task names, harness
revision, prompt format, shot count, metric names, and sample counts must be
frozen before execution.

## 3. Evaluated objects

Run the approved suite against:

1. the pinned base model
   `google/gemma-4-E2B-it@70af34e20bd4b7a91f0de6b22675850c43922a03`;
2. the pinned SafeRide adapter
   `esherialabs/saferide-gemma-4-e2b-v058-original-419806-adapter@019dd8182883ad0721ffa70f4680d6977b7be99b`;
   and
3. a deterministic parity subset replayed through the exact `.litertlm`
   artifact where the benchmark adapter supports only Transformers/PEFT.

Base and adapter runs must use identical generation/scoring settings. Results
from the PEFT adapter must not be described as direct `.litertlm` scores. The
mobile parity subset must be reported separately.

## 4. Reproducibility contract

The execution record must bind:

- exact benchmark harness repository revision and package version;
- exact benchmark dataset revisions;
- base, adapter, tokenizer, chat-template and mobile-artifact revisions/hashes;
- prompt/chat-template mode, few-shot count, batch size and decoding settings;
- operating system, Python, PyTorch, Transformers, PEFT and accelerator
  versions;
- GPU/CPU model and memory;
- task name, sample count, metric name, score and standard error where
  available;
- command line with credentials and private paths removed; and
- complete output-file hashes.

Do not use `--limit` for the checkpoint result unless the mentor explicitly
approves a pilot subset. Any pilot subset must be labeled as such and cannot be
presented as the full benchmark.

## 5. Required outputs

Publish public-safe files beside the model:

- `benchmarks/external-standard-benchmark-config.json`
- `benchmarks/external-standard-benchmark-results.json`
- `benchmarks/external-standard-benchmark-results.csv`
- `benchmarks/external-standard-benchmark-environment.json`
- `benchmarks/external-standard-benchmark-report.md`

The public package must not include raw private prompts, hidden instructions,
credentials, local paths, survivor narratives, or unrestricted model
completions. Standard benchmark item identifiers and official dataset content
must follow the benchmark's own license and publication rules.

## 6. Review and acceptance

| Decision | Accountable role | Target date | Current status |
| --- | --- | --- | --- |
| Approve or amend the external task set | UNICEF/technical AI mentor | August 17, 2026 | Pending external decision |
| Provision approved compute with pinned base already available, or authorize the required downloads | ML release owner | August 18, 2026 | Blocked; no retained approved model environment is currently available |
| Execute and hash the complete suite | ML evaluation lead | August 21, 2026 | Pending task approval and compute |
| Independent review of configuration and results | Technical/ML reviewer | August 22, 2026 | Reviewer unassigned |
| Publish and byte-verify result files on Hugging Face | ML release owner | August 22, 2026 | Pending completed results |

## 7. Current blocker

The accessible execution host does not retain the exact v0.5.8 model package
or pinned base snapshot, and the current verification scope prohibits silently
initiating new multi-gigabyte model downloads. Therefore the external suite
cannot be truthfully executed until an approved compute environment or explicit
download authorization is provided.

The existing internal aggregate benchmark files may still be published now;
their classification must remain `internal-custom-not-external-standard` until
this plan is executed or a mentor formally accepts the internal suite as the
checkpoint's intended standard benchmark.