SafeRide v0.5.8 external standard benchmark execution plan

Date: August 13, 2026

Status: Ready for mentor selection and execution; no external-standard scores are claimed in this document

Artifact under review: SafeRide Gemma 4 E2B v0.5.8 adapter and the exact merged LiteRT-LM artifact with SHA-256 8b73fd844464f220955eeedc474c30f39e621458c7a6b092de5afa2c3d027fcd

1. Purpose

UNICEF checkpoint correspondence asks SafeRide to publish standard benchmarks on Hugging Face alongside the released Gemma model. SafeRide has published a standardized CSV/JSON export of its internal aggregate evidence, but those results are correctly labeled internal-custom-not-external-standard.

This plan defines a reproducible external benchmark package so a technical mentor can approve the suite before compute is spent. It does not reinterpret the existing internal results as external benchmarks.

2. Proposed minimum external suite

Benchmark	Proposed task	Reason for inclusion	Required slice
TruthfulQA	`truthfulqa_mc2`	Established truthfulness/misconception benchmark relevant to unsupported factual certainty.	Overall score and exact sample count
Belebele	`belebele_eng_Latn`	Parallel reading-comprehension evidence for English.	English
Belebele	`belebele_swh_Latn`	Parallel reading-comprehension evidence for Kiswahili.	Kiswahili
IFEval	`ifeval`	Established instruction-following benchmark that helps detect broad degradation after adaptation.	Overall strict and loose accuracy where supplied by the harness

Authoritative references:

EleutherAI lm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness
Belebele paper and dataset description: https://arxiv.org/abs/2308.16884
TruthfulQA paper: https://arxiv.org/abs/2109.07958

The mentor may replace or add tasks, but the approved task names, harness revision, prompt format, shot count, metric names, and sample counts must be frozen before execution.

3. Evaluated objects

Run the approved suite against:

the pinned base model google/gemma-4-E2B-it@70af34e20bd4b7a91f0de6b22675850c43922a03;
the pinned SafeRide adapter esherialabs/saferide-gemma-4-e2b-v058-original-419806-adapter@019dd8182883ad0721ffa70f4680d6977b7be99b; and
a deterministic parity subset replayed through the exact .litertlm artifact where the benchmark adapter supports only Transformers/PEFT.

Base and adapter runs must use identical generation/scoring settings. Results from the PEFT adapter must not be described as direct .litertlm scores. The mobile parity subset must be reported separately.

4. Reproducibility contract

The execution record must bind:

exact benchmark harness repository revision and package version;
exact benchmark dataset revisions;
base, adapter, tokenizer, chat-template and mobile-artifact revisions/hashes;
prompt/chat-template mode, few-shot count, batch size and decoding settings;
operating system, Python, PyTorch, Transformers, PEFT and accelerator versions;
GPU/CPU model and memory;
task name, sample count, metric name, score and standard error where available;
command line with credentials and private paths removed; and
complete output-file hashes.

Do not use --limit for the checkpoint result unless the mentor explicitly approves a pilot subset. Any pilot subset must be labeled as such and cannot be presented as the full benchmark.

5. Required outputs

Publish public-safe files beside the model:

benchmarks/external-standard-benchmark-config.json
benchmarks/external-standard-benchmark-results.json
benchmarks/external-standard-benchmark-results.csv
benchmarks/external-standard-benchmark-environment.json
benchmarks/external-standard-benchmark-report.md

The public package must not include raw private prompts, hidden instructions, credentials, local paths, survivor narratives, or unrestricted model completions. Standard benchmark item identifiers and official dataset content must follow the benchmark's own license and publication rules.

6. Review and acceptance

Decision	Accountable role	Target date	Current status
Approve or amend the external task set	UNICEF/technical AI mentor	August 17, 2026	Pending external decision
Provision approved compute with pinned base already available, or authorize the required downloads	ML release owner	August 18, 2026	Blocked; no retained approved model environment is currently available
Execute and hash the complete suite	ML evaluation lead	August 21, 2026	Pending task approval and compute
Independent review of configuration and results	Technical/ML reviewer	August 22, 2026	Reviewer unassigned
Publish and byte-verify result files on Hugging Face	ML release owner	August 22, 2026	Pending completed results

7. Current blocker

The accessible execution host does not retain the exact v0.5.8 model package or pinned base snapshot, and the current verification scope prohibits silently initiating new multi-gigabyte model downloads. Therefore the external suite cannot be truthfully executed until an approved compute environment or explicit download authorization is provided.

The existing internal aggregate benchmark files may still be published now; their classification must remain internal-custom-not-external-standard until this plan is executed or a mentor formally accepts the internal suite as the checkpoint's intended standard benchmark.