Akiki-MiniLM-L6-v2-clover-intent

Developed by AKIKI.AI.

An English six-way intent router for Clover, built on a completely frozen sentence-transformers/all-MiniLM-L6-v2. Only two small pieces were trained: an identity-initialized 384 × 384 Akiki input affine on the word embeddings (147,840 parameters) and a six-class head (2,310 parameters), 150,150 in total. The 22,713,216 backbone parameters are untouched and hash-checked against the original.

Given the current user message and, optionally, earlier conversation turns, it returns one of:

Token	Label	Meaning
A	chat	Conversation, explanation, how-to advice, general knowledge
B	question	Inspect or report the state of the user's actual system, without changing it
C	build	Create or modify software, a feature, a site or an app
D	task	Perform a specified operational action (restart, move, change a setting)
E	fix	Diagnose and remedy a malfunction when no specific remedy is prescribed
F	unclear	The route or referent cannot be identified from message and context

Experimental. Labels are synthetic and model-reviewed; human review is pending. This model classifies intent only. It does not write contracts, run actions, or implement Clover's refusal policy. It has not been approved for serving.

Results

Held-out test split: 714 examples (120 per class; 114 unclear), balanced across easy / medium / hard.

Method (MiniLM-L6-v2, frozen backbone)	Test accuracy	Macro F1	Hard accuracy
Label-description similarity, no training	34.9%	0.313	36.7%
Frozen embeddings + regularized linear classifier	92.3%	0.923	91.7%
Head-only training, 3-seed mean	91.7% ± 0.3	0.917	91.1%
Akiki affine + head, 3-seed mean	96.6% ± 0.6	0.966	95.6%
This checkpoint (Akiki, seed 2)	97.2%	0.972	96.2%

Which seed, and why: three seeds were trained. Seed 2 is released because it has the lowest validation NLL (0.121 vs 0.131 and 0.141), which is the experiment's checkpoint criterion. Test results played no part in the choice. Seed 2 happens to be the best on test as well, so treat the three-seed mean (96.6%) as the honest estimate of what this recipe gives. All three seeds ship in checkpoints/.

Seed 2 detail: easy 99.6%, medium 95.8%, hard 96.2%; test NLL 0.082, Brier 0.043.

Class	Precision	Recall	F1
chat	0.991	0.958	0.975
question	0.967	0.992	0.979
build	1.000	0.967	0.983
task	0.934	0.950	0.942
fix	0.967	0.992	0.979
unclear	0.974	0.974	0.974

Rows are gold labels, columns are predictions:

gold \ pred	chat	question	build	task	fix	unclear
chat	115	2	0	1	0	2
question	1	119	0	0	0	0
build	0	0	116	3	0	1
task	0	2	0	114	4	0
fix	0	0	0	1	119	0
unclear	0	0	0	3	0	111

Pass mark not met. The experiment requires at least 95% accuracy and zero action predictions (build/task/fix) on chat/question. This checkpoint makes 1 action prediction on 240 chat/question examples (a chat message read as task) and 3 on 114 unclear examples. Downstream code should therefore not treat a task/build/fix label as authorization by itself.

Usage

pip install -r requirements.txt
python model.py 'Restart my web server.'                  # -> D
python model.py --output label 'Restart my web server.'   # -> task
python model.py --output json --context-file ctx.json 'yes'

from model import CloverIntent

clf = CloverIntent.from_pretrained(".")          # loads base_model/ + adapter.safetensors, CPU by default
clf.predict("Restart my web server.")["label"]   # 'task'
clf.predict("yes", context=[
    {"role": "assistant", "text": "Would you like me to check whether last night's backup ran?"},
])["label"]                                       # 'question'
clf.probabilities([{"message": "fix it"}])       # (1, 6) tensor, label order above

predict returns label, token (A–F) and the six probabilities. Everything loads from this repository with local_files_only=True; nothing is downloaded at run time. Download the whole repo first, e.g. hf download Akiki-AI/Akiki-MiniLM-L6-v2-clover-intent --local-dir clover-intent.

Input format

The model sees exactly this text (the context block is omitted when empty):

Conversation context:
assistant: Would you like me to restart the web server?

Current user message: yes

Context turns are {"role": ..., "text": ...} objects. The limit is 512 tokens including special tokens. Longer inputs raise an error rather than being truncated. The longest training/test input is 126 tokens, so these results say nothing about long briefings or full chat histories.

Probabilities are a plain softmax of the head logits, with no calibration fitted.

Architecture

tokens → MiniLM word embeddings (frozen)
       → Akiki affine  W·x + b   (384×384, identity-initialized, FP32)
       → + position/type embeddings → 6 BERT layers (frozen)
       → attention-masked mean pooling → L2 normalize
       → linear head (384 → 6) → softmax

adapter.safetensors holds interface.proj.{weight,bias} and head.{weight,bias}. base_model/ is an unmodified copy of all-MiniLM-L6-v2 at revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41.

Training

• Data: 9,913 train / 707 validation / 714 test synthetic English examples (data/, documented in data/README.md). Scenario groups never cross splits; near-duplicates across splits were removed, protecting test first. Hard examples need the conversation context to resolve the intent, using follow-up phrase families kept separate between splits.
• Labels: generated locally by Qwen3.8-27B-FP8 and independently classified by Qwen3.5-9B; disagreements were rejected. Not human-verified.
• Optimization: Adam; head lr 1e-3, affine lr 1e-4; early stopping on validation NLL. Seed 2 selected epoch 8 of 11. Each fit took about 30 s on one RTX PRO 6000. The full recipe is in provenance/original/protocol/plan.json.

Latency

Warm CPU inference with FP32 PyTorch/Transformers, 8 threads, batch size 1, on a short contextual input (tokenization included, model loading excluded): 2.47 ms median, 2.62 ms p95. Head-only measured 2.40 ms. This was measured with seed 0, which is architecturally identical, on the training server, not on a laptop. See results/inference-comparison.json and results/cpu-latency.json.

Verification

checks/packaged-inference-verification.json re-runs this repository's model.py on CPU for all three seeds. It checks that:

• the bundled backbone's tensor hash equals the frozen training backbone;
• every test label matches the recorded GPU predictions, with probabilities within 5.4e-6;
• validation accuracy and NLL match the training history.

Verify file integrity with sha256sum -c SHA256SUMS.

Repository contents

Path	What it is
model.py	Portable inference: CloverIntent class and CLI
adapter.safetensors	Released weights (seed 2)
checkpoints/seed-{0,1,2}.safetensors	All three trained seeds
base_model/	Unmodified all-MiniLM-L6-v2
model_config.json	Labels, tokens, hashes, parameter counts
results/	Test report, per-example predictions and uncertainty, per-seed histories, baselines, latency
data/	Exact train/validation/test splits, dataset card, manifest, audit
checks/	Packaged-inference verification
provenance/original/	Original experiment protocol, reports and source, under internal project names

Files copied verbatim from the experiment (under results/, data/ and provenance/) use the internal method name ariadne, which is the Akiki input affine described here.

Licence and credits

Apache-2.0 (see LICENSE and NOTICE). The base model, sentence-transformers/all-MiniLM-L6-v2, is Apache-2.0; its original card is in base_model/README.md.