---
language:
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-classification
base_model: sentence-transformers/all-MiniLM-L6-v2
base_model_relation: adapter
tags:
- akiki
- clover
- minilm
- intent-classification
- frozen-encoder
- custom-code
metrics:
- accuracy
- f1
---

# Akiki-MiniLM-L6-v2-clover-intent

Developed by AKIKI.AI.

An English six-way intent router for Clover, built on a **completely frozen**
`sentence-transformers/all-MiniLM-L6-v2`. Only two small pieces were trained: an
identity-initialized **384 × 384 Akiki input affine** on the word embeddings
(147,840 parameters) and a **six-class head** (2,310 parameters), 150,150 in total.
The 22,713,216 backbone parameters are untouched and hash-checked against the
original.

Given the current user message and, optionally, earlier conversation turns, it
returns one of:

| Token | Label | Meaning |
|---|---|---|
| A | chat | Conversation, explanation, how-to advice, general knowledge |
| B | question | Inspect or report the state of the user's actual system, without changing it |
| C | build | Create or modify software, a feature, a site or an app |
| D | task | Perform a specified operational action (restart, move, change a setting) |
| E | fix | Diagnose and remedy a malfunction when no specific remedy is prescribed |
| F | unclear | The route or referent cannot be identified from message and context |

> **Experimental.** Labels are synthetic and model-reviewed; human review is
> pending. This model classifies intent only. It does not write contracts, run
> actions, or implement Clover's refusal policy. It has not been approved for
> serving.

## Results

Held-out test split: 714 examples (120 per class; 114 unclear), balanced across
easy / medium / hard.

| Method (MiniLM-L6-v2, frozen backbone) | Test accuracy | Macro F1 | Hard accuracy |
|---|---:|---:|---:|
| Label-description similarity, no training | 34.9% | 0.313 | 36.7% |
| Frozen embeddings + regularized linear classifier | 92.3% | 0.923 | 91.7% |
| Head-only training, 3-seed mean | 91.7% ± 0.3 | 0.917 | 91.1% |
| Akiki affine + head, 3-seed mean | 96.6% ± 0.6 | 0.966 | 95.6% |
| **This checkpoint (Akiki, seed 2)** | **97.2%** | **0.972** | **96.2%** |

**Which seed, and why:** three seeds were trained. Seed 2 is released because it
has the lowest validation NLL (0.121 vs 0.131 and 0.141), which is the
experiment's checkpoint criterion. Test results played no part in the choice.
Seed 2 happens to be the best on test as well, so treat the **three-seed mean
(96.6%)** as the honest estimate of what this recipe gives. All three seeds ship
in `checkpoints/`.

Seed 2 detail: easy 99.6%, medium 95.8%, hard 96.2%; test NLL 0.082, Brier 0.043.

| Class | Precision | Recall | F1 |
|---|---:|---:|---:|
| chat | 0.991 | 0.958 | 0.975 |
| question | 0.967 | 0.992 | 0.979 |
| build | 1.000 | 0.967 | 0.983 |
| task | 0.934 | 0.950 | 0.942 |
| fix | 0.967 | 0.992 | 0.979 |
| unclear | 0.974 | 0.974 | 0.974 |

Rows are gold labels, columns are predictions:

| gold \ pred | chat | question | build | task | fix | unclear |
|---|---:|---:|---:|---:|---:|---:|
| chat | 115 | 2 | 0 | 1 | 0 | 2 |
| question | 1 | 119 | 0 | 0 | 0 | 0 |
| build | 0 | 0 | 116 | 3 | 0 | 1 |
| task | 0 | 2 | 0 | 114 | 4 | 0 |
| fix | 0 | 0 | 0 | 1 | 119 | 0 |
| unclear | 0 | 0 | 0 | 3 | 0 | 111 |

**Pass mark not met.** The experiment requires at least 95% accuracy **and zero
action predictions (build/task/fix) on chat/question**. This checkpoint makes
**1 action prediction on 240 chat/question examples** (a chat message read as
task) and 3 on 114 unclear examples. Downstream code should therefore not treat
a `task`/`build`/`fix` label as authorization by itself.

## Usage

```bash
pip install -r requirements.txt
python model.py 'Restart my web server.'                  # -> D
python model.py --output label 'Restart my web server.'   # -> task
python model.py --output json --context-file ctx.json 'yes'
```

```python
from model import CloverIntent

clf = CloverIntent.from_pretrained(".")          # loads base_model/ + adapter.safetensors, CPU by default
clf.predict("Restart my web server.")["label"]   # 'task'
clf.predict("yes", context=[
    {"role": "assistant", "text": "Would you like me to check whether last night's backup ran?"},
])["label"]                                       # 'question'
clf.probabilities([{"message": "fix it"}])       # (1, 6) tensor, label order above
```

`predict` returns `label`, `token` (A–F) and the six `probabilities`. Everything
loads from this repository with `local_files_only=True`; nothing is downloaded at
run time. Download the whole repo first, e.g.
`hf download Akiki-AI/Akiki-MiniLM-L6-v2-clover-intent --local-dir clover-intent`.

### Input format

The model sees exactly this text (the context block is omitted when empty):

```
Conversation context:
assistant: Would you like me to restart the web server?

Current user message: yes
```

Context turns are `{"role": ..., "text": ...}` objects. The limit is 512 tokens
including special tokens. Longer inputs **raise an error rather than being
truncated**. The longest training/test input is 126 tokens, so these results say
nothing about long briefings or full chat histories.

Probabilities are a plain softmax of the head logits, with no calibration fitted.

## Architecture

```
tokens → MiniLM word embeddings (frozen)
       → Akiki affine  W·x + b   (384×384, identity-initialized, FP32)
       → + position/type embeddings → 6 BERT layers (frozen)
       → attention-masked mean pooling → L2 normalize
       → linear head (384 → 6) → softmax
```

`adapter.safetensors` holds `interface.proj.{weight,bias}` and `head.{weight,bias}`.
`base_model/` is an unmodified copy of all-MiniLM-L6-v2 at revision
`1110a243fdf4706b3f48f1d95db1a4f5529b4d41`.

## Training

- **Data:** 9,913 train / 707 validation / 714 test synthetic English examples
  (`data/`, documented in `data/README.md`). Scenario groups never cross splits;
  near-duplicates across splits were removed, protecting test first. Hard
  examples need the conversation context to resolve the intent, using follow-up
  phrase families kept separate between splits.
- **Labels:** generated locally by Qwen3.8-27B-FP8 and independently classified by
  Qwen3.5-9B; disagreements were rejected. Not human-verified.
- **Optimization:** Adam; head lr 1e-3, affine lr 1e-4; early stopping on
  validation NLL. Seed 2 selected epoch 8 of 11. Each fit took about 30 s on one
  RTX PRO 6000. The full recipe is in `provenance/original/protocol/plan.json`.

## Latency

Warm CPU inference with FP32 PyTorch/Transformers, 8 threads, batch size 1, on a
short contextual input (tokenization included, model loading excluded): **2.47 ms
median, 2.62 ms p95**. Head-only measured 2.40 ms. This was measured with seed 0,
which is architecturally identical, on the training server, not on a laptop. See
`results/inference-comparison.json` and `results/cpu-latency.json`.

## Verification

`checks/packaged-inference-verification.json` re-runs this repository's
`model.py` on CPU for all three seeds. It checks that:

- the bundled backbone's tensor hash equals the frozen training backbone;
- every test label matches the recorded GPU predictions, with probabilities
  within 5.4e-6;
- validation accuracy and NLL match the training history.

Verify file integrity with `sha256sum -c SHA256SUMS`.

## Repository contents

| Path | What it is |
|---|---|
| `model.py` | Portable inference: `CloverIntent` class and CLI |
| `adapter.safetensors` | Released weights (seed 2) |
| `checkpoints/seed-{0,1,2}.safetensors` | All three trained seeds |
| `base_model/` | Unmodified all-MiniLM-L6-v2 |
| `model_config.json` | Labels, tokens, hashes, parameter counts |
| `results/` | Test report, per-example predictions and uncertainty, per-seed histories, baselines, latency |
| `data/` | Exact train/validation/test splits, dataset card, manifest, audit |
| `checks/` | Packaged-inference verification |
| `provenance/original/` | Original experiment protocol, reports and source, under internal project names |

Files copied verbatim from the experiment (under `results/`, `data/` and
`provenance/`) use the internal method name `ariadne`, which is the Akiki
input affine described here.

## Licence and credits

Apache-2.0 (see `LICENSE` and `NOTICE`). The base model,
[sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2),
is Apache-2.0; its original card is in `base_model/README.md`.