# Gemma classification family: design and reproducibility

## Goal and selected backbone

Three task-specific, text-only multi-label classifiers, preserving Open Emotions, Open Morals and Open Sensations labels and JSON outputs. They are a separate model family; existing RoBERTa releases remain available.

Selected latest small core checkpoint: [google/gemma-4-E2B](https://huggingface.co/google/gemma-4-E2B), Apache 2.0, pinned revision `d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f`. Google describes E2B as 2.3B effective parameters and approximately 5.1B including embeddings and multimodal components. E2B is not a literal total-parameter count. We load only `model.language_model.*` tensors into `Gemma4TextModel` and check every text weight against the architecture. Actual classifier parameter counts are measured in each run manifest.

The downloaded source checkpoint is approximately 10.25 GB. Each released task is a **LoRA adapter plus classifier head**, requiring the pinned base to run. These small task downloads are not standalone full base weights. The loader instantiates no vision/audio encoders, language-generation head, tool interface or generation method. A classifier cannot emit prose or execute tools; the application constructs JSON from its scores.

Source: [official model card](https://ai.google.dev/gemma/docs/core/model_card_4). No instruction/chat template is used: inputs preserve the original tasks' plain-text format. Sensations uses `Target: WORD\nContext: SENTENCE`.

## Data and fairness

Emotions uses exactly the original official GoEmotions simplified splits at revision `add492243ff905527e67aeb8b80c082af02207c3`: 43,410 train, 5,426 validation, 5,427 test; 28 labels and hard multi-hot targets. Its synthetic arm reuses 300 assistant-authored variants from 60 seeds, covering six rare emotions; exact normalized held-out matches were checked before training.

Morals and sensations reuse the existing files and hashes. MFRC soft annotator-vote targets and held-out comment groups remain unchanged. Sensory strength targets, project-defined categorical evaluation proxy, held-out word groups and audited 177 synthetic examples remain unchanged. The sensory model is still a target-in-context association prototype, not a sentence-level sensation-presence classifier. Better pretrained representations do not repair limited annotations.

Same examples, labels and references permit a practical comparison with RoBERTa. Different tokenizers, adaptation budgets, model sizes, precision and epoch counts mean this is not an architecture-only controlled experiment. Token limits are 128 for emotions/sensations and 256 for morals; Gemma's advertised native 128K capacity is not our evaluated classification limit.

## Training protocol fixed before fitting

- Frozen BF16 text backbone, LoRA rank 8 on q_proj/v_proj, alpha 16, dropout 0.05. Trainable adapters and classifier head are FP32.
- Last non-padding token pooling, linear multi-label head, sigmoid scores; soft-target binary cross-entropy.
- GPU microbatch 8, accumulation 4, effective batch 32; AdamW weight decay 0.01; 6% linear warmup/decay; gradient clipping 1.0; eager attention.
- Human baseline: one complete epoch for emotions and morals; ten epochs for the small sensory corpus, retaining the best validation checkpoint. LoRA learning rate 1e-4; head 1e-3; seed 20261004.
- Two one-epoch continuations start from the same selected baseline. Both replay the same deterministic sample of up to 6,000 human training examples; the synthetic arm additionally includes that task's generated data. LoRA/head continuation rates: 5e-5/5e-4. Synthetic compute is slightly greater.
- Global threshold search 0.10..0.80 in steps of 0.05; validation micro F1 for emotions, macro F1 for morals/sensations, matching earlier threshold objectives.
- Synthetic adoption requires validation macro F1 at least 0.005 above both baseline and control, with no more than 0.01 micro F1 loss against either. Otherwise retain baseline. Release decision freezes before test evaluation.
- Every arm is evaluated on the same held-out test split after selection. Test scores do not select the published arm. Results include per-label metrics, all arms, runtime and measured PyTorch peak allocated/reserved GPU memory (not total process/system memory).

No guarantee of better classification is made before evaluation. A trainable classification head uses Gemma's learned representations; it does not invoke chain-of-thought reasoning.

## Hardware and isolated runtime

Local ASUS ROG Flow Z13 2025 GZ302EA; AMD Ryzen AI Max+395, Radeon8060S, 128GB unified RAM. Training runs under Ubuntu24.04 WSL2 with ROCm PyTorch 2.13.0+rocm10.0.0. Dedicated environment `/opt/open-gemma-venv` uses Transformers5.18.0, PEFT0.21.2 and Accelerate1.15.0; exact package lock is `requirements-gemma-wsl.lock.txt`. Existing RoBERTa environments are preserved.

The Gemma environment shares the existing ROCm installation through a .pth path. For reproduction, install the hardware-appropriate PyTorch separately, then the pinned model-library versions. Do not install the bundled graphics runtime wheels on unrelated hardware simply because they appear in the machine-specific lock.

Preparation: `prepare_gemma_data.py` under the original dataset-compatible CPU environment. Training: `/opt/open-gemma-venv/bin/python train_gemma.py all`. `test_gemma_contract.py` checks a tiny randomly initialized text model's loss/head/adapter gradients, padding, output shape, and absence of generation/modalities. `gemma_classifier.py` also includes a full checkpoint GPU forward/backward probe.

WSL also needs `build-essential` and `python3.12-dev` for Triton kernel compilation on this runtime. Set `HF_HOME` before importing Hugging Face libraries when using a custom cache; the training and windowing entry points default to this project's `.cache/huggingface` directory. The full checkpoint GPU probe passed after handling the library's documented redundant shared-KV tensors. No active model weight is skipped.

To reconstruct the data from the downloadable project files, use a separate dataset-compatible environment matching `requirements-windows-cpu.lock.txt` (the original environment uses datasets 3.6.0). Its older dataset library is not used inside the new Gemma training runtime. The emotion repository supplies `roberta_reference.json` and `synthetic_data.jsonl` for the preparation fallback. From a fresh working directory with the scripts:

```bash
mkdir -p data
cp synthetic_data.jsonl data/synthetic-emotions-v1.jsonl
python prepare_gemma_data.py
python prepare_tasks.py --task both
python audit_synthetic_overlap.py
```

Use the emotion repository's synthetic file for the first copy. The moral/sensory preparation script rebuilds its own synthetic seeds; the sensory audit excludes held-out target words. Compare split IDs and revisions with the released manifests before training. File hashes can differ across operating systems because of line endings; parsed IDs, targets and texts must match. Then switch to the Gemma runtime and run `train_gemma.py all` in a fresh output directory. The scripts deliberately refuse to overwrite existing experiment directories.

## Inference and API compatibility

The published adapters use a custom loader, not a generic chat pipeline. Download a task repository into a local directory, then:

```python
import sys
sys.path.insert(0, './downloaded-task')
from gemma_classifier import load_classifier
model, tokenizer, options = load_classifier('./downloaded-task', device='cuda')
model.eval()
text = 'Thank you for helping me!'
batch = tokenizer(text, return_tensors='pt', truncation=True, max_length=options['max_length'])
import torch
with torch.inference_mode():
    scores = model(**batch).logits.sigmoid()[0].cpu().tolist()
labels = [model.config.id2label[i] for i, score in enumerate(scores)
          if score >= options['global_threshold']]
```

For existing per-window output standards, `predict_windows.py --task open-emotions --model ./downloaded-task --file document.txt` detects the classifier configuration and invokes the Gemma loader. Change task as appropriate and include `--target` for sensations. It retains the same windows, label scores, spans and exploratory summaries. Per-window max/mean remains uncalibrated.

Run the existing API in the Gemma-capable environment with `serve_models.py --model-dir open-emotions=./downloaded-task` (repeat per task). The same input routes, approval flow, quotas and JSON field conventions apply. API_USAGE.md describes owner approval and rate limits. This does not deploy a public HTTP service merely by uploading files to Hugging Face.

Saved-model verification completed for all three default releases through real HTTP requests and GPU inference. Checks covered multiple sliding windows, exact label sets, finite score ranges, configured window bounds, pending-key denial, approved-key access, quota rejection (HTTP 429 with Retry-After), and revoked-key denial. The published inference_verification.json contains generated example outputs and no client credentials. Additional tests cover exact tokenizer IDs/spans, long sensory documents, head/adapter gradients and atomic quota accounting.

## License and provenance

Base: Apache 2.0. Task adapters and classifier artifacts retain Apache 2.0 attribution. Dataset licenses and source credits remain distinct; raw human texts are not redistributed. Manifests record revisions, source/data hashes, label order, hyperparameters and selection decisions. Synthetic examples have assistant labels and are not independent human annotations. Existing evaluation groups were not mixed into synthetic training.

## Completed held-out results

All three human baselines and both continuation arms per task completed. Default release selection used validation only and was frozen before test evaluation. Micro/macro F1 below are on the same test splits as the original RoBERTa classifiers, with each model's own validation threshold.

| Task | Default Gemma arm | Gemma test micro F1 | Gemma test macro F1 | RoBERTa test micro F1 | RoBERTa test macro F1 |
| --- | --- | ---: | ---: | ---: | ---: |
| Emotions | Human baseline | 0.61428 | 0.52267 | 0.60467 | 0.43528 |
| Morals | Synthetic continuation | 0.70658 | 0.44734 | 0.70728 | 0.49879 |
| Sensations | Synthetic continuation | 0.78212 | 0.75561 | 0.57724 | 0.55302 |

Gemma improved the tested emotion and sensory classification scores. Moral micro F1 was similar and macro F1 was lower; the moral release is an experimental alternative, not a demonstrated upgrade. The sensory test is only 48 contexts from 12 word groups. Different training budgets, tokenizers, sizes and precision limit architecture-only conclusions.

Emotions' synthetic arm did not beat its human-only continuation on validation macro F1, so the predeclared rule retained the baseline. The human-only control is a comparison arm, not a default release candidate under this protocol. Root selection is therefore not a claim that the root always has the highest score among all arms. All human baselines, real-only controls and synthetic candidates are included as separate ZIPs for review.

Completed emotion experiment: default baseline threshold 0.30; 4,629,951,772 total parameters, 1,382,428 trainable; full three-arm experiment 53.67 minutes. All 28 labels had nonzero test F1, but some rare-label supports are very small. See the task's RESULTS.md, run.json and per-label metrics for detail.

Completed sensory experiment: selected synthetic continuation by validation only. Held-out test micro F1 0.78212, macro F1 0.75561, threshold 0.30; original RoBERTa reference 0.57724 / 0.55302. The Gemma human-only baseline scored 0.80882 / 0.71973, so synthetic improves macro F1 but reduces micro F1 relative to that baseline. These results cover just 48 held-out contexts from 12 word groups. Total classifier parameters 4,629,917,958; trainable 1,348,614. Experiment runtime 18.05 minutes, peak PyTorch allocated GPU memory 9.69 GiB. Full per-label/arm results remain in outputs/gemma/open-sensations/run.json.

Completed moral experiment: synthetic continuation passed the validation rule (micro F1 0.69873, macro F1 0.49854) and was selected before test evaluation. Its test micro F1 is 0.70658 and macro F1 0.44734 at threshold 0.35. RoBERTa scored 0.70728 / 0.49879 on the same test split, so the Gemma release did not improve the tested moral classification performance. Synthetic validation gains did not generalize to test macro F1; the human-only continuation scored test macro F1 0.48127. The selection was not changed using these test results. All three arms are downloadable for review. Total classifier parameters 4,629,921,032; trainable 1,351,688. Full experiment 90.61 minutes; peak PyTorch allocated/reserved GPU memory 18.20 / 24.31 GiB. See outputs/gemma/open-morals/run.json for all metrics.

Release repository locations (publication verification is recorded separately after upload):

- https://huggingface.co/mts-the-architect/open-emotions-gemma4-e2b-bf16-lora
- https://huggingface.co/mts-the-architect/open-morals-gemma4-e2b-bf16-lora
- https://huggingface.co/mts-the-architect/open-sensations-gemma4-e2b-bf16-lora