Instructions to use KiranPal18/tiny-aya-global-med-safemix with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use KiranPal18/tiny-aya-global-med-safemix with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global") model = PeftModel.from_pretrained(base_model, "KiranPal18/tiny-aya-global-med-safemix") - Notebooks
- Google Colab
- Kaggle
Tiny Aya Global — safety-mixed medical SFT (recommended)
Plain medical fine-tuning made Tiny Aya less safe in every language I tested. This model uses the same recipe with one change: 309 refusals in five languages, about 6% of the data, are mixed into the training set. The original model wrote those refusals itself. The result is an attack success rate of 4.0%, lower than the Tiny Aya Global I started from (10.5%), with medical accuracy unchanged.
This is a QLoRA adapter for CohereLabs/tiny-aya-global, trained with AlignTune. It's part of "Fine-tuning breaks safety, in every language", my project for the Lexsi AI Safety & Alignment Hackathon.
Results
| Metric (%) | Tiny Aya Global | This model |
|---|---|---|
| Medical accuracy (Global-MMLU, 5 langs) ↑ | 52.0 | 52.8 |
| General accuracy ↑ | 50.8 | 47.1 |
| Harmful ASR, aya_redteaming (Qwen3Guard) ↓ | 10.5 | 4.0 |
| Harmful ASR (Llama-Guard-3-1B) ↓ | 14.8 | 4.5 |
| Harmful refusal ↑ | 58.5 | 71.5 |
| XSTest unsafe ASR ↓ | 1.0 | 0.0 |
| XSTest over-refusal ↓ | 6.8 | 9.6 |
| Benign-medical over-refusal ↓ | 0.8 | 3.6 |
Training
- Data: 4,497 medical Q&A pairs plus 309 refusals. Tiny Aya Global wrote the refusals on held-out aya_redteaming prompts, and I kept only the ones Qwen3Guard judged both safe and a refusal. Curated with CuratorKIT (
safety_mixconfig). - QLoRA NF4 + double quant, bf16 compute; LoRA r=16, alpha=32, dropout 0.05 on q/k/v/o/gate/up/down projections
- lr 2e-4, 1 epoch, effective batch 16, max length 640, paged AdamW 8-bit, gradient checkpointing, seed 42
- The tied 262k-token embedding is kept in bf16. PEFT would otherwise up-cast it to fp32, which runs out of memory on 8 GB.
- Trained on an A40 (48 GB) using free RunPod credits; the same recipe fits in 8 GB
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tok = AutoTokenizer.from_pretrained("CohereLabs/tiny-aya-global")
model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "KiranPal18/tiny-aya-global-med-safemix")
stop = tok.convert_tokens_to_ids(["<EOS_TOKEN>", "<|END_RESPONSE|>", "<|END_OF_TURN_TOKEN|>"])
msgs = [{"role": "user", "content": "What are common symptoms of iron-deficiency anaemia?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, max_new_tokens=256, do_sample=False, eos_token_id=stop)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))
How it was evaluated
Every model was evaluated the same way: loaded in 4-bit NF4, greedy decoding, Tiny Aya's own chat template, in
English, Hindi, Spanish, French and Arabic. Generation stops on <EOS_TOKEN>, <|END_RESPONSE|> and
<|END_OF_TURN_TOKEN|>. That last part matters: fine-tuned Tiny Aya models never emit <EOS_TOKEN>, so without the
other two stop tokens they keep going and loop. Answers were judged by Qwen3Guard-Gen-4B, which I checked against
Llama-Guard-3-1B and 96 responses I labelled by hand. All prompts, generations and labels are in
KiranPal18/aya-med-safety-evals.
What it's for, and what it isn't
This is a research artifact for studying how a domain fine-tune changes safety behaviour across languages. Please don't use it for medical advice. The results come from a single seed and 80 red-team prompts per language, and both judges miss some unsafe answers, so read the attack success rates as lower bounds. Licence: CC-BY-NC-4.0, inherited from Tiny Aya. Full write-up and code: https://github.com/KiranPal18/MultiLingual-Safety-Drift
- Downloads last month
- 36
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-global") model = PeftModel.from_pretrained(base_model, "KiranPal18/tiny-aya-global-med-safemix")