SweMedMCQ-EuroLLM-1.7B-Trace-EN

EuroLLM-1.7B fine-tuned for English and Swedish medical multiple-choice questions, trained on reasoning-trace data in English. It is one of the 14 student models released with the paper "Benchmarking and Adapting LLMs for English–Swedish Medical Question Answering" and the SweMedMCQ benchmark (code and data in the repository linked from the paper).

Training

• Base model: utter-project/EuroLLM-1.7B-Instruct.
• Data: 549,162 examples from 125,678 English training questions, from the training splits of the ten MedicalAgentsBench datasets. No test item was used for training or model selection.
• Supervision: Reasoning-trace supervision: responses of two open teacher models to the training questions, kept only when the answer is correct and the explanation is not empty (length outliers removed). GPT-OSS-120B answered each question three times (greedy and twice at temperature 0.7) and its reasoning is kept as a <think>…</think> block before the JSON answer; Qwen3.6-35B-A3B answered twice (greedy and at temperature 0.7) and only its explanation and answer are kept. A question can therefore have up to five training examples.
• Training language: English.
• Settings: full fine-tuning (embeddings trained), two epochs, effective batch 128 (32 per GPU on 4 GPUs), learning rate 5e-5 with cosine schedule and 5% warm-up, weight decay 0.01, gradient clipping 1.0, bfloat16, padding-free packing, sequences up to 12,288 tokens, loss on the response tokens, seed 13.
• Checkpoint: best validation loss, step 8,400 of 8,580, validation loss 0.6655 (evaluated every 200 steps).

Prompt format

{question} is the question text and {options} the options, one per paragraph, as A) …, B) … (letters in order, separated by a blank line). The prompt is sent as a single user message, with no system prompt. The model answers with {"explanation": "...", "answer_index": "<letter>"}, after a <think>…</think> reasoning block.

English explanation prompt

You are a highly knowledgeable and careful medical expert.

Your task is to analyze the following multiple-choice medical question and determine the single best answer based on established medical knowledge and reasoning.

Follow these steps carefully:
1. Read the question and all available answer options.
2. Identify key clinical findings, pathophysiology, or reasoning relevant to the question.
3. Select the single best answer from the listed options.
4. Provide a concise, evidence-based explanation for your choice.

---

Return your final output **only** in valid JSON format that matches this schema:

{
  "explanation": "A brief medical reasoning explaining why this option is correct and others are not.",
  "answer_index": "The letter corresponding to the correct answer option (e.g., 'A', 'B', 'C', etc.)."
}

---

## Input

**Question:**
{question}

**Options:**

{options}

---

Swedish explanation prompt

Du är en mycket kunnig och noggrann medicinsk expert.

Din uppgift är att analysera följande flervalsfråga och avgöra vilket alternativ som är det enda bästa svaret baserat på etablerad medicinsk kunskap och resonemang.

Följ dessa steg noggrant:
1. Läs frågan och alla tillgängliga svarsalternativ.
2. Identifiera viktiga kliniska fynd, patofysiologi eller relevant medicinskt resonemang.
3. Välj det enda bästa svaret bland de angivna alternativen.
4. Ge en kortfattad, evidensbaserad förklaring till ditt val.

---

Returnera ditt slutliga svar **endast** i giltigt JSON-format som följer detta schema:

{
  "explanation": "En kort medicinsk motivering som förklarar varför detta alternativ är korrekt och varför de andra inte är det.",
  "answer_index": "Bokstaven som motsvarar det korrekta svarsalternativet (t.ex. 'A', 'B', 'C' osv.)."
}

---

## Indata

**Fråga:**
{question}

**Alternativ:**

{options}

---

EuroLLM's chat template does not open the thinking block the model was trained with, so it is appended after the generation prompt: <think>\n for reasoning, <think>\n\n</think>\n\n for a direct answer.

Context length. The config keeps EuroLLM's max_position_embeddings of 4,096. Training and evaluation used longer sequences, so load the model with hf_overrides={"max_position_embeddings": 12288} in vLLM (or set it in the config for transformers).

Usage with vLLM

Greedy decoding, as in the evaluation. The two prompts above are in prompts.json in this repository (sv and en); fill them with str.format. The example is in English, the training language of this model. Swedish options are listed in label order; English options in their original order.

import json
import re

from huggingface_hub import hf_hub_download
from vllm import LLM, SamplingParams
from vllm.sampling_params import StructuredOutputsParams


def answer_letter(text):
    """Answer letter in the JSON after the reasoning, or None if the model wrote none."""
    match = re.search(r'"answer_index"\s*:\s*"\s*([A-Z])', text.split("</think>")[-1])
    return match.group(1) if match else None


MODEL = "Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN"
PROMPTS = json.load(open(hf_hub_download(MODEL, "prompts.json"), encoding="utf-8"))
llm = LLM(MODEL, max_model_len=12288, hf_overrides={"max_position_embeddings": 12288})
tok = llm.get_tokenizer()

question = "What is the band of electromagnetic waves used in a thermographic camera?"
options = {"A": "X-ray", "B": "Microwave", "C": "Ultraviolet", "D": "Infrared"}
opts = "\n\n".join(f"{k}) {v}" for k, v in options.items() if v)
user = PROMPTS["en"].format(question=question, options=opts)
messages = [{"role": "user", "content": user}]
base = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
prompt = base + "<think>\n"                 # reasoning
direct = base + "<think>\n\n</think>\n\n"   # direct answer, then JSON

schema = {"type": "object", "properties": {"explanation": {"type": "string"}, "answer_index": {"type": "string"}}, "required": ["explanation", "answer_index"], "additionalProperties": False}
# Reasoning: greedy decoding; the JSON answer follows "</think>".
text = llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=10240))[0].outputs[0].text
print(answer_letter(text))

# Direct answer: JSON-schema decoding right after the empty thinking block.
params = SamplingParams(temperature=0.0, max_tokens=10240,
                        structured_outputs=StructuredOutputsParams(json=schema))
print(json.loads(llm.generate([direct], params)[0].outputs[0].text)["answer_index"])

del llm  # shut the engine down, so the script exits

In the paper's evaluation the JSON after </think> was also produced with schema-constrained decoding (a reasoning parser that switches the constraint on once the block is closed; see the evaluation code in the repository). With unconstrained decoding as above, the model sometimes writes no valid answer (answer_letter returns None); the paper counts such answers as wrong.

Serving with vLLM

Start an OpenAI-compatible server:

vllm serve Anonymiserad/SweMedMCQ-EuroLLM-1.7B-Trace-EN --max-model-len 12288 --hf-overrides '{"max_position_embeddings": 12288}'

and send the same prompt (here messages as in the example above). The server rejects requests whose prompt + max_tokens exceed the context, so ask for at most 10,240 new tokens:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# The reasoning block is opened in the prompt, so use the completions endpoint with the templated text.
prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) + "<think>\n"
r = client.completions.create(model=MODEL, prompt=prompt, temperature=0.0, max_tokens=10240)
print(answer_letter(r.choices[0].text))

tok is the model's tokenizer, as in the example above (or transformers.AutoTokenizer.from_pretrained(MODEL)).

Results

Accuracy (%) on the 9,733 core items of the SweMedMCQ test set (English source and Swedish benchmark text, greedy decoding):

Decoding	English	Swedish	Gap (EN − SV)
reasoning	43.2	39.4	3.8
direct answer	44.7	39.0	5.8

The base EuroLLM-1.7B reaches 17.5% in English and 23.5% in Swedish with the same prompt.

Limitations

• Research model. It is not a medical device and must not be used for diagnosis, treatment, or any clinical decision.
• Trained and evaluated on exam-style multiple-choice questions only; accuracy is far below that of large models, and explanations can be wrong even when the answer is right.
• Swedish training and test text is machine-translated from English and may contain translation errors.

Licence

Apache 2.0, as the base model.

Citation

@misc{anonymous2026swemedmcq,
  title  = {Benchmarking and Adapting {LLMs} for {English}--{Swedish} Medical Question Answering},
  author = {Anonymous},
  year   = {2026},
  note   = {Under review}
}