README.md

---
language:
- sv
- da
- no
tags:
- medical
- scandinavian
- nordic
- healthcare
- clinical
- gpt-oss
- unsloth
base_model: unsloth/gpt-oss-20b
datasets:
- NbAiLab/NCC
license: apache-2.0
pipeline_tag: text-generation
library_name: transformers
---

# Scandinavian Medical GPT-OSS 20B

A specialized fine-tuned version of OpenAI's GPT-OSS 20B model optimized for natural, fluent medical writing in Swedish, Danish, and Norwegian. This model addresses the need for AI systems that can generate natural sounding Scandinavian medical journal-style language with proper terminology and natural flow.

## **Model Purpose**

This fine-tuned model is specifically designed for healthcare professionals and researchers who need:
- Natural medical report writing in Scandinavian languages
- Authentic medical terminology usage across Swedish, Danish, and Norwegian
- Fluent medical documentation that reads like native medical journal content
- Cross-linguistic medical knowledge transfer between Scandinavian languages

## **Fine-tuning Strategy**

### Training Approach
- **Base Model**: OpenAI GPT-OSS 20B (`unsloth/gpt-oss-20b`)
- **Method**: LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning
- **Focus**: Medical language fluency and natural writing style in Scandinavian languages
- **Training Paradigm**: Supervised Fine-Tuning (SFT) with medical conversation format

### Architecture Modifications
- **LoRA Rank (r)**: 16 - Enhanced for medical domain adaptation
- **LoRA Alpha**: 32 - Increased for stronger medical language learning
- **Target Modules**: `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]`
- **Quantization**: 4-bit quantization for memory efficiency
- **Trainable Parameters**: ~8M out of 20.9B total parameters (0.04%)

##  **Training Data**

### Dataset composition (1,611 total samples):

1. **Norwegian Colossal Corpus (NCC)** - 111 samples
   - Real-world Norwegian text filtered for medical/formal content
   - Natural language patterns from authentic Norwegian sources
   - Focus on formal, academic, and medical terminology

2. **Synthetic Medical Conversations** - 1,500 samples
   - **Languages**: Swedish, Danish, Norwegian (500 samples each)
   - **Medical Conditions**: Myocardial infarction, post-operative care, diabetes, hypertension, asthma
   - **Content Types**: Medical assessments, patient reports, treatment recommendations
   - **Format**: Structured as doctor-patient conversations with proper system prompts

### Language-Specific System Prompts:
- **Swedish**: "Du är en erfaren läkare som skriver medicinska rapporter och journaler på svenska. Skriv naturligt och flytande med korrekt medicinsk terminologi."
- **Danish**: "Du er en erfaren læge, der skriver medicinske rapporter og journaler på dansk. Skriv naturligt og flydende med korrekt medicinsk terminologi."
- **Norwegian**: "Du er en erfaren lege som skriver medisinske rapporter og journaler på norsk. Skriv naturlig og flytande med korrekt medisinsk terminologi."

## **Training Parameters**

```yaml
Training Configuration:
  Epochs: 2
  Total Steps: 202
  Batch Size: 2 per device
  Gradient Accumulation: 8 steps
  Effective Batch Size: 16
  Learning Rate: 1e-4
  Scheduler: Cosine
  Warmup Steps: 50
  Optimizer: AdamW 8-bit
  Weight Decay: 0.01
  Max Sequence Length: 2048 tokens
  
LoRA Configuration:
  Rank: 16
  Alpha: 32
  Dropout: 0.0
  Bias: none
  Use RSLoRA: false
```

## **Installation & Setup**

### Prerequisites
```bash
pip install --upgrade uv
uv pip install \
    "torch>=2.8.0" "triton>=3.4.0" numpy torchvision bitsandbytes "transformers>=4.55.3" \
    "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \
    "unsloth[base] @ git+https://github.com/unslothai/unsloth"
```

### Hardware Requirements
- **GPU**: NVIDIA GPU with at least 16GB VRAM (tested on RTX A6000)
- **RAM**: 32GB+ recommended
- **Storage**: 50GB+ for model and dependencies

## **Loading the Model**

```python
from unsloth import FastLanguageModel
import torch

# Load the fine-tuned model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="VaibhavSxn/scandinavian-medical-gpt-oss-20b",  # Path to your saved model
    max_seq_length=2048,
    dtype=None,  # Auto-detection
    load_in_4bit=True,  # Memory efficient loading
)

print("Scandinavian Medical GPT-OSS model loaded successfully!")
```

## **Inference Examples**

### Swedish Medical Assessment
```python
from transformers import TextStreamer

messages = [
    {
        "role": "system", 
        "content": "Du är en erfaren läkare som skriver medicinska rapporter och journaler på svenska. Skriv naturligt och flytande med korrekt medicinsk terminologi."
    },
    {
        "role": "user", 
        "content": "Skriv en kort medicinsk bedömning för en patient med diabetes typ 2 som behöver justera sin behandling."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
    reasoning_effort="medium",  # GPT-OSS reasoning control
).to(model.device)

# Generate response
_ = model.generate(
    **inputs, 
    max_new_tokens=300, 
    streamer=TextStreamer(tokenizer),
    temperature=0.7,
    do_sample=True
)
```

### Norwegian Medical Report
```python
messages = [
    {
        "role": "system", 
        "content": "Du er en erfaren lege som skriver medisinske rapporter og journaler på norsk. Skriv naturlig og flytande med korrekt medicinsk terminologi."
    },
    {
        "role": "user", 
        "content": "Skriv en postoperativ rapport for en pasient som har gjennomgått laparoskopisk kolecystektomi."
    }
]

# Same inference pattern as above
```

### Danish Treatment Recommendation
```python
messages = [
    {
        "role": "system", 
        "content": "Du er en erfaren læge, der skriver medicinske rapporter og journaler på dansk. Skriv naturligt og flydende med korrekt medicinsk terminologi."
    },
    {
        "role": "user", 
        "content": "Skriv en behandlingsanbefaling for en patient med hypertension, der ikke responderer på nuværende terapi."
    }
]

# Same inference pattern as above
```

## **GPT-OSS Reasoning Effort Control**

One unique feature of GPT-OSS is adjustable reasoning effort:

```python
# Available reasoning levels:
reasoning_effort_options = ["low", "medium", "high"]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
    reasoning_effort="high",  # For complex medical reasoning
).to(model.device)
```

- **Low**: Fast responses, minimal reasoning
- **Medium**: Balanced performance and speed (recommended for most medical tasks)
- **High**: Maximum reasoning capability for complex cases

## **Model Performance**

### Training Metrics
- **Peak GPU Memory**: ~47.5GB total, ~19.4GB reserved
- **Training Time**: ~10-15 minutes on RTX A6000
- **Memory Efficiency**: 30% VRAM reduction with Unsloth optimizations
- **Parameter Efficiency**: Only 0.04% of parameters trained via LoRA

## **Be wary of the following**

- This model is for **research and educational purposes** only
- **Not intended for direct clinical use** without human oversight
- All medical content generated should be **reviewed by qualified healthcare professionals**
- The model may generate plausible-sounding but incorrect medical information
- Always verify medical facts and recommendations through proper medical sources

## **Citation**

If you use this model in your research, please cite:

```bibtex
@misc{scandinavian-medical-gpt-oss-2025,
  title={Scandinavian Medical GPT-OSS: A Fine-tuned Model for Natural Medical Language Generation},
  author={Vaibhav Saxena},
  year={2025},
  howpublished={Fine-tuned from OpenAI GPT-OSS 20B using Unsloth},
}
```

## **Related Resources**

- **Base Model**: [OpenAI GPT-OSS 20B](https://huggingface.co/unsloth/gpt-oss-20b)
- **Training Framework**: [Unsloth](https://github.com/unslothai/unsloth)
- **Norwegian Corpus**: [NbAiLab/NCC](https://huggingface.co/datasets/NbAiLab/NCC)
- **Medical Terminology**: Scandinavian medical dictionaries and guidelines

## **Contributing**

Contributions to improve the model are welcome! Please consider:
- Adding more diverse medical training data
- Expanding to additional Scandinavian languages (Icelandic, Faroese)
- Improving evaluation metrics for medical accuracy
- Creating domain-specific benchmarks

## **License**

Apache 2.0, go bonkers!

---