File size: 1,927 Bytes
86c8491 dacc61c 4a70a2d 405679d dacc61c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
---
language: en
license: mit
tags:
- vision
- text-generation
- medical
- chest-xray
- healthcare
- multimodal
pipeline_tag: image-to-text
---
# 🩺 ChestX – Chest X-ray Report Generation (ViT-GPT2)
This model generates **medical diagnostic reports from chest X-ray images**.
It was developed for the **TWESD Healthcare AI Competition 2024** as part of my final-year engineering project.
The architecture combines a **Vision Transformer (ViT)** for image encoding with **GPT-2** as the language decoder, forming an **encoder–decoder multimodal model**.
---
## 📌 Model Description
- **Architecture:** VisionEncoderDecoderModel (ViT + GPT-2)
- **Input:** Chest X-ray image
- **Output:** Text report describing findings
- **Framework:** PyTorch + Hugging Face Transformers
---
## 💡 Intended Uses & Limitations
✅ Intended for:
- Research in **medical AI & multimodal learning**
- Exploring **vision-to-text generation**
- Educational and prototyping purposes
⚠️ Limitations:
- Not intended for **real clinical diagnosis**
- Trained on a limited dataset (IU Chest X-ray), may not generalize to all populations
---
## 🛠️ How to Use
```python
from transformers import VisionEncoderDecoderModel, AutoTokenizer, AutoFeatureExtractor
from PIL import Image
import torch
# Load model and tokenizer
model = VisionEncoderDecoderModel.from_pretrained("Molkaatb/ChestX").to("cuda")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token
feature_extractor = AutoFeatureExtractor.from_pretrained("google/vit-base-patch16-224-in21k")
# Example image
image = Image.open("example_xray.png").convert("RGB")
inputs = feature_extractor(images=image, return_tensors="pt").pixel_values.to("cuda")
# Generate report
outputs = model.generate(inputs, max_length=512, num_beams=4)
report = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(report)
|