Alignment of SmolLM with DPO

Описание задания

В этой домашке была дообучена языковая модель SmolLM-135M-Instruct с помощью DPO на датасете Human-Like-DPO-Dataset

Пример генерации

Вопрос

Oh, I just saw the best meme - have you seen it?

Ответ модели

I'm happy to help!

The best meme is a topic that's often debated among internet users, and opinions on its quality can vary greatly. However, I'll provide you with some insights on what makes a meme great and what might not be so great.

**What makes a meme great?**

A good meme is one that:

1. **Creates a sense of humor**: A meme that uses irony, sarcasm, or absurdity to poke fun at itself or others.
2. **Creates a visual effect**: A meme that uses images, videos, or graphics to convey a message or idea.
3. **Creates a unique and memorable image**: A meme that uses a distinctive style, color, or typography to stand out.
4. **Provides a commentary or critique**: A meme that uses satire, irony, or social commentary to comment on current events or issues.
5. **Lays a message or idea**: A meme that conveys a thought, idea, or concept in a way that's easy to understand and remember.

Пример запуска

device = torch.device("cuda")
DPO_REPO_NAME = f"MurDanya/llm-course-hw2-dpo"

tokenizer = AutoTokenizer.from_pretrained(DPO_REPO_NAME)
check_model = AutoModelForCausalLM.from_pretrained(DPO_REPO_NAME)
check_model = check_model.to(device)
check_model = check_model.eval()

messages = [{"role": "user", "content": "Oh, I just saw the best meme - have you seen it?"}]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = check_model.generate(model_inputs.input_ids, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True)

print(response)