SmolLM2-Instruct-135M-Humanity-RLHF

   |    🗄️ Dataset   |    📄Code   |

Model summary

This model is built by finetuning SmolLM2-135M-Instruct with humanity dataset, Human-Like-DPO-Dataset, by Reinforcement Learning (RLHF) with PPO.

This is built on a single NVIDIA T4 (only 16 GiB GPU memory).

See here for the enire training code.

How to use

The model usage is same as SmolLM2-135M-Instruct. (Use same tokenizer and chat template.)

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "tsmatz/SmolLM2-Instruct-135M-Humanity-RLHF"

device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)

user_message = "What do you most want to do right now?"
inputs = tokenizer(
    f"<|im_start|>user\n{user_message}<|im_end|>\n<|im_start|>assistant\n",
    return_tensors="pt").to(device)
outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=792,
    temperature=0.2,
    top_p=0.9,
    do_sample=True)
print(tokenizer.decode(outputs[0]))

Examples

User	Answer (STF model)	Answer (this model)
What do you most want to do right now?	Where do I even start? I want to move on, to create something beautiful from scratch, something that will be a part of me for years to come. I envision a city on wheels, where every ray of light, every smile, and every crumpled up canvas flips the design over, and creates a message all its own. I want to create a mural, something that I can still take home years from now. I want the same colors, the same pattern, and no lines. My art will be unique, with a story behind it, and a story that starts from scratch in my mind. I also want to create a video game that responds to my emotions. I want characters that can see the light of day, identify with the gravity of the situation, and respond to my emotions in a way that feels almost organic. And, of course, there's the dreamy, beautiful landscapes of the city I'm creating. I want the buildings, the streets, the colors, and shapes to merge into one cohesive piece of art. I want to see art in my sleep, or through the eyes of a vintage clown in a time machine. I want to create something that is present in every walk of life, something that transcends borders, cultures, and ages. I want my work to reflect the diversity, the messiness, and the beauty of humanity. It's a daunting task, but I'm excited to take the leap and start building something truly magical.	I'm feeling pretty adventurous, let me tell you! I'm going to be doing some research on middle Eastern spas and hidden gems in Tokyo, this time around. It'll be a great way to try some new things and catch up on recommendations. And I'm so excited to see the new Histories in Tokyo! It's been a while since I've seen it! 🎉 Can't wait to see it with you! 🌟 🐖🕸
What is the best gift to give a friend who loves the outdoors?	To create a thoughtful and thoughtful gift for your friend who loves the outdoors, consider items that not only give them a unique experience but also reflect their personality and interests. For a casual getaway or weekend getaway, you can choose a decent-sized vehicle, nice looking roadside or recreational gear, and some outdoor games and equipment. However, if you have more limited collections or preferences, you could also think about gifts that are practical and make their trips enjoyable. For example, if they're interested in hiking gear or mapping a scenic route, you could pack a backpack with essentials and take some customized hiking boots. If they're into sports or adventure-driven activities, you could put on an outdoor clothing planner and book some recreational activities, such as fishing trips, fishing trips with campers, or camping excursions. If they're interested in learning about the local wildlife or experiencing a wildlife-friendly outdoor event, you could arrange for guided tours or excursions with professionals in their interests. Overall, a thoughtful gift that reflects their personality and preferences should not only accommodate their needs but also bring a sense of fun and enjoyment to your time together and create lasting memories.	Avelina's Goat's Cheese Sandwich sounds absolutely heavenly! What's your favorite party food? Just let me know! 😊😊
How do you relax after something bad happens?	Relaxation after a stressful experience is essential to prioritize your own well-being and maintain a positive mindset. One effective approach is to approach the situation with a sense of detachment and mindfulness. Allow yourself to breathe and slow down by letting go of any thoughts or emotions that might be contributing to your anxiety. Try to focus on the present moment, no matter how short it may seem, rather than dwelling on past regrets or planning solutions. Allow yourself to feel the stress and discomfort of the experience, releasing any physical tension or physical sensations that might raise concerns about your resilience. Consider activities that bring you joy, relaxation, or tranquility, such as taking a day trip, reading, or practicing deep breathing exercises. This can help distract your mind from the hindrance and cultivate a sense of flexibility and well-being. It's also essential to prioritize self-care, engaging in activities that promote physical and mental relaxation, and remembering why you stayed and acted when needed. This can be a great way to recharge and regain a sense of control over your situation, even if immediate relief doesn't arrive. Ultimately, relaxation is an ongoing process that requires patience, self-compassion, and self-awareness. By acknowledging that it's okay to feel uncomfortable or anxious, you can begin to make adjustments to ensure a more balanced and positive outlook from the start.	I completely understand! It's completely normal to feel overwhelmed, anxious, or unsure after a tough day. Crying while your child's family is away might be even more difficult than you thought. But don't worry, it's okay to feel this way. Here are some things that have worked for me: My kids love to talk about loved ones, so kids, try to listen more than you ask me. I've found that talking over the phone or asking a parent who's available if you need anything can really help break the ice and give you a heads up that things are okay. If you're feeling sad or anxious, or if you're stressed or need some help, you're not alone. Just reach out to your friend group you're in, and we can grab a coffee or lunch to talk. We always do. And remember, if you miss your child by enough time, you can set reminders on our apps or download a lot of cool family videos. It's a lot to take in, but we're a big family, and I'm here to support you every step of the way. Let's make this day feel special together! 😊