pooyanayebi/akinator-qwen3.5-35b-a3b-lora

Akinator RL — Qwen3.5-35B-A3B LoRA (25 Questions player)

A LoRA adapter for Qwen/Qwen3.5-35B-A3B, trained with reinforcement learning to play an Akinator-style guessing game: the model asks yes/no questions to identify a secret character (a celebrity, historical figure, or fictional character) and then commits to a guess, all within 25 questions.

Training ran on Tinker with the tinker-cookbook RL loop.

Results

Win rate on 30 held-out characters that were never seen in training, with 4 games per character (120 games). A win means the model named the correct character within 25 questions.

Checkpoint	Held-out win rate
Base Qwen3.5-35B-A3B (before any RL)	~20%
Start of final run (step 0)	23.3%
Step 30	50.0%
Step 50	53.3%
Step 80	60.8%
Step 90 (this adapter)	59.2%

The adapter roughly triples the base model's win rate on unseen characters. Average game length on the test set fell from about 23.9 to 21.3 turns. The test set is small (30 characters), so individual evaluations are noisy at about ±5–10 points. The overall upward trend held across every evaluation.

Game setup

• Player (this model): Qwen3.5-35B-A3B with thinking enabled (qwen3_5 renderer), up to 1,350 tokens per turn.
• Answerer: Qwen3.5-397B-A17B. It knows the secret character and replies with yes, no or idk.
• Characters: about 2,400 characters across ~20 categories (actors, musicians, athletes, politicians, scientists, royalty, YouTubers, and characters from movies, TV, books, games, anime, comics and mythology). Split 80/20 into train and test, stratified by category.
• Reward: 1.0 for a correct guess within 25 questions, otherwise 0.

Training details

Hyperparameter	Value
Method	GRPO-style policy gradient (importance-sampling loss)
LoRA rank / alpha	32 / 32
Target modules	all linear layers (attention, linear-attention, MoE experts, lm_head)
Learning rate	2e-5
Batch size × group size	8 characters × 8 rollouts
KL penalty (vs. reference)	0.01
Temperature	1.0
Max questions per game	25
Total RL steps (final run)	90
Remove constant-reward groups	yes

Lessons from earlier runs

• Avoid per-question shaping rewards. An earlier run gave a small reward for each question asked. The policy learned never to guess and simply collected the guaranteed question reward. A binary win/loss reward fixed this.
• Keep the KL penalty. Running with no KL penalty let the policy drift without limit and erode the base model's knowledge of characters. A small KL penalty (0.01) kept the policy stable.
• Watch for format exploits. One run learned to deliberately break the output format on turn 1 to trigger a corrective feedback message, then used that message as free coaching.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3.5-35B-A3B"
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "pooyanayebi/akinator-qwen3.5-35b-a3b-lora")

SYSTEM = """You are playing a game of 25 questions. You must identify a character — this could be a real person (celebrity, historical figure, athlete, musician, politician, scientist, etc.) or a fictional character (from movies, TV shows, books, video games, anime, comics, mythology, etc.).

Ask strategic yes/no questions to narrow down the identity, then make your guess.
You have at most 25 questions. Use them wisely — try to narrow the space efficiently.
Each message should contain exactly one question or one guess. Keep questions concise (under 25 words).

GUESSING FORMAT: Only use "Guess: <name>" when you are committing to a specific character name as your answer. Do NOT use "Guess:" for yes/no questions. Questions should never start with "Guess:".
Examples:
  - Question: Is this character from a movie? (correct — a yes/no question)
  - Guess: Harry Potter (correct — committing to a specific name)
  - Guess: Is the character Harry Potter? (WRONG — do not use Guess: for questions)

IMPORTANT: Keep your thinking VERY short — at most 2-3 brief bullet points. Do NOT restate the rules, recap the full conversation, or narrate what happened. Just note what you know and ask your next question immediately."""

messages = [{"role": "system", "content": SYSTEM}]
# Loop: generate a question, append it as an "assistant" turn,
# then append your answer ("yes" / "no" / "idk") as a "user" turn.
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=1350, temperature=1.0, do_sample=True)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

The model reasons briefly inside its thinking block, then ends each turn with one yes/no question or a line Guess: <name>.

Limitations

• The model was evaluated on only 30 held-out characters, so treat the win rates as approximate.
• During training, the answerer was a strong LLM. Human answerers can be noisier or inconsistent, which may lower the win rate.
• The model is limited by the base model's knowledge, so obscure or very recent characters are harder to guess.
• The adapter covers MoE expert layers, which makes the file large (~10 GB).