Questions and suggestions: vision support, coding, and strict format adherence
Hi, first of all, what a great model. It's the best MoE model I've tried for its size, thank you very much for it.
I have two questions and a suggestion:
- Vision: Is there a plan to make future versions of the model compatible with vision (multimodal input)?
- Coding: Do you plan to improve its programming capabilities in future releases?
- Strict format following: In one of my tests the model produced good content but did not follow the delimiter rules of a strict output format. Details below. Would it be possible to improve this in future versions?
Setup
Backend / quantization: [llama.cpp / LM Studio, quant]
Thinking mode on
Temperature 1.0, top_p 0.95, top_k 20, min_p 0, repetition_penalty 1.0, presence_penalty [0 / 1.5]
Max tokens 100000
Task: I asked the model to generate 60 multiple-choice questions from attached Spanish study documents. The prompt (in Spanish) asks for one question per line, the correct option always first, each option followed by its own short explanation, and no intro text and no numbering. The relevant part of the prompt is:
Usa exclusivamente el símbolo "|" (barra vertical) para separar la pregunta de las opciones. Cada opción debe llevar su propia explicación breve separada del texto con " :: ". Prohibido usar "|" dentro de textos o explicaciones.
Required format: [Pregunta] | [Opción correcta] :: [Explicación] | [Distractor 1] :: [Explicación] | [Distractor 2] :: [Explicación] | [Distractor 3] :: [Explicación]
What went right: no intro, no numbering, the correct option is always first, the output is in Spanish, and every option has its own explanation.
What went wrong: the model used "|" between each option and its explanation instead of " :: ". No " :: " appeared in the visible output, so each line ends up with 9 fields instead of 5 and a parser splitting on "|" breaks. Excerpt:
¿Qué constituye el índice morfológico facial y qué cara se considera cuando el índice es igual a 1? | Índice entre la distancia entre nasion y mentón dividida entre la distancia intercigomática; con valor 1 corresponde a cara mesofacial | El documento define el índice morfológico facial como … | Índice entre nasion-mentón dividido por distancia bicanina | La fórmula correcta usa distancia intercigomática, … | …
It is possible that my prompt is ambiguous ("exclusively |" may conflict with the " :: " rule). I'm going to test a clearer wording and will report back if it changes anything.
Minor observations: in its reasoning, the model drafted the whole 60-question exam twice and re-planned the topic distribution several times before answering. Some distractors were weak (unrelated to the question) even though the prompt asked for highly plausible ones.
I'm happy to share the full prompt and the raw output if it helps.
Thanks again for your work!
Hi LorenzoLt, thanks for the kind words and thoughtful feedback! We really appreciate you taking the time to test the model and share your detailed observations.
Regarding your questions and suggestion:
- Vision: We’re considering multimodal support for future versions, though we can’t share a specific timeline at this point.
- Coding: Yes! We’re planning further improvements to coding capabilities for the official release.
- Strict format following: Thanks for highlighting this issue and providing a concrete example. We agree that reliable adherence to strict output formats is important, especially for parser-dependent workflows. We’ll take this feedback into account as we continue improving instruction following.
Your observations about repeated reasoning and distractor quality are also very helpful.
Thanks again for the detailed testing and support. We hope you'll have an even better experience with the official release!
Oh, and LorenzoLt, we'd love to take a look at the full prompt and raw output if you could share them!