TheDrummer

/

Behemoth-ReduX-123B-v1

like 12

This part of the chat template is too restrictive

{#- This block checks for alternating user/assistant messages, skipping tool calling messages #}
{%- set ns = namespace() %}
{%- set ns.index = 0 %}
{%- for message in loop_messages %}
    {%- if not (message.role == "tool" or message.role == "tool_results" or (message.tool_calls is defined and message.tool_calls is not none)) %}
        {%- if (message["role"] == "user") != (ns.index % 2 == 0) %}
            {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
        {%- endif %}
        {%- set ns.index = ns.index + 1 %}
    {%- endif %}
{%- endfor %}

I keep hitting it in vllm using ChatterUI as a frontend even on an empty chat.

Furthermore the check for ns.index % 2 == 0 is fragile and the conversation can be setup with an set initial assistant message.