jart25

/

Qwen3-Next-80B-A3B-Instruct-Int4-GPTQ

like 2

Tool calls working correctly to custom MCP.

Tool calls working correctly to custom MCP.

docker run -d
--name vllm1
--restart unless-stopped
--shm-size '48gb'
--network host
--ipc host
--privileged
--cap-add SYS_PTRACE
--security-opt seccomp=unconfined
--group-add video
-e HSA_OVERRIDE_GFX_VERSION=11.0.0
-e VLLM_USE_V1=1
-e VLLM_V1_USE_PREFILL_DECODE_ATTENTION=0
--device /dev/kfd:/dev/kfd
--device /dev/dri:/dev/dri
-v ./models:/models
vllm-rocm-251007
vllm serve
"/models/Qwen3-Next-80B-A3B-Instruct-w4g128"
--gpu-memory-utilization "0.95"
--max_model_len "262144"
-tp "4"
--served-model-name "QWEN3_80"
--port "80"
--disable-log-requests
--dtype=float16
--tool-call-parser "hermes"
--chat-template "/chat-template-tools.jinja"
--override-generation-config '{"temperature": 0.7, "top_p": 0.8, "top_k": 20, "repeat_penality": 1.05}'
--enable-auto-tool-choice
--compilation-config '{"full_cuda_graph": true}'
--no-enable-chunked-prefill
It is my command to run the model

Thanks jart, i'll see if the flags make a difference.
--chat-template "/chat-template-tools.jinja"
--compilation-config '{"full_cuda_graph": true}'
--no-enable-chunked-prefill

So far my startup command is performing flawlessly.

Appreciate you posting this, finding a quant to run on these cards was, challenging.

from unsloth

{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0].role == 'system' %}
{{- messages[0].content + '\n\n' }}
{%- endif %}
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{"name": , "arguments": }\n<|im_end|>\n" }}
{%- else %}
{%- if messages[0].role == 'system' %}
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
{%- for message in messages[::-1] %}
{%- set index = (messages|length - 1) - loop.index0 %}
{%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}
{%- set ns.multi_step_tool = false %}
{%- set ns.last_query_index = index %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{%- if message.content is string %}
{%- set content = message.content %}
{%- else %}
{%- set content = '' %}
{%- endif %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{%- set reasoning_content = '' %}
{%- if message.reasoning_content is string %}
{%- set reasoning_content = message.reasoning_content %}
{%- else %}
{%- if '' in content %}
{%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %}
{%- set content = content.split('')[-1].lstrip('\n') %}
{%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_query_index %}
{%- if loop.last or (not loop.last and reasoning_content) %}
{{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- else %}
{{- '<|im_start|>' + message.role + '\n' + content }}
{%- endif %}
{%- if message.tool_calls %}
{%- for tool_call in message.tool_calls %}
{%- if (loop.first and content) or (not loop.first) %}
{{- '\n' }}
{%- endif %}
{%- if tool_call.function %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{%- if tool_call.arguments is string %}
{{- tool_call.arguments }}
{%- else %}
{{- tool_call.arguments | tojson }}
{%- endif %}
{{- '}\n' }}
{%- endfor %}
{%- endif %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n\n' }}
{{- content }}
{{- '\n' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}

Thanks jart, i'll see if the flags make a difference.
--chat-template "/chat-template-tools.jinja"
--compilation-config '{"full_cuda_graph": true}'
--no-enable-chunked-prefill

So far my startup command is performing flawlessly.

Appreciate you posting this, finding a quant to run on these cards was, challenging.

It was quite a challenge, I ran into the same problem executing the MOEs (Mixtures of Experts), the limitations of rocm-no-aiter.

Rocm no Aiter barely has support for quantizations; this is the only one that, along with IsoTropy, has been left stable. It involved figuring out how the fused_moe were executed and more stories, so I had to fix it, hahaha. Anything you need, I'm here for you :)

Thanks and just fyi:
gfx1201 on rocm7 crashes generating cuda graphs, ho-hum, kind of expected.

Thus far this is hands down the absolute best model for my hardware, the right size of parameters, fast enough, tool calls without failures,

Amazing. Will see how it performs as agent over the weekend, but based on what I've seen thus far... it'll do great.

Post me the log and the full docker run command please

Removing as the problem was that cuda graphs and the previous version of rocm/vllm i was using were fundamentally incompatible.