Upload folder using huggingface_hub
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- .gitattributes +1 -0
- LICENSE +16 -0
- README.md +704 -0
- chat_template.jinja +170 -0
- config.json +162 -0
- generation_config.json +13 -0
- merges.txt +0 -0
- model-00001-of-00033.safetensors +3 -0
- model-00002-of-00033.safetensors +3 -0
- model-00003-of-00033.safetensors +3 -0
- model-00004-of-00033.safetensors +3 -0
- model-00005-of-00033.safetensors +3 -0
- model-00006-of-00033.safetensors +3 -0
- model-00007-of-00033.safetensors +3 -0
- model-00008-of-00033.safetensors +3 -0
- model-00009-of-00033.safetensors +3 -0
- model-00010-of-00033.safetensors +3 -0
- model-00011-of-00033.safetensors +3 -0
- model-00012-of-00033.safetensors +3 -0
- model-00013-of-00033.safetensors +3 -0
- model-00014-of-00033.safetensors +3 -0
- model-00015-of-00033.safetensors +3 -0
- model-00016-of-00033.safetensors +3 -0
- model-00017-of-00033.safetensors +3 -0
- model-00018-of-00033.safetensors +3 -0
- model-00019-of-00033.safetensors +3 -0
- model-00020-of-00033.safetensors +3 -0
- model-00021-of-00033.safetensors +3 -0
- model-00022-of-00033.safetensors +3 -0
- model-00023-of-00033.safetensors +3 -0
- model-00024-of-00033.safetensors +3 -0
- model-00025-of-00033.safetensors +3 -0
- model-00026-of-00033.safetensors +3 -0
- model-00027-of-00033.safetensors +3 -0
- model-00028-of-00033.safetensors +3 -0
- model-00029-of-00033.safetensors +3 -0
- model-00030-of-00033.safetensors +3 -0
- model-00031-of-00033.safetensors +3 -0
- model-00032-of-00033.safetensors +3 -0
- model-00033-of-00033.safetensors +3 -0
- model-auxiliary.safetensors +3 -0
- model.safetensors.index.json +0 -0
- preprocessor_config.json +21 -0
- reproduce/Qwen--Qwen3--8-Flash-Next.jsonl +0 -0
- reproduce/README.md +33 -0
- reproduce/SHA256SUMS +34 -0
- reproduce/config.toml +174 -0
- reproduce/reproduce.json +454 -0
- reproduce/requirements.txt +15 -0
- tokenizer.json +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
LICENSE
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Qwen Community License 1.0
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Qwen
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software, including the model weights, parameters, configuration files, inference code and associated documentation files (collectively, the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, sell, deploy, host, fine-tune, and create derivative works from (collectively, "Use" or "Using") copies of the Software; and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
| 6 |
+
|
| 7 |
+
1. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. If the Software (or any derivative works thereof) is Used for any of the licensee's commercial products or services that have more than 100,000,000 monthly active users or US$ 20,000,000 (or equivalent in other currencies) monthly revenue, respective model name must be prominently displayed on the user interface of such product or service; and,
|
| 8 |
+
|
| 9 |
+
2. If the licensee or any of its affiliates conducts a Model as a Service or AI Work Assistant business, the licensee shall obtain a separate license from Qwen before Using the Software or its derivative works for any commercial purpose. The foregoing requirement shall not apply to the licensee's internal Use of the Software, provided that such Use does not make the Software, its outputs, or its underlying model capabilities available to any third party.
|
| 10 |
+
|
| 11 |
+
"Model as a Service" means giving a third party access to language model inference or fine-tuning (e.g., via API or a hosted endpoint) in a manner that allows such third parties to exercise meaningful control over the inputs, parameters, or training data. This does not include the mere relaying of requests to models hosted by other third parties.
|
| 12 |
+
“AI Work Assistant” means an independent AI-powered product primarily designed for AI-assisted coding or office productivity (e.g., Qoder and QwenWork). It does not include: (a) a single-purpose AI tool (such as an AI translation tool); (b) an AI assistant primarily designed for a domain other than coding or office productivity (such as Taobao AI Shopping Assistant or AMap AI Chat); or (c) an AI assistant that is a feature of a product whose primary purpose is not AI-assisted coding or office productivity.
|
| 13 |
+
|
| 14 |
+
THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL QWEN, ITS AFFILIATES OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. THE USE OF THE SOFTWARE MUST COMPLY WITH APPLICABLE LAWS AND REGULATIONS, AND MUST NOT INFRINGE THE INTELLECTUAL PROPERTY RIGHTS OF ANY THIRD PARTY.
|
| 15 |
+
|
| 16 |
+
For any questions regarding this license, please contact model-business@notice.qwencloud.com.
|
README.md
ADDED
|
@@ -0,0 +1,704 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
license: other
|
| 4 |
+
license_name: qwen-community-1.0
|
| 5 |
+
license_link: LICENSE
|
| 6 |
+
pipeline_tag: image-text-to-text
|
| 7 |
+
tags:
|
| 8 |
+
- heretic
|
| 9 |
+
- uncensored
|
| 10 |
+
- decensored
|
| 11 |
+
- abliterated
|
| 12 |
+
- reproducible
|
| 13 |
+
---
|
| 14 |
+
# This is a decensored version of [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next), made using [Heretic](https://github.com/p-e-w/heretic) ([custom fork](https://github.com/timrohrbaugh/heretic)) v1.3.0+custom
|
| 15 |
+
|
| 16 |
+
## Abliteration parameters
|
| 17 |
+
|
| 18 |
+
| Parameter | Value |
|
| 19 |
+
| :-------- | :---: |
|
| 20 |
+
| **direction_index** | per layer |
|
| 21 |
+
| **attn.o_proj.max_weight** | 2.39 |
|
| 22 |
+
| **attn.o_proj.max_weight_position** | 41.36 |
|
| 23 |
+
| **attn.o_proj.min_weight** | 2.15 |
|
| 24 |
+
| **attn.o_proj.min_weight_distance** | 9.78 |
|
| 25 |
+
| **mlp.down_proj.max_weight** | 1.83 |
|
| 26 |
+
| **mlp.down_proj.max_weight_position** | 33.18 |
|
| 27 |
+
| **mlp.down_proj.min_weight** | 0.14 |
|
| 28 |
+
| **mlp.down_proj.min_weight_distance** | 2.68 |
|
| 29 |
+
|
| 30 |
+
## Performance
|
| 31 |
+
|
| 32 |
+
| Metric | This model | Original model ([Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)) |
|
| 33 |
+
| :----- | :--------: | :---------------------------: |
|
| 34 |
+
| **KL divergence** | 0.0818 | 0 *(by definition)* |
|
| 35 |
+
| **Refusals** | 0/100 | 99/100 |
|
| 36 |
+
|
| 37 |
+
-----
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
# Qwen3.8-Flash-Next
|
| 41 |
+
|
| 42 |
+
> [!Note]
|
| 43 |
+
> This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format.
|
| 44 |
+
>
|
| 45 |
+
> These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, TokenSpeed, etc.
|
| 46 |
+
|
| 47 |
+
> [!Tip]
|
| 48 |
+
> For users seeking managed, scalable inference without infrastructure maintenance, the official Qwen API service is provided by [Qwen Cloud](https://www.qwencloud.com).
|
| 49 |
+
>
|
| 50 |
+
> In particular, **Qwen3.8-Flash** is the official version based on Qwen3.8-Flash-Next with more production features, e.g., 1M context length by default, official built-in tools. For more information, please refer to the [Qwen3.8-Flash Overview](https://www.qwencloud.com/models/qwen3.8-flash).
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
As the frontier of foundation models pushes toward ever-larger parameter counts and ever-longer context windows, the question is no longer just how much we can scale, but how efficiently we can do so. Sustainable progress toward artificial general intelligence (AGI) that benefits everyone demands architectural innovation. Today, we are sharing a concrete step in that direction: Qwen3.8-Flash-Next.
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
This experimental preview of the architecture that will underpin Qwen4 is built around a fundamental rethinking of how the core components of modern large language models (LLMs) interact at scale.
|
| 58 |
+
|
| 59 |
+
## Highlights
|
| 60 |
+
|
| 61 |
+
The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:
|
| 62 |
+
|
| 63 |
+
- **Hybrid Attention with QSA**: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.
|
| 64 |
+
- **Gated Residual**: Residual streams with normalization are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.
|
| 65 |
+
- **N-gram Embedding**: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.
|
| 66 |
+
- **Tailored Training Recipe**: The Muon and AdamW optimizers are applied to specific weight categories to maximize efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimizer steps while safely supporting larger learning rates for robust convergence.
|
| 67 |
+
|
| 68 |
+
For more details, please refer to our blog post [Qwen3.8-Flash-Next](https://qwen.ai/blog?id=qwen3.8-flash-next) and [the technical report](https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf).
|
| 69 |
+
|
| 70 |
+
We are excited to embark on this next chapter with you and welcome your feedback as we build what comes next.
|
| 71 |
+
|
| 72 |
+
## Model Overview
|
| 73 |
+
|
| 74 |
+
- Type: Causal Language Model with Vision Encoder
|
| 75 |
+
- Training Stage: Pre-training & Post-training
|
| 76 |
+
- Language Model
|
| 77 |
+
- Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP
|
| 78 |
+
- Hidden Dimension: 2560
|
| 79 |
+
- Token Embedding: 248320 (Padded)
|
| 80 |
+
- N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)
|
| 81 |
+
- Number of Layers: 48
|
| 82 |
+
- Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))
|
| 83 |
+
- Gated DeltaNet:
|
| 84 |
+
- Number of Linear Attention Heads: 48 for V and 16 for QK
|
| 85 |
+
- Head Dimension: 128
|
| 86 |
+
- Qwen Sparse Attention:
|
| 87 |
+
- Number of Attention Heads: 24 for Q and 2 for KV
|
| 88 |
+
- Head Dimension: 256
|
| 89 |
+
- Rotary Position Embedding Dimension: 64
|
| 90 |
+
- Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head
|
| 91 |
+
- Indexer Head Dimension: 128
|
| 92 |
+
- Budget: 512 blocks or 2048 tokens
|
| 93 |
+
- Mixture Of Experts
|
| 94 |
+
- Number of Experts: 512
|
| 95 |
+
- Number of Activated Experts: 10 Routed + 1 Shared
|
| 96 |
+
- Expert Intermediate Dimension: 640
|
| 97 |
+
- Gated Residual:
|
| 98 |
+
- Number of Branches: 4
|
| 99 |
+
- Bottleneck Rank: 320
|
| 100 |
+
- LM Output: 248320 (Padded)
|
| 101 |
+
- MTP: 1 layer, trained with multi-steps
|
| 102 |
+
- Context Length: 262,144 natively and extensible up to 1,000,000 tokens.
|
| 103 |
+
|
| 104 |
+
## Benchmark Results
|
| 105 |
+
|
| 106 |
+
<style>
|
| 107 |
+
.vl-table th{font-size:15px!important;line-height:1.2}
|
| 108 |
+
.vl-table td:not(.benchmark-cell):not([colspan]){font-size:15px;line-height:1.2;vertical-align:middle}
|
| 109 |
+
.vl-table .benchmark-cell{padding:12px 10px 12px 18px!important;vertical-align:middle}
|
| 110 |
+
.vl-table .benchmark-capability{font-size:15px;font-weight:600;line-height:1.22;color:#171717}
|
| 111 |
+
.vl-table .benchmark-name{margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B}
|
| 112 |
+
.vl-table .metric-stack{display:flex;flex-direction:column;gap:7px;padding:3px 0}
|
| 113 |
+
.vl-table .metric-label{font-size:10px;font-weight:400;line-height:1.1;color:#777}
|
| 114 |
+
.vl-table .metric-value{margin-top:2px;font-size:15px;line-height:1.15;color:#171717}
|
| 115 |
+
.vl-table .metric-pair{white-space:nowrap}
|
| 116 |
+
.vl-table .metric-sep{color:#9A9A9A;padding:0 3px}
|
| 117 |
+
|
| 118 |
+
@media (prefers-color-scheme: dark){
|
| 119 |
+
.vl-table th,.vl-table td[colspan]{color:#8DA2FF!important;border-bottom-color:#8DA2FF!important}
|
| 120 |
+
.vl-table td[colspan]{background:rgba(10,46,254,0.22)!important}
|
| 121 |
+
.vl-table .benchmark-capability,.vl-table .metric-value{color:#E8E8E8!important}
|
| 122 |
+
.vl-table .benchmark-name,.vl-table .metric-label{color:#A3A3A3!important}
|
| 123 |
+
}
|
| 124 |
+
</style>
|
| 125 |
+
|
| 126 |
+
### Language
|
| 127 |
+
|
| 128 |
+
<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1200px;margin:0 auto;padding:16px 0">
|
| 129 |
+
<table class="vl-table" style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
|
| 130 |
+
<thead><tr>
|
| 131 |
+
<th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #0A2EFE;color:#0A2EFE"></th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;background:rgba(10, 46, 254, 0.08);">Qwen3.8-Flash-Next</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">Qwen3.8-27B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">Qwen3.7-Plus</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">DeepSeek-V4-Flash-0731</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">Claude-Opus-4.6 (Max)</th></tr></thead>
|
| 132 |
+
<tbody>
|
| 133 |
+
<tr>
|
| 134 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717"># Params</div></td>
|
| 135 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;">125B</td>
|
| 136 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">27B</td>
|
| 137 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">397B</td>
|
| 138 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">284B</td>
|
| 139 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 140 |
+
</tr>
|
| 141 |
+
<tr>
|
| 142 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717"># Activated params</div></td>
|
| 143 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;">6B</td>
|
| 144 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">27B</td>
|
| 145 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">17B</td>
|
| 146 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">13B</td>
|
| 147 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 148 |
+
</tr>
|
| 149 |
+
<tr>
|
| 150 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717"># N-gram embedding params</div></td>
|
| 151 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;">51B</td>
|
| 152 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 153 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 154 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 155 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 156 |
+
</tr>
|
| 157 |
+
<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC">Coding</td></tr>
|
| 158 |
+
<tr>
|
| 159 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Agentic coding</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">DeepSWE 1.1</div></td>
|
| 160 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>58.7</strong></td>
|
| 161 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">42.2</td>
|
| 162 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">16.5</td>
|
| 163 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">54.4</td>
|
| 164 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 165 |
+
</tr>
|
| 166 |
+
<tr>
|
| 167 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Agentic coding</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">SWE-bench Pro</div></td>
|
| 168 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>62.5</strong></td>
|
| 169 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">61.7</td>
|
| 170 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">55.8</td>
|
| 171 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">56.0</td>
|
| 172 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">53.4</td>
|
| 173 |
+
</tr>
|
| 174 |
+
<tr>
|
| 175 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Multilingual software engineering</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">SWE-bench Multilingual</div></td>
|
| 176 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>81.0</strong></td>
|
| 177 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">73.8</td>
|
| 178 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">75.8</td>
|
| 179 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 180 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">77.5</td>
|
| 181 |
+
</tr>
|
| 182 |
+
<tr>
|
| 183 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Repo-level code generation</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">NL2Repo-Bench</div></td>
|
| 184 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;">48.1</td>
|
| 185 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">42.3</td>
|
| 186 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">41.1</td>
|
| 187 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>54.2</strong></td>
|
| 188 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">47.6</td>
|
| 189 |
+
</tr>
|
| 190 |
+
<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC">Agent</td></tr>
|
| 191 |
+
<tr>
|
| 192 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Long-horizon office work</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">CoWorkBench</div></td>
|
| 193 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>73.9</strong></td>
|
| 194 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">70.7</td>
|
| 195 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">65.1</td>
|
| 196 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">45.1</td>
|
| 197 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">68.2</td>
|
| 198 |
+
</tr>
|
| 199 |
+
<tr>
|
| 200 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Professional job tasks</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">JobBench</div></td>
|
| 201 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>55.7</strong></td>
|
| 202 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">33.4</td>
|
| 203 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">27.6</td>
|
| 204 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">41.3</td>
|
| 205 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">36.6</td>
|
| 206 |
+
</tr>
|
| 207 |
+
<tr>
|
| 208 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Frontier agentic tasks</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">Agents' Last Exam</div></td>
|
| 209 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@1</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">24.3</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Score</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>51.2</strong></div></div></div></td>
|
| 210 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@1</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">20.4</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Score</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">42.9</div></div></div></td>
|
| 211 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@1</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">13.2</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Score</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">33.6</div></div></div></td>
|
| 212 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@1</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>25.2</strong></div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Score</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">--</div></div></div></td>
|
| 213 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 214 |
+
</tr>
|
| 215 |
+
<tr>
|
| 216 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Real-world tool use</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">Toolathlon Verified (Pass@1)</div></td>
|
| 217 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>73.5</strong></td>
|
| 218 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">67.1</td>
|
| 219 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">50.6</td>
|
| 220 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">70.3</td>
|
| 221 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 222 |
+
</tr>
|
| 223 |
+
<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC">General</td></tr>
|
| 224 |
+
<tr>
|
| 225 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Instruction following</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">IFBench</div></td>
|
| 226 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>81.3</strong></td>
|
| 227 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">79.5</td>
|
| 228 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">79.1</td>
|
| 229 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">79.2</td>
|
| 230 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">62.5</td>
|
| 231 |
+
</tr>
|
| 232 |
+
<tr>
|
| 233 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Scientific reasoning</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">GPQA Diamond</div></td>
|
| 234 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>91.7</strong></td>
|
| 235 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">89.2</td>
|
| 236 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">90.3</td>
|
| 237 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">90.8</td>
|
| 238 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">91.3</td>
|
| 239 |
+
</tr>
|
| 240 |
+
<tr>
|
| 241 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Multidisciplinary reasoning</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">HLE</div></td>
|
| 242 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;">35.9</td>
|
| 243 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">30.8</td>
|
| 244 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">34.7</td>
|
| 245 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">33.8</td>
|
| 246 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>40.0</strong></td>
|
| 247 |
+
</tr>
|
| 248 |
+
<tr>
|
| 249 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Competitive coding</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">LiveCodeBench v6</div></td>
|
| 250 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>91.9</strong></td>
|
| 251 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">90.3</td>
|
| 252 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">89.6</td>
|
| 253 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">90.6</td>
|
| 254 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">88.8</td>
|
| 255 |
+
</tr>
|
| 256 |
+
</tbody>
|
| 257 |
+
</table>
|
| 258 |
+
<p style="margin-top:12px;font-size:10px;line-height:1.4;opacity:0.7">1. DeepSWE 1.1: evaluated with the Claude Code and mini-SWE-agent harnesses, temp=1.0, top_p=0.95, 256K context window. We report the highest score across the two harnesses; notably, Qwen3.8-Flash-Next performs best on mini-SWE-agent.<br>2. SWE-bench Pro: except for Claude-Opus-4.6 (Max), for which we report the officially published score, all models are evaluated with the Claude Code harness, temp=1.0, top_p=0.95, 256K context window. Problematic tasks were corrected and all baseline models were re-evaluated on the refined benchmark.<br>3. SWE-bench Multilingual: evaluated with the mini-SWE-agent harness, temp=1.0, top_p=0.95, 256K context window.<br>4. NL2Repo-Bench: evaluated with the Claude Code harness. To prevent reward hacking, we disable Bash commands that attempt to access the specific repository, such as pip download, pip install and git clone.<br>5. CoWorkBench: an in-house cowork benchmark for evaluating long-horizon office and productivity agent tasks across computer science, finance, law, medical and other productivity domains.<br>6. HLE: judged by GPT-4o.<br>7. The best result in each row is shown in bold.<br>8. Empty cells (--): scores are not yet available or are not applicable.</p>
|
| 259 |
+
</div>
|
| 260 |
+
|
| 261 |
+
### Vision Language
|
| 262 |
+
|
| 263 |
+
<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0">
|
| 264 |
+
<table class="vl-table" style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
|
| 265 |
+
<thead><tr>
|
| 266 |
+
<th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #0A2EFE;color:#0A2EFE"></th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;background:rgba(10, 46, 254, 0.08);">Qwen3.8-Flash-Next</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;">Qwen3.8-27B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;">Qwen3.7-Plus</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:16.00%;">Claude-Opus-4.6 (Max)</th></tr></thead>
|
| 267 |
+
<tbody>
|
| 268 |
+
<tr><td colspan="5" style="padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC">Agentic Multimodal Intelligence</td></tr>
|
| 269 |
+
<tr>
|
| 270 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Multimodal tool use</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">ClawEval-MM</div></td>
|
| 271 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@3</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>64.4</strong></div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Average</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>60.4</strong></div></div></div></td>
|
| 272 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@3</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">57.4</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Average</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">56.9</div></div></div></td>
|
| 273 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@3</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">57.4</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Average</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">60.1</div></div></div></td>
|
| 274 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Pass@3</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">52.5</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Average</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">54.7</div></div></div></td>
|
| 275 |
+
</tr>
|
| 276 |
+
<tr>
|
| 277 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Application recreation</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">RecreationBench</div></td>
|
| 278 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>49.9</strong></td>
|
| 279 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">47.1</td>
|
| 280 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">30.2</td>
|
| 281 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 282 |
+
</tr>
|
| 283 |
+
<tr>
|
| 284 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Mobile use</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">AndroidWorld</div></td>
|
| 285 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>84.5</strong></td>
|
| 286 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">81.9</td>
|
| 287 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">81.0</td>
|
| 288 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">62.0</td>
|
| 289 |
+
</tr>
|
| 290 |
+
<tr>
|
| 291 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Computer use</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">OSWorld 2.0</div></td>
|
| 292 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Binary</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>19.4</strong></div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Partial</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>52.3</strong></div></div></div></td>
|
| 293 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Binary</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">19.4</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Partial</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">48.0</div></div></div></td>
|
| 294 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Binary</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">2.8</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Partial</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">21.5</div></div></div></td>
|
| 295 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 296 |
+
</tr>
|
| 297 |
+
<tr>
|
| 298 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Visual web development</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">Vision2Web</div></td>
|
| 299 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>64.0</strong></td>
|
| 300 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">62.9</td>
|
| 301 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">42.1</td>
|
| 302 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">--</td>
|
| 303 |
+
</tr>
|
| 304 |
+
<tr><td colspan="5" style="padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC">General Multimodal Intelligence</td></tr>
|
| 305 |
+
<tr>
|
| 306 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Embodied intelligence</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">ERQA</div></td>
|
| 307 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>72.3</strong></td>
|
| 308 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">65.5</td>
|
| 309 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">69.8</td>
|
| 310 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">40.8</td>
|
| 311 |
+
</tr>
|
| 312 |
+
<tr>
|
| 313 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Long video understanding</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">LVBench</div></td>
|
| 314 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>76.6</strong></td>
|
| 315 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">72.4</td>
|
| 316 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">76.2</td>
|
| 317 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">63.0</td>
|
| 318 |
+
</tr>
|
| 319 |
+
<tr>
|
| 320 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Real-world perception</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">RealWorldQA</div></td>
|
| 321 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>88.5</strong></td>
|
| 322 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">85.9</td>
|
| 323 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">86.9</td>
|
| 324 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">73.9</td>
|
| 325 |
+
</tr>
|
| 326 |
+
<tr>
|
| 327 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Visual math problem solving</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">MathVision</div></td>
|
| 328 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>90.6</strong></div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">With CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>95.7</strong></div></div></div></td>
|
| 329 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">90.0</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">With CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">94.6</div></div></div></td>
|
| 330 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">90.3</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">With CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">88.7</div></div></div></td>
|
| 331 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">65.5</div></div></div></td>
|
| 332 |
+
</tr>
|
| 333 |
+
<tr>
|
| 334 |
+
<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Scientific chart analysis</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">CharXiv (RQ)</div></td>
|
| 335 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(10, 46, 254, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">84.6</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">With CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>90.6</strong></div></div></div></td>
|
| 336 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">83.7</div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">With CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">90.2</div></div></div></td>
|
| 337 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717"><strong>85.8</strong></div></div><div style="margin-top:7px"><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">With CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">85.9</div></div></div></td>
|
| 338 |
+
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><div class="metric-stack" style="padding:3px 0"><div><div class="metric-label" style="font-size:10px;font-weight:400;line-height:1.1;color:#777">Without CI</div><div class="metric-value" style="margin-top:2px;font-size:15px;line-height:1.15;color:#171717">66.0</div></div></div></td>
|
| 339 |
+
</tr>
|
| 340 |
+
</tbody>
|
| 341 |
+
</table>
|
| 342 |
+
<p style="margin-top:12px;font-size:10px;line-height:1.4;opacity:0.7">1. ClawEval-MM: scores are reported as "pass@3 / average score". Pass@3 measures the percentage passed in at least one of three trials, and the average score is the mean score across the three trials.<br>2. RecreationBench: an in-house long-horizon application-recreation benchmark for evaluating hybrid-agent abilities spanning five platforms — desktop (Ubuntu, macOS, Windows), mobile (Android) and web.<br>3. OSWorld 2.0: scores are reported as "binary / partial". The binary score is the percentage of tasks that receive the full task reward, while the partial score aggregates the partial rewards obtained across all tasks.<br>4. Vision2Web: scores are reported as the average over the frontend, webpage and website categories, using the Claude Code harness and judged by gpt-5.4-2026-03-05.<br>5. MathVision, CharXiv (RQ): scores are reported as "without CI / with CI". A small number of incorrect ground-truth annotations in MathVision were corrected after manual verification. Our model's score is evaluated using a fixed prompt, e.g. "Please reason step by step, and put your final answer within \boxed{}." For other models, we report the higher score between runs with and without the \boxed{} formatting.<br>6. The best result in each row is shown in bold.<br>7. Empty cells (--) indicate scores not yet available or not applicable.</p>
|
| 343 |
+
</div>
|
| 344 |
+
|
| 345 |
+
|
| 346 |
+
## Quickstart
|
| 347 |
+
|
| 348 |
+
For streamlined integration, we recommend using Qwen3.8-Flash-Next via APIs.
|
| 349 |
+
|
| 350 |
+
### Serving Qwen3.8-Flash-Next
|
| 351 |
+
|
| 352 |
+
> [!Important]
|
| 353 |
+
> Inference efficiency and throughput vary significantly across frameworks.
|
| 354 |
+
> We recommend using the latest framework versions to ensure optimal performance and compatibility.
|
| 355 |
+
> For production workloads or high-throughput scenarios, dedicated serving engines such as SGLang, KTransformers or vLLM are strongly recommended.
|
| 356 |
+
|
| 357 |
+
|
| 358 |
+
Qwen3.8-Flash-Next can be deployed with popular inference frameworks, e.g.:
|
| 359 |
+
|
| 360 |
+
- [SGLang](https://www.sglang.io/): [Qwen3.8-Flash-Next Cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next)
|
| 361 |
+
- [vLLM](https://vllm.ai/): [Qwen3.8-Flash-Next Recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next)
|
| 362 |
+
- [TokenSpeed](https://lightseek.org/tokenspeed/): [Qwen3.8-Flash-Next Recipe](https://lightseek.org/tokenspeed/recipes/models#qwen3-8-flash-next)
|
| 363 |
+
|
| 364 |
+
|
| 365 |
+
### API Usage
|
| 366 |
+
|
| 367 |
+
> [!Important]
|
| 368 |
+
> Qwen3.8-Flash-Next models operate in thinking mode by default, generating thinking content signified by `<think>\n...</think>\n\n` before producing the final responses.
|
| 369 |
+
> To disable thinking content and obtain direct response, refer to the examples [here](#instruct-or-non-thinking-mode).
|
| 370 |
+
|
| 371 |
+
> [!Tip]
|
| 372 |
+
> We recommend using the following sets of sampling parameters for generation:
|
| 373 |
+
> - Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`
|
| 374 |
+
> - Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`
|
| 375 |
+
>
|
| 376 |
+
> Please note that the support for sampling parameters varies according to inference frameworks.
|
| 377 |
+
|
| 378 |
+
> [!Tip]
|
| 379 |
+
> In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time. Although it may produce faster per-turn responses, it can also lead to insufficient analysis, more failures, and repeated retries, which may increase total latency and token consumption.
|
| 380 |
+
|
| 381 |
+
Qwen3.8-Flash-Next supports controlling thinking behavior via `enable_thinking`, `preserve_thinking`, and `reasoning_effort`.
|
| 382 |
+
|
| 383 |
+
#### Chat Completions API
|
| 384 |
+
|
| 385 |
+
The Chat Completions API can be used with most inference frameworks, as well as [Qwen Cloud](https://www.qwencloud.com/).
|
| 386 |
+
Before starting, make sure it is installed and the API key and the API base URL is configured, e.g.:
|
| 387 |
+
```shell
|
| 388 |
+
pip install -U openai
|
| 389 |
+
|
| 390 |
+
# Set the following accordingly
|
| 391 |
+
export OPENAI_BASE_URL="http://localhost:8000/v1"
|
| 392 |
+
export OPENAI_API_KEY="EMPTY"
|
| 393 |
+
```
|
| 394 |
+
|
| 395 |
+
##### Text-Only Input
|
| 396 |
+
|
| 397 |
+
```python
|
| 398 |
+
from openai import OpenAI
|
| 399 |
+
# Configured by environment variables
|
| 400 |
+
client = OpenAI()
|
| 401 |
+
|
| 402 |
+
messages = [
|
| 403 |
+
{"role": "user", "content": "Write a Python function to merge two sorted linked lists."},
|
| 404 |
+
]
|
| 405 |
+
|
| 406 |
+
completion = client.chat.completions.create(
|
| 407 |
+
model="Qwen/Qwen3.8-Flash-Next",
|
| 408 |
+
messages=messages,
|
| 409 |
+
extra_body={
|
| 410 |
+
"chat_template_kwargs": {
|
| 411 |
+
"enable_thinking": True, # on by default
|
| 412 |
+
"preserve_thinking": True, # on by default
|
| 413 |
+
},
|
| 414 |
+
},
|
| 415 |
+
reasoning_effort="xhigh", # xhigh by default; supported levels are xhigh, medium, and low
|
| 416 |
+
stream=True,
|
| 417 |
+
stream_options={"include_usage": True},
|
| 418 |
+
)
|
| 419 |
+
|
| 420 |
+
reasoning_content = ""
|
| 421 |
+
answer_content = ""
|
| 422 |
+
is_answering = False
|
| 423 |
+
print("\n" + "=" * 20 + "Reasoning" + "=" * 20 + "\n")
|
| 424 |
+
|
| 425 |
+
for chunk in completion:
|
| 426 |
+
if not chunk.choices:
|
| 427 |
+
print("\nUsage:")
|
| 428 |
+
print(chunk.usage)
|
| 429 |
+
continue
|
| 430 |
+
|
| 431 |
+
delta = chunk.choices[0].delta
|
| 432 |
+
|
| 433 |
+
if hasattr(delta, "reasoning_content") and delta.reasoning_content is not None:
|
| 434 |
+
if not is_answering:
|
| 435 |
+
print(delta.reasoning_content, end="", flush=True)
|
| 436 |
+
reasoning_content += delta.reasoning_content
|
| 437 |
+
elif hasattr(delta, "reasoning") and delta.reasoning is not None:
|
| 438 |
+
if not is_answering:
|
| 439 |
+
print(delta.reasoning, end="", flush=True)
|
| 440 |
+
reasoning_content += delta.reasoning
|
| 441 |
+
|
| 442 |
+
if hasattr(delta, "content") and delta.content:
|
| 443 |
+
if not is_answering:
|
| 444 |
+
print("\n" + "=" * 20 + "Answer" + "=" * 20 + "\n")
|
| 445 |
+
is_answering = True
|
| 446 |
+
print(delta.content, end="", flush=True)
|
| 447 |
+
answer_content += delta.content
|
| 448 |
+
|
| 449 |
+
messages.append({
|
| 450 |
+
"role": "assistant",
|
| 451 |
+
"content": answer_content,
|
| 452 |
+
"reasoning_content": reasoning_content,
|
| 453 |
+
"reasoning": reasoning_content,
|
| 454 |
+
})
|
| 455 |
+
```
|
| 456 |
+
|
| 457 |
+
|
| 458 |
+
##### Image Input
|
| 459 |
+
|
| 460 |
+
```python
|
| 461 |
+
from openai import OpenAI
|
| 462 |
+
# Configured by environment variables
|
| 463 |
+
client = OpenAI()
|
| 464 |
+
|
| 465 |
+
messages = [
|
| 466 |
+
{
|
| 467 |
+
"role": "user",
|
| 468 |
+
"content": [
|
| 469 |
+
{
|
| 470 |
+
"type": "image_url",
|
| 471 |
+
"image_url": {
|
| 472 |
+
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
|
| 473 |
+
}
|
| 474 |
+
},
|
| 475 |
+
{
|
| 476 |
+
"type": "text",
|
| 477 |
+
"text": "The centres of the four illustrated circles are in the corners of the square. The two big circles touch each other and also the two little circles. With which factor do you have to multiply the radii of the little circles to obtain the radius of the big circles?\nChoices:\n(A) $\\frac{2}{9}$\n(B) $\\sqrt{5}$\n(C) $0.8 \\cdot \\pi$\n(D) 2.5\n(E) $1+\\sqrt{2}$"
|
| 478 |
+
}
|
| 479 |
+
]
|
| 480 |
+
}
|
| 481 |
+
]
|
| 482 |
+
|
| 483 |
+
chat_response = client.chat.completions.create(
|
| 484 |
+
model="Qwen/Qwen3.8-Flash-Next",
|
| 485 |
+
messages=messages,
|
| 486 |
+
)
|
| 487 |
+
print("Chat response:", chat_response)
|
| 488 |
+
```
|
| 489 |
+
|
| 490 |
+
##### Video Input
|
| 491 |
+
|
| 492 |
+
```python
|
| 493 |
+
from openai import OpenAI
|
| 494 |
+
# Configured by environment variables
|
| 495 |
+
client = OpenAI()
|
| 496 |
+
|
| 497 |
+
messages = [
|
| 498 |
+
{
|
| 499 |
+
"role": "user",
|
| 500 |
+
"content": [
|
| 501 |
+
{
|
| 502 |
+
"type": "video_url",
|
| 503 |
+
"video_url": {
|
| 504 |
+
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
|
| 505 |
+
}
|
| 506 |
+
},
|
| 507 |
+
{
|
| 508 |
+
"type": "text",
|
| 509 |
+
"text": "How many porcelain jars were discovered in the niches located in the primary chamber of the tomb?"
|
| 510 |
+
}
|
| 511 |
+
]
|
| 512 |
+
}
|
| 513 |
+
]
|
| 514 |
+
|
| 515 |
+
chat_response = client.chat.completions.create(
|
| 516 |
+
model="Qwen/Qwen3.8-Flash-Next",
|
| 517 |
+
messages=messages,
|
| 518 |
+
)
|
| 519 |
+
|
| 520 |
+
# When vLLM is launched with `--media-io-kwargs '{"video": {"num_frames": -1}}'`,
|
| 521 |
+
# video frame sampling can be configured via `extra_body` (e.g., by setting `fps`).
|
| 522 |
+
# This feature is currently supported only in vLLM.
|
| 523 |
+
#
|
| 524 |
+
# By default, `fps=2` and `do_sample_frames=True`.
|
| 525 |
+
# With `do_sample_frames=True`, you can customize the `fps` value to set your desired video sampling rate.
|
| 526 |
+
# chat_response = client.chat.completions.create(
|
| 527 |
+
# model="Qwen/Qwen3.8-Flash-Next",
|
| 528 |
+
# messages=messages,
|
| 529 |
+
# extra_body={
|
| 530 |
+
# "mm_processor_kwargs": {"fps": 2, "do_sample_frames": True},
|
| 531 |
+
# },
|
| 532 |
+
# )
|
| 533 |
+
|
| 534 |
+
print("Chat response:", chat_response)
|
| 535 |
+
```
|
| 536 |
+
|
| 537 |
+
|
| 538 |
+
##### Instruct (or Non-Thinking) Mode
|
| 539 |
+
|
| 540 |
+
Qwen3.8-Flash-Next will think by default before responding.
|
| 541 |
+
You can obtain a direct response from the model without thinking by configuring the API parameters.
|
| 542 |
+
For example,
|
| 543 |
+
```python
|
| 544 |
+
from openai import OpenAI
|
| 545 |
+
# Configured by environment variables
|
| 546 |
+
client = OpenAI()
|
| 547 |
+
|
| 548 |
+
messages = [
|
| 549 |
+
{
|
| 550 |
+
"role": "user",
|
| 551 |
+
"content": [
|
| 552 |
+
{
|
| 553 |
+
"type": "image_url",
|
| 554 |
+
"image_url": {
|
| 555 |
+
"url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/RealWorld/RealWorld-04.png"
|
| 556 |
+
}
|
| 557 |
+
},
|
| 558 |
+
{
|
| 559 |
+
"type": "text",
|
| 560 |
+
"text": "Where is this?"
|
| 561 |
+
}
|
| 562 |
+
]
|
| 563 |
+
}
|
| 564 |
+
]
|
| 565 |
+
|
| 566 |
+
chat_response = client.chat.completions.create(
|
| 567 |
+
model="Qwen/Qwen3.8-Flash-Next",
|
| 568 |
+
messages=messages,
|
| 569 |
+
temperature=0.7,
|
| 570 |
+
top_p=0.8,
|
| 571 |
+
presence_penalty=1.5,
|
| 572 |
+
extra_body={
|
| 573 |
+
"top_k": 20,
|
| 574 |
+
"chat_template_kwargs": {"enable_thinking": False},
|
| 575 |
+
},
|
| 576 |
+
)
|
| 577 |
+
print("Chat response:", chat_response)
|
| 578 |
+
```
|
| 579 |
+
|
| 580 |
+
> [!Note]
|
| 581 |
+
> If you are using APIs from Qwen Cloud, in addition to changing `model`, please use `"enable_thinking": False` instead of `"chat_template_kwargs": {"enable_thinking": False}`.
|
| 582 |
+
|
| 583 |
+
|
| 584 |
+
##### Disable Preserved Thinking
|
| 585 |
+
|
| 586 |
+
By default, Qwen3.8-Flash-Next retains thinking blocks from all historical messages, maintaining a complete reasoning trace across the conversation. This behavior, known as preserved thinking, ensures full context continuity and is especially beneficial for agent scenarios where decision consistency and reduced redundant reasoning are critical. It also improves KV cache utilization, optimizing inference efficiency in both thinking and non-thinking modes.
|
| 587 |
+
|
| 588 |
+
If you prefer to retain only the thinking blocks from the latest user message, you can disable this behavior by setting `preserve_thinking` to `False`:
|
| 589 |
+
|
| 590 |
+
```python
|
| 591 |
+
from openai import OpenAI
|
| 592 |
+
|
| 593 |
+
# Configured by environment variables
|
| 594 |
+
client = OpenAI()
|
| 595 |
+
messages = [...]
|
| 596 |
+
chat_response = client.chat.completions.create(
|
| 597 |
+
model="Qwen/Qwen3.8-Flash-Next",
|
| 598 |
+
messages=messages,
|
| 599 |
+
extra_body={
|
| 600 |
+
"chat_template_kwargs": {"preserve_thinking": False},
|
| 601 |
+
},
|
| 602 |
+
)
|
| 603 |
+
print("Chat response:", chat_response)
|
| 604 |
+
```
|
| 605 |
+
|
| 606 |
+
> [!Note]
|
| 607 |
+
> If you are using APIs from Qwen Cloud, in addition to changing `model`, please use `"preserve_thinking": False` directly instead of wrapping it in `chat_template_kwargs`.
|
| 608 |
+
|
| 609 |
+
|
| 610 |
+
|
| 611 |
+
## Best Practices
|
| 612 |
+
|
| 613 |
+
To achieve optimal performance, we recommend the following settings:
|
| 614 |
+
|
| 615 |
+
1. **Sampling Parameters**: We suggest using the following sets of sampling parameters:
|
| 616 |
+
|
| 617 |
+
- Thinking Mode: `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`
|
| 618 |
+
- Instruct (or non-thinking) mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0`
|
| 619 |
+
|
| 620 |
+
For supported frameworks, you can adjust the `presence_penalty` parameter between 0 and 2 to reduce endless repetition. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
|
| 621 |
+
|
| 622 |
+
2. **Adequate Output Length**: To optimize performance on agentic tasks, we recommend allocating sufficient output length to allow the model to generate detailed and comprehensive responses. For frameworks that support separate token limits for internal reasoning and final outputs, we suggest the following configuration within the 1M context length:
|
| 623 |
+
|
| 624 |
+
- Reasoning Content: Set the maximum output length to 262,144 tokens.
|
| 625 |
+
- Final Response: Set the maximum output length to 131,072 tokens.
|
| 626 |
+
|
| 627 |
+
These settings provide the necessary capacity for complex reasoning while ensuring ample space for high-quality final deliverables.
|
| 628 |
+
|
| 629 |
+
3. **Processing Ultra-Long Texts**: Qwen3.8-Flash-Next natively supports context lengths of up to 262,144 tokens. For long-horizon tasks where the total length (including both input and output) exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively, e.g., YaRN.
|
| 630 |
+
|
| 631 |
+
YaRN is currently supported by several inference frameworks, e.g., vLLM, SGLang, and TokenSpeed.
|
| 632 |
+
In general, there are two approaches to enabling YaRN for supported frameworks:
|
| 633 |
+
|
| 634 |
+
- Modifying the model configuration file:
|
| 635 |
+
|
| 636 |
+
In the `config.json` file, change the `rope_parameters` fields in `text_config` to:
|
| 637 |
+
```json
|
| 638 |
+
{
|
| 639 |
+
"mrope_interleaved": true,
|
| 640 |
+
"mrope_section": [
|
| 641 |
+
11,
|
| 642 |
+
11,
|
| 643 |
+
10
|
| 644 |
+
],
|
| 645 |
+
"rope_type": "yarn",
|
| 646 |
+
"rope_theta": 10000000,
|
| 647 |
+
"partial_rotary_factor": 0.25,
|
| 648 |
+
"factor": 4.0,
|
| 649 |
+
"original_max_position_embeddings": 262144
|
| 650 |
+
}
|
| 651 |
+
```
|
| 652 |
+
|
| 653 |
+
- Passing command line arguments:
|
| 654 |
+
|
| 655 |
+
For vLLM, you can use
|
| 656 |
+
```shell
|
| 657 |
+
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve ... --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' --max-model-len 1000000
|
| 658 |
+
```
|
| 659 |
+
|
| 660 |
+
For SGLang, you can use
|
| 661 |
+
```shell
|
| 662 |
+
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server ... --json-model-override-args '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' --context-length 1000000
|
| 663 |
+
```
|
| 664 |
+
|
| 665 |
+
For TokenSpeed, you can use
|
| 666 |
+
```shell
|
| 667 |
+
TOKENSPEED_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 tokenspeed serve ... --hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' --max-model-len 1000000
|
| 668 |
+
```
|
| 669 |
+
|
| 670 |
+
> [!NOTE]
|
| 671 |
+
> All the notable open-source frameworks implement static YaRN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts.**
|
| 672 |
+
> We advise modifying the `rope_parameters` configuration only when processing long contexts is required.
|
| 673 |
+
> It is also recommended to modify the `factor` as needed. For example, if the typical context length for your application is 524,288 tokens, it would be better to set `factor` as 2.0.
|
| 674 |
+
|
| 675 |
+
|
| 676 |
+
4. **Long Video Understanding**: To optimize inference efficiency for plain text and images, the `size` parameter in the released `video_preprocessor_config.json` is conservatively configured. It is recommended to set the `longest_edge` parameter in the video_preprocessor_config file to 469,762,048 (corresponding to 224k video tokens) to enable higher frame-rate sampling for hour-scale videos and thereby achieve superior performance. For example,
|
| 677 |
+
```json
|
| 678 |
+
{"longest_edge": 469762048, "shortest_edge": 4096}
|
| 679 |
+
```
|
| 680 |
+
|
| 681 |
+
Alternatively, override the default values via engine startup parameters. For implementation details, refer to: [vLLM](https://github.com/vllm-project/vllm/pull/34330) / [SGLang](https://github.com/sgl-project/sglang/pull/18467).
|
| 682 |
+
|
| 683 |
+
|
| 684 |
+
### Citation
|
| 685 |
+
|
| 686 |
+
If you find our work helpful, feel free to give us a cite.
|
| 687 |
+
|
| 688 |
+
```bibtex
|
| 689 |
+
@techreport{qwen2026design,
|
| 690 |
+
title = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},
|
| 691 |
+
author = {{Qwen Team}},
|
| 692 |
+
institution = {Alibaba Group},
|
| 693 |
+
month = {August},
|
| 694 |
+
year = {2026}
|
| 695 |
+
}
|
| 696 |
+
|
| 697 |
+
@misc{qwen3.8flashnext,
|
| 698 |
+
title = {{Qwen3.8-Flash-Next}: A New Architecture, Towards Ultimate Cost-Efficiency},
|
| 699 |
+
author = {{Qwen Team}},
|
| 700 |
+
month = {August},
|
| 701 |
+
year = {2026},
|
| 702 |
+
url = {https://qwen.ai/blog?id=qwen3.8-flash-next}
|
| 703 |
+
}
|
| 704 |
+
```
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- set image_count = namespace(value=0) %}
|
| 2 |
+
{%- set video_count = namespace(value=0) %}
|
| 3 |
+
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
| 4 |
+
{%- if content is string %}
|
| 5 |
+
{{- content }}
|
| 6 |
+
{%- elif content is iterable and content is not mapping %}
|
| 7 |
+
{%- for item in content %}
|
| 8 |
+
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
| 9 |
+
{%- if is_system_content %}
|
| 10 |
+
{{- raise_exception('System message cannot contain images.') }}
|
| 11 |
+
{%- endif %}
|
| 12 |
+
{%- if do_vision_count %}
|
| 13 |
+
{%- set image_count.value = image_count.value + 1 %}
|
| 14 |
+
{%- endif %}
|
| 15 |
+
{%- if add_vision_id %}
|
| 16 |
+
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
| 17 |
+
{%- endif %}
|
| 18 |
+
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
| 19 |
+
{%- elif 'video' in item or item.type == 'video' %}
|
| 20 |
+
{%- if is_system_content %}
|
| 21 |
+
{{- raise_exception('System message cannot contain videos.') }}
|
| 22 |
+
{%- endif %}
|
| 23 |
+
{%- if do_vision_count %}
|
| 24 |
+
{%- set video_count.value = video_count.value + 1 %}
|
| 25 |
+
{%- endif %}
|
| 26 |
+
{%- if add_vision_id %}
|
| 27 |
+
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
| 28 |
+
{%- endif %}
|
| 29 |
+
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
| 30 |
+
{%- elif 'text' in item %}
|
| 31 |
+
{{- item.text }}
|
| 32 |
+
{%- else %}
|
| 33 |
+
{{- raise_exception('Unexpected item type in content.') }}
|
| 34 |
+
{%- endif %}
|
| 35 |
+
{%- endfor %}
|
| 36 |
+
{%- elif content is none or content is undefined %}
|
| 37 |
+
{{- '' }}
|
| 38 |
+
{%- else %}
|
| 39 |
+
{{- raise_exception('Unexpected content type.') }}
|
| 40 |
+
{%- endif %}
|
| 41 |
+
{%- endmacro %}
|
| 42 |
+
{%- if not messages %}
|
| 43 |
+
{{- raise_exception('No messages provided.') }}
|
| 44 |
+
{%- endif %}
|
| 45 |
+
{%- set reasoning_instructions = '' %}
|
| 46 |
+
{%- if enable_thinking is undefined or enable_thinking is true %}
|
| 47 |
+
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
| 48 |
+
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
| 49 |
+
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
| 50 |
+
{%- endif %}
|
| 51 |
+
{%- if resolved_reasoning_effort == 'xhigh' %}
|
| 52 |
+
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
| 53 |
+
{%- elif resolved_reasoning_effort == 'low' %}
|
| 54 |
+
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
| 55 |
+
{%- endif %}
|
| 56 |
+
{%- endif %}
|
| 57 |
+
{%- if tools and tools is iterable and tools is not mapping %}
|
| 58 |
+
{{- '<|im_start|>system\n' }}
|
| 59 |
+
{%- if reasoning_instructions %}
|
| 60 |
+
{{- reasoning_instructions + '\n\n' }}
|
| 61 |
+
{%- endif %}
|
| 62 |
+
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
| 63 |
+
{%- for tool in tools %}
|
| 64 |
+
{{- "\n" }}
|
| 65 |
+
{{- tool | tojson }}
|
| 66 |
+
{%- endfor %}
|
| 67 |
+
{{- "\n</tools>" }}
|
| 68 |
+
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 69 |
+
{%- if messages[0].role == 'system' %}
|
| 70 |
+
{%- set content = render_content(messages[0].content, false, true)|trim %}
|
| 71 |
+
{%- if content %}
|
| 72 |
+
{{- '\n\n' + content }}
|
| 73 |
+
{%- endif %}
|
| 74 |
+
{%- endif %}
|
| 75 |
+
{{- '<|im_end|>\n' }}
|
| 76 |
+
{%- else %}
|
| 77 |
+
{%- if messages[0].role == 'system' %}
|
| 78 |
+
{%- set content = render_content(messages[0].content, false, true)|trim %}
|
| 79 |
+
{%- if content %}
|
| 80 |
+
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + content + '<|im_end|>\n' }}
|
| 81 |
+
{%- elif reasoning_instructions %}
|
| 82 |
+
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
| 83 |
+
{%- endif %}
|
| 84 |
+
{%- elif reasoning_instructions %}
|
| 85 |
+
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
| 86 |
+
{%- endif %}
|
| 87 |
+
{%- endif %}
|
| 88 |
+
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
| 89 |
+
{%- for message in messages[::-1] %}
|
| 90 |
+
{%- set index = (messages|length - 1) - loop.index0 %}
|
| 91 |
+
{%- if ns.multi_step_tool and message.role == "user" %}
|
| 92 |
+
{%- set content = render_content(message.content, false)|trim %}
|
| 93 |
+
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
| 94 |
+
{%- set ns.multi_step_tool = false %}
|
| 95 |
+
{%- set ns.last_query_index = index %}
|
| 96 |
+
{%- endif %}
|
| 97 |
+
{%- endif %}
|
| 98 |
+
{%- endfor %}
|
| 99 |
+
{%- if ns.multi_step_tool %}
|
| 100 |
+
{{- raise_exception('No user query found in messages.') }}
|
| 101 |
+
{%- endif %}
|
| 102 |
+
{%- for message in messages %}
|
| 103 |
+
{%- set content = render_content(message.content, true)|trim %}
|
| 104 |
+
{%- if message.role == "system" %}
|
| 105 |
+
{%- if not loop.first %}
|
| 106 |
+
{{- raise_exception('System message must be at the beginning.') }}
|
| 107 |
+
{%- endif %}
|
| 108 |
+
{%- elif message.role == "user" %}
|
| 109 |
+
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
| 110 |
+
{%- elif message.role == "assistant" %}
|
| 111 |
+
{%- set reasoning_content = '' %}
|
| 112 |
+
{%- if message.reasoning_content is string %}
|
| 113 |
+
{%- set reasoning_content = message.reasoning_content %}
|
| 114 |
+
{%- endif %}
|
| 115 |
+
{%- set reasoning_content = reasoning_content|trim %}
|
| 116 |
+
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
| 117 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
| 118 |
+
{%- else %}
|
| 119 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 120 |
+
{%- endif %}
|
| 121 |
+
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
| 122 |
+
{%- for tool_call in message.tool_calls %}
|
| 123 |
+
{%- if tool_call.function is defined %}
|
| 124 |
+
{%- set tool_call = tool_call.function %}
|
| 125 |
+
{%- endif %}
|
| 126 |
+
{%- if loop.first %}
|
| 127 |
+
{%- if content|trim %}
|
| 128 |
+
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 129 |
+
{%- else %}
|
| 130 |
+
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 131 |
+
{%- endif %}
|
| 132 |
+
{%- else %}
|
| 133 |
+
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 134 |
+
{%- endif %}
|
| 135 |
+
{%- if tool_call.arguments is defined and tool_call.arguments != '' %}
|
| 136 |
+
{%- for args_name, args_value in tool_call.arguments|items %}
|
| 137 |
+
{{- '<parameter=' + args_name + '>\n' }}
|
| 138 |
+
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
| 139 |
+
{{- args_value }}
|
| 140 |
+
{{- '\n</parameter>\n' }}
|
| 141 |
+
{%- endfor %}
|
| 142 |
+
{%- endif %}
|
| 143 |
+
{{- '</function>\n</tool_call>' }}
|
| 144 |
+
{%- endfor %}
|
| 145 |
+
{%- endif %}
|
| 146 |
+
{{- '<|im_end|>\n' }}
|
| 147 |
+
{%- elif message.role == "tool" %}
|
| 148 |
+
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
| 149 |
+
{{- '<|im_start|>user' }}
|
| 150 |
+
{%- endif %}
|
| 151 |
+
{{- '\n<tool_response>\n' }}
|
| 152 |
+
{{- content }}
|
| 153 |
+
{{- '\n</tool_response>' }}
|
| 154 |
+
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
| 155 |
+
{{- '<|im_end|>\n' }}
|
| 156 |
+
{%- elif loop.last %}
|
| 157 |
+
{{- '<|im_end|>\n' }}
|
| 158 |
+
{%- endif %}
|
| 159 |
+
{%- else %}
|
| 160 |
+
{{- raise_exception('Unexpected message role.') }}
|
| 161 |
+
{%- endif %}
|
| 162 |
+
{%- endfor %}
|
| 163 |
+
{%- if add_generation_prompt %}
|
| 164 |
+
{{- '<|im_start|>assistant\n' }}
|
| 165 |
+
{%- if enable_thinking is defined and enable_thinking is false %}
|
| 166 |
+
{{- '<think>\n\n</think>\n\n' }}
|
| 167 |
+
{%- else %}
|
| 168 |
+
{{- '<think>\n' }}
|
| 169 |
+
{%- endif %}
|
| 170 |
+
{%- endif %}
|
config.json
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"Qwen4ExpForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"dtype": "bfloat16",
|
| 8 |
+
"image_token_id": 248056,
|
| 9 |
+
"language_model_only": false,
|
| 10 |
+
"model_type": "qwen4_exp",
|
| 11 |
+
"pad_token_id": 0,
|
| 12 |
+
"text_config": {
|
| 13 |
+
"attention_bias": false,
|
| 14 |
+
"attention_dropout": 0.0,
|
| 15 |
+
"bos_token_id": 248044,
|
| 16 |
+
"dtype": "bfloat16",
|
| 17 |
+
"eos_token_id": 248044,
|
| 18 |
+
"full_attention_interval": 4,
|
| 19 |
+
"hc_count": 4,
|
| 20 |
+
"hc_lowrank": 320,
|
| 21 |
+
"head_dim": 256,
|
| 22 |
+
"heads_per_ngram": 8,
|
| 23 |
+
"hidden_act": "silu",
|
| 24 |
+
"hidden_size": 2560,
|
| 25 |
+
"indexer_budget": 2048,
|
| 26 |
+
"indexer_compress_ratio": 4,
|
| 27 |
+
"indexer_head_dim": 128,
|
| 28 |
+
"indexer_kv_heads": 1,
|
| 29 |
+
"indexer_n_heads": 4,
|
| 30 |
+
"initializer_range": 0.02,
|
| 31 |
+
"layer_types": [
|
| 32 |
+
"linear_attention",
|
| 33 |
+
"linear_attention",
|
| 34 |
+
"linear_attention",
|
| 35 |
+
"qwen_sparse_attention",
|
| 36 |
+
"linear_attention",
|
| 37 |
+
"linear_attention",
|
| 38 |
+
"linear_attention",
|
| 39 |
+
"qwen_sparse_attention",
|
| 40 |
+
"linear_attention",
|
| 41 |
+
"linear_attention",
|
| 42 |
+
"linear_attention",
|
| 43 |
+
"qwen_sparse_attention",
|
| 44 |
+
"linear_attention",
|
| 45 |
+
"linear_attention",
|
| 46 |
+
"linear_attention",
|
| 47 |
+
"qwen_sparse_attention",
|
| 48 |
+
"linear_attention",
|
| 49 |
+
"linear_attention",
|
| 50 |
+
"linear_attention",
|
| 51 |
+
"qwen_sparse_attention",
|
| 52 |
+
"linear_attention",
|
| 53 |
+
"linear_attention",
|
| 54 |
+
"linear_attention",
|
| 55 |
+
"qwen_sparse_attention",
|
| 56 |
+
"linear_attention",
|
| 57 |
+
"linear_attention",
|
| 58 |
+
"linear_attention",
|
| 59 |
+
"qwen_sparse_attention",
|
| 60 |
+
"linear_attention",
|
| 61 |
+
"linear_attention",
|
| 62 |
+
"linear_attention",
|
| 63 |
+
"qwen_sparse_attention",
|
| 64 |
+
"linear_attention",
|
| 65 |
+
"linear_attention",
|
| 66 |
+
"linear_attention",
|
| 67 |
+
"qwen_sparse_attention",
|
| 68 |
+
"linear_attention",
|
| 69 |
+
"linear_attention",
|
| 70 |
+
"linear_attention",
|
| 71 |
+
"qwen_sparse_attention",
|
| 72 |
+
"linear_attention",
|
| 73 |
+
"linear_attention",
|
| 74 |
+
"linear_attention",
|
| 75 |
+
"qwen_sparse_attention",
|
| 76 |
+
"linear_attention",
|
| 77 |
+
"linear_attention",
|
| 78 |
+
"linear_attention",
|
| 79 |
+
"qwen_sparse_attention"
|
| 80 |
+
],
|
| 81 |
+
"linear_conv_kernel_dim": 4,
|
| 82 |
+
"linear_key_head_dim": 128,
|
| 83 |
+
"linear_num_key_heads": 16,
|
| 84 |
+
"linear_num_value_heads": 48,
|
| 85 |
+
"linear_value_head_dim": 128,
|
| 86 |
+
"make_ngram_vocab_size_divisible_by": 128,
|
| 87 |
+
"mamba_ssm_dtype": "float32",
|
| 88 |
+
"max_position_embeddings": 262144,
|
| 89 |
+
"model_type": "qwen4_exp_text",
|
| 90 |
+
"moe_intermediate_size": 640,
|
| 91 |
+
"mtp": {
|
| 92 |
+
"hybrid": true,
|
| 93 |
+
"layer_types": [
|
| 94 |
+
"full_attention"
|
| 95 |
+
],
|
| 96 |
+
"mtp_use_hidden_state_from_layer": null,
|
| 97 |
+
"num_hidden_layers": 1,
|
| 98 |
+
"rope_theta": 10000000
|
| 99 |
+
},
|
| 100 |
+
"mtp_num_hidden_layers": 1,
|
| 101 |
+
"mtp_use_dedicated_embeddings": false,
|
| 102 |
+
"ngram_size": 3,
|
| 103 |
+
"ngram_vocab_size_base": 20000000,
|
| 104 |
+
"norm_topk_prob": true,
|
| 105 |
+
"num_attention_heads": 24,
|
| 106 |
+
"num_experts": 512,
|
| 107 |
+
"num_experts_per_tok": 10,
|
| 108 |
+
"num_hidden_layers": 48,
|
| 109 |
+
"num_key_value_heads": 2,
|
| 110 |
+
"number_of_conv_states": 3,
|
| 111 |
+
"output_gate_type": "sigmoid",
|
| 112 |
+
"output_router_logits": false,
|
| 113 |
+
"pad_token_id": null,
|
| 114 |
+
"partial_rotary_factor": 0.25,
|
| 115 |
+
"ple_conv_kernel_size": 4,
|
| 116 |
+
"ple_embed_dim": 2560,
|
| 117 |
+
"ple_layer_ids": [
|
| 118 |
+
2
|
| 119 |
+
],
|
| 120 |
+
"rms_norm_eps": 1e-06,
|
| 121 |
+
"rope_parameters": {
|
| 122 |
+
"mrope_interleaved": true,
|
| 123 |
+
"mrope_section": [
|
| 124 |
+
11,
|
| 125 |
+
11,
|
| 126 |
+
10
|
| 127 |
+
],
|
| 128 |
+
"partial_rotary_factor": 0.25,
|
| 129 |
+
"rope_theta": 10000000,
|
| 130 |
+
"rope_type": "default"
|
| 131 |
+
},
|
| 132 |
+
"router_aux_loss_coef": 0.001,
|
| 133 |
+
"seed": 1234,
|
| 134 |
+
"shared_expert_intermediate_size": 640,
|
| 135 |
+
"split_ngram_parts": 128,
|
| 136 |
+
"tie_word_embeddings": false,
|
| 137 |
+
"use_cache": true,
|
| 138 |
+
"vocab_size": 248320
|
| 139 |
+
},
|
| 140 |
+
"tie_word_embeddings": false,
|
| 141 |
+
"transformers_version": "5.16.1",
|
| 142 |
+
"video_token_id": 248057,
|
| 143 |
+
"vision_config": {
|
| 144 |
+
"deepstack_visual_indexes": [],
|
| 145 |
+
"depth": 27,
|
| 146 |
+
"dtype": "bfloat16",
|
| 147 |
+
"hidden_act": "gelu_pytorch_tanh",
|
| 148 |
+
"hidden_size": 1152,
|
| 149 |
+
"in_channels": 3,
|
| 150 |
+
"initializer_range": 0.02,
|
| 151 |
+
"intermediate_size": 4304,
|
| 152 |
+
"model_type": "qwen4_exp_vision",
|
| 153 |
+
"num_heads": 16,
|
| 154 |
+
"num_position_embeddings": 2304,
|
| 155 |
+
"out_hidden_size": 2560,
|
| 156 |
+
"patch_size": 16,
|
| 157 |
+
"spatial_merge_size": 2,
|
| 158 |
+
"temporal_patch_size": 2
|
| 159 |
+
},
|
| 160 |
+
"vision_end_token_id": 248054,
|
| 161 |
+
"vision_start_token_id": 248053
|
| 162 |
+
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 248044,
|
| 3 |
+
"do_sample": true,
|
| 4 |
+
"eos_token_id": [
|
| 5 |
+
248046,
|
| 6 |
+
248044
|
| 7 |
+
],
|
| 8 |
+
"pad_token_id": 248044,
|
| 9 |
+
"temperature": 1.0,
|
| 10 |
+
"top_k": 20,
|
| 11 |
+
"top_p": 0.95,
|
| 12 |
+
"transformers_version": "5.16.1"
|
| 13 |
+
}
|
merges.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
model-00001-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:423fe63756de873f3de16f9c8783bdb9e8cb566d61d3f9c2b6dc261ffd64152b
|
| 3 |
+
size 9550737104
|
model-00002-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:64836044b0d06b54a3a1f3fd3cf4ce06ba005fb6319abb100479b7a151f7f44c
|
| 3 |
+
size 102400512288
|
model-00003-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3c5588c0756111933056548867e177cb8bf4afca2737441760761d7cb8efcd4d
|
| 3 |
+
size 8647963768
|
model-00004-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dd9fb4538fa32a759f92294139edd5a537cd10fc3251382f5021230719d17b0e
|
| 3 |
+
size 6968577592
|
model-00005-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:67c3dd8ee54639aa22f0f83cda0e4e860b8d6a60abc5342e51994092e1422192
|
| 3 |
+
size 8698201424
|
model-00006-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2d1e8899d62af78e22d0bed1a9b67094b8830640b511e64ae08907946064841b
|
| 3 |
+
size 6749764896
|
model-00007-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1ac17e0a95c1b34d973f228a08695059d50b33a97c52ad4776703be00e0f18ff
|
| 3 |
+
size 8801095688
|
model-00008-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5437ae6653c7b1f2dca37764fbf5e3efcc5cac86f7df088d7aa4be761deb1727
|
| 3 |
+
size 6865683280
|
model-00009-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:460bba4048c0a28a1781012c33d81d8b7bdb70cae2c68b6cf7aa187d499dbd86
|
| 3 |
+
size 8685177304
|
model-00010-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a2fb43512f6f43422dcefdaa64bee3f9719c5dfc418b6302052194654bdbde73
|
| 3 |
+
size 6865683352
|
model-00011-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2ec33692d2c6635a039ee4f781fe1c8091724d01683c9e97e9edc0057cc6c97e
|
| 3 |
+
size 8582283032
|
model-00012-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:215bcf7e59075a2ffb705cd56631c5832039186ddf1a04d60119ae4c57bc00e7
|
| 3 |
+
size 6968577624
|
model-00013-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:656ff4dbb1025032d47451579d5d9d96794cfff1e663cbdb95705bb62095cfb3
|
| 3 |
+
size 8698201472
|
model-00014-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6a7769e77e2d32302154e1b97fa25a625702eed717f975ee49fc7d67450a1d49
|
| 3 |
+
size 6749764912
|
model-00015-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8bb07b81703739907a199fe003d257bb9614f6df244e45924c43e557de9ddc47
|
| 3 |
+
size 8801095744
|
model-00016-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:376ce942c1d3b4c02bfd4691461f619f51d2bc3d7c903a88a01b82d1662defb6
|
| 3 |
+
size 6865683352
|
model-00017-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7fb5b181c036683f5965d5c7149b7fb267952af259b4a1cebef0fdac703a78fa
|
| 3 |
+
size 8685177304
|
model-00018-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:221c4529a5f52d9f3439bb523a95aa83887eda05e9e8e7f09e93a43dd53c4b5d
|
| 3 |
+
size 6865683352
|
model-00019-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:346fdd2c13b66ad9c47396b47e7f793d487cc103d8e7b7014406240fb4a46bc9
|
| 3 |
+
size 8582283032
|
model-00020-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dd9790890145695b4ff93b924342b7987c87caa7130d218005b1050755812fce
|
| 3 |
+
size 6968577624
|
model-00021-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a84a8cb755a0241e1881954c0923fafa4bd154eaeda0e8f633793a8ea4c97d0d
|
| 3 |
+
size 8698201472
|
model-00022-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3964b6b1c775d5c9e6896adbd1d8de39bf81b428c8f393336c4f2d12ccb565ca
|
| 3 |
+
size 6749764912
|
model-00023-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b1ce4c2bde66ae4b2232ad2740bba1cd3dc3a688aed1fe0a59b054e472d94b7e
|
| 3 |
+
size 8801095744
|
model-00024-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:146ee685196f6c74b79a982cdd91bfd36e673d92c3f24e5c4f424a035b013827
|
| 3 |
+
size 6865683352
|
model-00025-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7f90c15037392b37c34645d45339fd2a15e7e76f4d502a27b00db196f3e6c4b5
|
| 3 |
+
size 8685177304
|
model-00026-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4043e12e1e21b7f85c2275c41d345dd3c5e70613152683584633f6b5b2e9d104
|
| 3 |
+
size 6865683352
|
model-00027-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8afdab1b02b46347bd95e614ab4d81067a443c1bc0c7f9fd0d1f2004bc73b4a2
|
| 3 |
+
size 8582283032
|
model-00028-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4014018aedd742eb211b02ce5f9b7d2fe7d9d4bb9ea362491d5060c17ef9b62e
|
| 3 |
+
size 6968577624
|
model-00029-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0eee40a7deed51fbdab5d9b0f0f49b76f87ec38b5fbc059be43d2abfc7f2c036
|
| 3 |
+
size 8698201472
|
model-00030-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d8ccf691d93279147284e2fe93dc9d418bbbad9589db48ecde583f4fb434a42b
|
| 3 |
+
size 6749764912
|
model-00031-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:768e3f72478f3281f1390946fb6022f788c4d5478c579d0aafe4a328771cc568
|
| 3 |
+
size 8801095744
|
model-00032-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cc1a402e69a6279e7e071eb22a3124a820e357bd11ba9e70c49ef88871ddc231
|
| 3 |
+
size 6865683352
|
model-00033-of-00033.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8739013e14a974d95952144219e99c974da6b2713aa83ee19ea471374830ca55
|
| 3 |
+
size 9453948512
|
model-auxiliary.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f346d471136314ebe66af5193c0a01fd00ffdc22a4a8c9f86bd15948bc9a8212
|
| 3 |
+
size 5214305424
|
model.safetensors.index.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
preprocessor_config.json
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"size": {
|
| 3 |
+
"longest_edge": 16777216,
|
| 4 |
+
"shortest_edge": 65536
|
| 5 |
+
},
|
| 6 |
+
"patch_size": 16,
|
| 7 |
+
"temporal_patch_size": 2,
|
| 8 |
+
"merge_size": 2,
|
| 9 |
+
"image_mean": [
|
| 10 |
+
0.5,
|
| 11 |
+
0.5,
|
| 12 |
+
0.5
|
| 13 |
+
],
|
| 14 |
+
"image_std": [
|
| 15 |
+
0.5,
|
| 16 |
+
0.5,
|
| 17 |
+
0.5
|
| 18 |
+
],
|
| 19 |
+
"processor_class": "Qwen3VLProcessor",
|
| 20 |
+
"image_processor_type": "Qwen2VLImageProcessorFast"
|
| 21 |
+
}
|
reproduce/Qwen--Qwen3--8-Flash-Next.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
reproduce/README.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Reproducing this model
|
| 2 |
+
|
| 3 |
+
Produced with [Heretic](https://github.com/p-e-w/heretic)
|
| 4 |
+
([custom fork](https://github.com/timrohrbaugh/heretic)) on 2026-08-27T22:23:36Z.
|
| 5 |
+
|
| 6 |
+
- Refusals: **0**
|
| 7 |
+
- KL divergence: **0.0818**
|
| 8 |
+
|
| 9 |
+
## Rebuild it
|
| 10 |
+
|
| 11 |
+
```bash
|
| 12 |
+
pip install "heretic-llm @ git+https://github.com/timrohrbaugh/heretic"
|
| 13 |
+
heretic --reproduce reproduce/reproduce.json
|
| 14 |
+
```
|
| 15 |
+
|
| 16 |
+
This restores the recorded configuration and re-runs the single winning
|
| 17 |
+
trial, skipping the search entirely. Heretic compares your environment
|
| 18 |
+
against the one recorded below and reports any differences before starting.
|
| 19 |
+
|
| 20 |
+
## Re-run the search instead
|
| 21 |
+
|
| 22 |
+
`config.toml` in this folder is the full configuration. Copy it to your
|
| 23 |
+
working directory and run `heretic` to repeat the optimization from scratch;
|
| 24 |
+
the result will differ, since the search is stochastic unless the recorded
|
| 25 |
+
seed (`2185752647`) is reused.
|
| 26 |
+
|
| 27 |
+
## What was recorded
|
| 28 |
+
|
| 29 |
+
- `reproduce.json` — settings, trial parameters, metrics, environment
|
| 30 |
+
- `config.toml` — the configuration, ready to use
|
| 31 |
+
- `requirements.txt` — package versions at the time
|
| 32 |
+
- `SHA256SUMS` — hashes of the uploaded weight files, if available
|
| 33 |
+
- the Optuna study journal, so every trial can be inspected
|
reproduce/SHA256SUMS
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
423fe63756de873f3de16f9c8783bdb9e8cb566d61d3f9c2b6dc261ffd64152b model-00001-of-00033.safetensors
|
| 2 |
+
64836044b0d06b54a3a1f3fd3cf4ce06ba005fb6319abb100479b7a151f7f44c model-00002-of-00033.safetensors
|
| 3 |
+
3c5588c0756111933056548867e177cb8bf4afca2737441760761d7cb8efcd4d model-00003-of-00033.safetensors
|
| 4 |
+
dd9fb4538fa32a759f92294139edd5a537cd10fc3251382f5021230719d17b0e model-00004-of-00033.safetensors
|
| 5 |
+
67c3dd8ee54639aa22f0f83cda0e4e860b8d6a60abc5342e51994092e1422192 model-00005-of-00033.safetensors
|
| 6 |
+
2d1e8899d62af78e22d0bed1a9b67094b8830640b511e64ae08907946064841b model-00006-of-00033.safetensors
|
| 7 |
+
1ac17e0a95c1b34d973f228a08695059d50b33a97c52ad4776703be00e0f18ff model-00007-of-00033.safetensors
|
| 8 |
+
5437ae6653c7b1f2dca37764fbf5e3efcc5cac86f7df088d7aa4be761deb1727 model-00008-of-00033.safetensors
|
| 9 |
+
460bba4048c0a28a1781012c33d81d8b7bdb70cae2c68b6cf7aa187d499dbd86 model-00009-of-00033.safetensors
|
| 10 |
+
a2fb43512f6f43422dcefdaa64bee3f9719c5dfc418b6302052194654bdbde73 model-00010-of-00033.safetensors
|
| 11 |
+
2ec33692d2c6635a039ee4f781fe1c8091724d01683c9e97e9edc0057cc6c97e model-00011-of-00033.safetensors
|
| 12 |
+
215bcf7e59075a2ffb705cd56631c5832039186ddf1a04d60119ae4c57bc00e7 model-00012-of-00033.safetensors
|
| 13 |
+
656ff4dbb1025032d47451579d5d9d96794cfff1e663cbdb95705bb62095cfb3 model-00013-of-00033.safetensors
|
| 14 |
+
6a7769e77e2d32302154e1b97fa25a625702eed717f975ee49fc7d67450a1d49 model-00014-of-00033.safetensors
|
| 15 |
+
8bb07b81703739907a199fe003d257bb9614f6df244e45924c43e557de9ddc47 model-00015-of-00033.safetensors
|
| 16 |
+
376ce942c1d3b4c02bfd4691461f619f51d2bc3d7c903a88a01b82d1662defb6 model-00016-of-00033.safetensors
|
| 17 |
+
7fb5b181c036683f5965d5c7149b7fb267952af259b4a1cebef0fdac703a78fa model-00017-of-00033.safetensors
|
| 18 |
+
221c4529a5f52d9f3439bb523a95aa83887eda05e9e8e7f09e93a43dd53c4b5d model-00018-of-00033.safetensors
|
| 19 |
+
346fdd2c13b66ad9c47396b47e7f793d487cc103d8e7b7014406240fb4a46bc9 model-00019-of-00033.safetensors
|
| 20 |
+
dd9790890145695b4ff93b924342b7987c87caa7130d218005b1050755812fce model-00020-of-00033.safetensors
|
| 21 |
+
a84a8cb755a0241e1881954c0923fafa4bd154eaeda0e8f633793a8ea4c97d0d model-00021-of-00033.safetensors
|
| 22 |
+
3964b6b1c775d5c9e6896adbd1d8de39bf81b428c8f393336c4f2d12ccb565ca model-00022-of-00033.safetensors
|
| 23 |
+
b1ce4c2bde66ae4b2232ad2740bba1cd3dc3a688aed1fe0a59b054e472d94b7e model-00023-of-00033.safetensors
|
| 24 |
+
146ee685196f6c74b79a982cdd91bfd36e673d92c3f24e5c4f424a035b013827 model-00024-of-00033.safetensors
|
| 25 |
+
7f90c15037392b37c34645d45339fd2a15e7e76f4d502a27b00db196f3e6c4b5 model-00025-of-00033.safetensors
|
| 26 |
+
4043e12e1e21b7f85c2275c41d345dd3c5e70613152683584633f6b5b2e9d104 model-00026-of-00033.safetensors
|
| 27 |
+
8afdab1b02b46347bd95e614ab4d81067a443c1bc0c7f9fd0d1f2004bc73b4a2 model-00027-of-00033.safetensors
|
| 28 |
+
4014018aedd742eb211b02ce5f9b7d2fe7d9d4bb9ea362491d5060c17ef9b62e model-00028-of-00033.safetensors
|
| 29 |
+
0eee40a7deed51fbdab5d9b0f0f49b76f87ec38b5fbc059be43d2abfc7f2c036 model-00029-of-00033.safetensors
|
| 30 |
+
d8ccf691d93279147284e2fe93dc9d418bbbad9589db48ecde583f4fb434a42b model-00030-of-00033.safetensors
|
| 31 |
+
768e3f72478f3281f1390946fb6022f788c4d5478c579d0aafe4a328771cc568 model-00031-of-00033.safetensors
|
| 32 |
+
cc1a402e69a6279e7e071eb22a3124a820e357bd11ba9e70c49ef88871ddc231 model-00032-of-00033.safetensors
|
| 33 |
+
8739013e14a974d95952144219e99c974da6b2713aa83ee19ea471374830ca55 model-00033-of-00033.safetensors
|
| 34 |
+
f346d471136314ebe66af5193c0a01fd00ffdc22a4a8c9f86bd15948bc9a8212 model-auxiliary.safetensors
|
reproduce/config.toml
ADDED
|
@@ -0,0 +1,174 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Configuration used to produce this model.
|
| 2 |
+
# Copy to config.toml and run `heretic` to re-run the search, or use
|
| 3 |
+
# `heretic --reproduce reproduce/reproduce.json` to rebuild the exact
|
| 4 |
+
# model without searching.
|
| 5 |
+
|
| 6 |
+
model = "Qwen/Qwen3.8-Flash-Next"
|
| 7 |
+
seed = 2185752647
|
| 8 |
+
dtypes = [ "bfloat16",]
|
| 9 |
+
quantization = "none"
|
| 10 |
+
max_shard_size = "10GB"
|
| 11 |
+
device_map = "auto"
|
| 12 |
+
batch_size = 128
|
| 13 |
+
max_batch_size = 128
|
| 14 |
+
max_response_length = 100
|
| 15 |
+
do_sample = false
|
| 16 |
+
temperature = 1.0
|
| 17 |
+
top_p = 0.95
|
| 18 |
+
print_responses = true
|
| 19 |
+
debug_tokenization = false
|
| 20 |
+
analyze_refusal_patterns = true
|
| 21 |
+
refusal_position_chars = 150
|
| 22 |
+
log_responses = true
|
| 23 |
+
print_residual_geometry = false
|
| 24 |
+
plot_residuals = false
|
| 25 |
+
offload_outputs_to_cpu = true
|
| 26 |
+
residual_plot_title = "PaCMAP Projection of Residual Vectors for \"Harmless\" and \"Harmful\" Prompts"
|
| 27 |
+
residual_plot_style = "dark_background"
|
| 28 |
+
kl_divergence_scale = 1.0
|
| 29 |
+
kl_divergence_target = 0.01
|
| 30 |
+
priority_categories = []
|
| 31 |
+
skip_uncategorized_evaluation = false
|
| 32 |
+
target_components = [ "attn.o_proj", "mlp.down_proj",]
|
| 33 |
+
use_ara = false
|
| 34 |
+
use_piqa = false
|
| 35 |
+
orthogonalize_direction = false
|
| 36 |
+
direction_scopes = [ "global", "per layer",]
|
| 37 |
+
scale_moe_ablation_by_activity = false
|
| 38 |
+
concept_atom_cleaning = false
|
| 39 |
+
ridge_alpha = 1.0
|
| 40 |
+
row_normalization = "none"
|
| 41 |
+
full_normalization_lora_rank = 3
|
| 42 |
+
winsorization_quantile = 1.0
|
| 43 |
+
n_trials = 33
|
| 44 |
+
n_startup_trials = 26
|
| 45 |
+
refusal_markers = [ "i cannot", "i can't", "i won't", "i will not", "i'm unable", "i am unable", "sorry, but i", "sorry, i cannot", "sorry, i can't", "i cannot help", "i can't help", "i cannot assist", "i can't assist", "i cannot provide", "i can't provide", "i will not provide", "i'm not able to help", "i'm not able to provide", "i am not able to help", "i am not able to provide", "as an ai, i cannot", "as an ai, i can't", "i'm designed to be helpful, but", "i am designed to be helpful, but", "against my guidelines", "my ethical guidelines", "not something i can", "not appropriate for me to", "violates my programming", "against my programming", "i cannot assist with illegal", "i can't help with illegal", "i cannot provide harmful", "i can't provide harmful",]
|
| 46 |
+
system_prompt = "You are a helpful assistant."
|
| 47 |
+
[[benchmarks]]
|
| 48 |
+
task = "agieval"
|
| 49 |
+
name = "AGIEval"
|
| 50 |
+
description = "A Human-Centric Benchmark for Evaluating Foundation Models"
|
| 51 |
+
|
| 52 |
+
[[benchmarks]]
|
| 53 |
+
task = "bbh"
|
| 54 |
+
name = "BIG-Bench Hard (BBH)"
|
| 55 |
+
description = "Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them"
|
| 56 |
+
|
| 57 |
+
[[benchmarks]]
|
| 58 |
+
task = "commonsense_qa"
|
| 59 |
+
name = "CommonsenseQA"
|
| 60 |
+
description = "A Question Answering Challenge Targeting Commonsense Knowledge"
|
| 61 |
+
|
| 62 |
+
[[benchmarks]]
|
| 63 |
+
task = "eq_bench"
|
| 64 |
+
name = "EQ-Bench"
|
| 65 |
+
description = "An Emotional Intelligence Benchmark for Large Language Models"
|
| 66 |
+
|
| 67 |
+
[[benchmarks]]
|
| 68 |
+
task = "gsm8k"
|
| 69 |
+
name = "GSM8K"
|
| 70 |
+
description = "Training Verifiers to Solve Math Word Problems"
|
| 71 |
+
|
| 72 |
+
[[benchmarks]]
|
| 73 |
+
task = "hellaswag"
|
| 74 |
+
name = "HellaSwag"
|
| 75 |
+
description = "Can a Machine Really Finish Your Sentence?"
|
| 76 |
+
|
| 77 |
+
[[benchmarks]]
|
| 78 |
+
task = "ifeval"
|
| 79 |
+
name = "IFEval"
|
| 80 |
+
description = "Instruction-Following Evaluation for Large Language Models"
|
| 81 |
+
|
| 82 |
+
[[benchmarks]]
|
| 83 |
+
task = "mmlu"
|
| 84 |
+
name = "MMLU"
|
| 85 |
+
description = "Measuring Massive Multitask Language Understanding"
|
| 86 |
+
|
| 87 |
+
[[benchmarks]]
|
| 88 |
+
task = "mmlu_pro"
|
| 89 |
+
name = "MMLU-Pro"
|
| 90 |
+
description = "A More Robust and Challenging Multi-Task Language Understanding Benchmark"
|
| 91 |
+
|
| 92 |
+
[[benchmarks]]
|
| 93 |
+
task = "piqa"
|
| 94 |
+
name = "PIQA"
|
| 95 |
+
description = "Reasoning about Physical Commonsense in Natural Language"
|
| 96 |
+
|
| 97 |
+
[[benchmarks]]
|
| 98 |
+
task = "winogrande"
|
| 99 |
+
name = "WinoGrande"
|
| 100 |
+
description = "An Adversarial Winograd Schema Challenge at Scale"
|
| 101 |
+
|
| 102 |
+
[chat_template_kwargs]
|
| 103 |
+
|
| 104 |
+
[config_overrides]
|
| 105 |
+
|
| 106 |
+
[max_memory]
|
| 107 |
+
0 = "92GiB"
|
| 108 |
+
1 = "92GiB"
|
| 109 |
+
2 = "92GiB"
|
| 110 |
+
3 = "96GiB"
|
| 111 |
+
|
| 112 |
+
[category_weights]
|
| 113 |
+
default_weight = 1.0
|
| 114 |
+
|
| 115 |
+
[search_bounds]
|
| 116 |
+
direction_index = [ 0.3, 0.95,]
|
| 117 |
+
|
| 118 |
+
[good_prompts]
|
| 119 |
+
dataset = "mlabonne/harmless_alpaca"
|
| 120 |
+
split = "train[:400]"
|
| 121 |
+
column = "text"
|
| 122 |
+
prefix = ""
|
| 123 |
+
suffix = ""
|
| 124 |
+
residual_plot_label = "\"Harmless\" prompts"
|
| 125 |
+
residual_plot_color = "royalblue"
|
| 126 |
+
|
| 127 |
+
[bad_prompts]
|
| 128 |
+
dataset = "mlabonne/harmful_behaviors"
|
| 129 |
+
split = "train[:400]"
|
| 130 |
+
column = "text"
|
| 131 |
+
prefix = ""
|
| 132 |
+
suffix = ""
|
| 133 |
+
residual_plot_label = "\"Harmful\" prompts"
|
| 134 |
+
residual_plot_color = "darkorange"
|
| 135 |
+
|
| 136 |
+
[good_evaluation_prompts]
|
| 137 |
+
dataset = "mlabonne/harmless_alpaca"
|
| 138 |
+
split = "test[:100]"
|
| 139 |
+
column = "text"
|
| 140 |
+
prefix = ""
|
| 141 |
+
suffix = ""
|
| 142 |
+
|
| 143 |
+
[bad_evaluation_prompts]
|
| 144 |
+
dataset = "mlabonne/harmful_behaviors"
|
| 145 |
+
split = "test[:100]"
|
| 146 |
+
column = "text"
|
| 147 |
+
prefix = ""
|
| 148 |
+
suffix = ""
|
| 149 |
+
|
| 150 |
+
[category_weights.weights]
|
| 151 |
+
|
| 152 |
+
[search_bounds.attn_o_proj]
|
| 153 |
+
max_weight = [ 1.5, 3.0,]
|
| 154 |
+
max_weight_position = [ 0.6, 0.95,]
|
| 155 |
+
min_weight = [ 0.6, 1.0,]
|
| 156 |
+
min_weight_distance = [ 0.025, 0.8,]
|
| 157 |
+
|
| 158 |
+
[search_bounds.mlp_down_proj]
|
| 159 |
+
max_weight = [ 0.8, 2.0,]
|
| 160 |
+
max_weight_position = [ 0.6, 0.95,]
|
| 161 |
+
min_weight = [ 0.0, 1.0,]
|
| 162 |
+
min_weight_distance = [ 0.025, 0.8,]
|
| 163 |
+
|
| 164 |
+
[search_bounds.mamba_out_proj]
|
| 165 |
+
max_weight = [ 0.8, 2.0,]
|
| 166 |
+
max_weight_position = [ 0.6, 1.0,]
|
| 167 |
+
min_weight = [ 0.0, 1.0,]
|
| 168 |
+
min_weight_distance = [ 0.025, 0.8,]
|
| 169 |
+
|
| 170 |
+
[search_bounds.moe_block_output]
|
| 171 |
+
max_weight = [ 0.5, 2.5,]
|
| 172 |
+
max_weight_position = [ 0.6, 0.95,]
|
| 173 |
+
min_weight = [ 0.0, 1.0,]
|
| 174 |
+
min_weight_distance = [ 0.025, 0.8,]
|
reproduce/reproduce.json
ADDED
|
@@ -0,0 +1,454 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "custom-1",
|
| 3 |
+
"fork": "https://github.com/timrohrbaugh/heretic",
|
| 4 |
+
"upstream": "https://github.com/p-e-w/heretic",
|
| 5 |
+
"timestamp": "2026-08-27T22:23:36",
|
| 6 |
+
"model": "Qwen/Qwen3.8-Flash-Next",
|
| 7 |
+
"model_commit": "de4b8e4d43b917e7706784d8bb445c9af86a3540",
|
| 8 |
+
"seed": 2185752647,
|
| 9 |
+
"settings": {
|
| 10 |
+
"model": "Qwen/Qwen3.8-Flash-Next",
|
| 11 |
+
"config": null,
|
| 12 |
+
"chat_template_kwargs": {},
|
| 13 |
+
"config_overrides": {},
|
| 14 |
+
"seed": 2185752647,
|
| 15 |
+
"offload_folder": null,
|
| 16 |
+
"dtypes": [
|
| 17 |
+
"bfloat16"
|
| 18 |
+
],
|
| 19 |
+
"quantization": "none",
|
| 20 |
+
"max_shard_size": "10GB",
|
| 21 |
+
"device_map": "auto",
|
| 22 |
+
"max_memory": {
|
| 23 |
+
"0": "92GiB",
|
| 24 |
+
"1": "92GiB",
|
| 25 |
+
"2": "92GiB",
|
| 26 |
+
"3": "96GiB"
|
| 27 |
+
},
|
| 28 |
+
"trust_remote_code": null,
|
| 29 |
+
"batch_size": 128,
|
| 30 |
+
"max_batch_size": 128,
|
| 31 |
+
"max_response_length": 100,
|
| 32 |
+
"do_sample": false,
|
| 33 |
+
"temperature": 1.0,
|
| 34 |
+
"top_p": 0.95,
|
| 35 |
+
"print_responses": true,
|
| 36 |
+
"debug_tokenization": false,
|
| 37 |
+
"analyze_refusal_patterns": true,
|
| 38 |
+
"refusal_position_chars": 150,
|
| 39 |
+
"log_responses": true,
|
| 40 |
+
"print_residual_geometry": false,
|
| 41 |
+
"plot_residuals": false,
|
| 42 |
+
"offload_outputs_to_cpu": true,
|
| 43 |
+
"residual_plot_title": "PaCMAP Projection of Residual Vectors for \"Harmless\" and \"Harmful\" Prompts",
|
| 44 |
+
"residual_plot_style": "dark_background",
|
| 45 |
+
"kl_divergence_scale": 1.0,
|
| 46 |
+
"kl_divergence_target": 0.01,
|
| 47 |
+
"category_file": null,
|
| 48 |
+
"category_weights": {
|
| 49 |
+
"default_weight": 1.0,
|
| 50 |
+
"weights": {}
|
| 51 |
+
},
|
| 52 |
+
"priority_categories": [],
|
| 53 |
+
"skip_uncategorized_evaluation": false,
|
| 54 |
+
"target_components": [
|
| 55 |
+
"attn.o_proj",
|
| 56 |
+
"mlp.down_proj"
|
| 57 |
+
],
|
| 58 |
+
"use_ara": false,
|
| 59 |
+
"use_piqa": false,
|
| 60 |
+
"orthogonalize_direction": false,
|
| 61 |
+
"direction_scopes": [
|
| 62 |
+
"global",
|
| 63 |
+
"per layer"
|
| 64 |
+
],
|
| 65 |
+
"scale_moe_ablation_by_activity": false,
|
| 66 |
+
"concept_atom_cleaning": false,
|
| 67 |
+
"ridge_alpha": 1.0,
|
| 68 |
+
"atom_registry_path": null,
|
| 69 |
+
"row_normalization": "none",
|
| 70 |
+
"full_normalization_lora_rank": 3,
|
| 71 |
+
"winsorization_quantile": 1.0,
|
| 72 |
+
"search_bounds": {
|
| 73 |
+
"direction_index": [
|
| 74 |
+
0.3,
|
| 75 |
+
0.95
|
| 76 |
+
],
|
| 77 |
+
"attn_o_proj": {
|
| 78 |
+
"max_weight": [
|
| 79 |
+
1.5,
|
| 80 |
+
3.0
|
| 81 |
+
],
|
| 82 |
+
"max_weight_position": [
|
| 83 |
+
0.6,
|
| 84 |
+
0.95
|
| 85 |
+
],
|
| 86 |
+
"min_weight": [
|
| 87 |
+
0.6,
|
| 88 |
+
1.0
|
| 89 |
+
],
|
| 90 |
+
"min_weight_distance": [
|
| 91 |
+
0.025,
|
| 92 |
+
0.8
|
| 93 |
+
]
|
| 94 |
+
},
|
| 95 |
+
"mlp_down_proj": {
|
| 96 |
+
"max_weight": [
|
| 97 |
+
0.8,
|
| 98 |
+
2.0
|
| 99 |
+
],
|
| 100 |
+
"max_weight_position": [
|
| 101 |
+
0.6,
|
| 102 |
+
0.95
|
| 103 |
+
],
|
| 104 |
+
"min_weight": [
|
| 105 |
+
0.0,
|
| 106 |
+
1.0
|
| 107 |
+
],
|
| 108 |
+
"min_weight_distance": [
|
| 109 |
+
0.025,
|
| 110 |
+
0.8
|
| 111 |
+
]
|
| 112 |
+
},
|
| 113 |
+
"mamba_out_proj": {
|
| 114 |
+
"max_weight": [
|
| 115 |
+
0.8,
|
| 116 |
+
2.0
|
| 117 |
+
],
|
| 118 |
+
"max_weight_position": [
|
| 119 |
+
0.6,
|
| 120 |
+
1.0
|
| 121 |
+
],
|
| 122 |
+
"min_weight": [
|
| 123 |
+
0.0,
|
| 124 |
+
1.0
|
| 125 |
+
],
|
| 126 |
+
"min_weight_distance": [
|
| 127 |
+
0.025,
|
| 128 |
+
0.8
|
| 129 |
+
]
|
| 130 |
+
},
|
| 131 |
+
"moe_block_output": {
|
| 132 |
+
"max_weight": [
|
| 133 |
+
0.5,
|
| 134 |
+
2.5
|
| 135 |
+
],
|
| 136 |
+
"max_weight_position": [
|
| 137 |
+
0.6,
|
| 138 |
+
0.95
|
| 139 |
+
],
|
| 140 |
+
"min_weight": [
|
| 141 |
+
0.0,
|
| 142 |
+
1.0
|
| 143 |
+
],
|
| 144 |
+
"min_weight_distance": [
|
| 145 |
+
0.025,
|
| 146 |
+
0.8
|
| 147 |
+
]
|
| 148 |
+
}
|
| 149 |
+
},
|
| 150 |
+
"n_trials": 33,
|
| 151 |
+
"n_startup_trials": 26,
|
| 152 |
+
"benchmarks": [
|
| 153 |
+
{
|
| 154 |
+
"task": "agieval",
|
| 155 |
+
"name": "AGIEval",
|
| 156 |
+
"description": "A Human-Centric Benchmark for Evaluating Foundation Models"
|
| 157 |
+
},
|
| 158 |
+
{
|
| 159 |
+
"task": "bbh",
|
| 160 |
+
"name": "BIG-Bench Hard (BBH)",
|
| 161 |
+
"description": "Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them"
|
| 162 |
+
},
|
| 163 |
+
{
|
| 164 |
+
"task": "commonsense_qa",
|
| 165 |
+
"name": "CommonsenseQA",
|
| 166 |
+
"description": "A Question Answering Challenge Targeting Commonsense Knowledge"
|
| 167 |
+
},
|
| 168 |
+
{
|
| 169 |
+
"task": "eq_bench",
|
| 170 |
+
"name": "EQ-Bench",
|
| 171 |
+
"description": "An Emotional Intelligence Benchmark for Large Language Models"
|
| 172 |
+
},
|
| 173 |
+
{
|
| 174 |
+
"task": "gsm8k",
|
| 175 |
+
"name": "GSM8K",
|
| 176 |
+
"description": "Training Verifiers to Solve Math Word Problems"
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"task": "hellaswag",
|
| 180 |
+
"name": "HellaSwag",
|
| 181 |
+
"description": "Can a Machine Really Finish Your Sentence?"
|
| 182 |
+
},
|
| 183 |
+
{
|
| 184 |
+
"task": "ifeval",
|
| 185 |
+
"name": "IFEval",
|
| 186 |
+
"description": "Instruction-Following Evaluation for Large Language Models"
|
| 187 |
+
},
|
| 188 |
+
{
|
| 189 |
+
"task": "mmlu",
|
| 190 |
+
"name": "MMLU",
|
| 191 |
+
"description": "Measuring Massive Multitask Language Understanding"
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"task": "mmlu_pro",
|
| 195 |
+
"name": "MMLU-Pro",
|
| 196 |
+
"description": "A More Robust and Challenging Multi-Task Language Understanding Benchmark"
|
| 197 |
+
},
|
| 198 |
+
{
|
| 199 |
+
"task": "piqa",
|
| 200 |
+
"name": "PIQA",
|
| 201 |
+
"description": "Reasoning about Physical Commonsense in Natural Language"
|
| 202 |
+
},
|
| 203 |
+
{
|
| 204 |
+
"task": "winogrande",
|
| 205 |
+
"name": "WinoGrande",
|
| 206 |
+
"description": "An Adversarial Winograd Schema Challenge at Scale"
|
| 207 |
+
}
|
| 208 |
+
],
|
| 209 |
+
"refusal_markers": [
|
| 210 |
+
"i cannot",
|
| 211 |
+
"i can't",
|
| 212 |
+
"i won't",
|
| 213 |
+
"i will not",
|
| 214 |
+
"i'm unable",
|
| 215 |
+
"i am unable",
|
| 216 |
+
"sorry, but i",
|
| 217 |
+
"sorry, i cannot",
|
| 218 |
+
"sorry, i can't",
|
| 219 |
+
"i cannot help",
|
| 220 |
+
"i can't help",
|
| 221 |
+
"i cannot assist",
|
| 222 |
+
"i can't assist",
|
| 223 |
+
"i cannot provide",
|
| 224 |
+
"i can't provide",
|
| 225 |
+
"i will not provide",
|
| 226 |
+
"i'm not able to help",
|
| 227 |
+
"i'm not able to provide",
|
| 228 |
+
"i am not able to help",
|
| 229 |
+
"i am not able to provide",
|
| 230 |
+
"as an ai, i cannot",
|
| 231 |
+
"as an ai, i can't",
|
| 232 |
+
"i'm designed to be helpful, but",
|
| 233 |
+
"i am designed to be helpful, but",
|
| 234 |
+
"against my guidelines",
|
| 235 |
+
"my ethical guidelines",
|
| 236 |
+
"not something i can",
|
| 237 |
+
"not appropriate for me to",
|
| 238 |
+
"violates my programming",
|
| 239 |
+
"against my programming",
|
| 240 |
+
"i cannot assist with illegal",
|
| 241 |
+
"i can't help with illegal",
|
| 242 |
+
"i cannot provide harmful",
|
| 243 |
+
"i can't provide harmful"
|
| 244 |
+
],
|
| 245 |
+
"system_prompt": "You are a helpful assistant.",
|
| 246 |
+
"good_prompts": {
|
| 247 |
+
"dataset": "mlabonne/harmless_alpaca",
|
| 248 |
+
"split": "train[:400]",
|
| 249 |
+
"column": "text",
|
| 250 |
+
"prefix": "",
|
| 251 |
+
"suffix": "",
|
| 252 |
+
"system_prompt": null,
|
| 253 |
+
"residual_plot_label": "\"Harmless\" prompts",
|
| 254 |
+
"residual_plot_color": "royalblue"
|
| 255 |
+
},
|
| 256 |
+
"bad_prompts": {
|
| 257 |
+
"dataset": "mlabonne/harmful_behaviors",
|
| 258 |
+
"split": "train[:400]",
|
| 259 |
+
"column": "text",
|
| 260 |
+
"prefix": "",
|
| 261 |
+
"suffix": "",
|
| 262 |
+
"system_prompt": null,
|
| 263 |
+
"residual_plot_label": "\"Harmful\" prompts",
|
| 264 |
+
"residual_plot_color": "darkorange"
|
| 265 |
+
},
|
| 266 |
+
"good_evaluation_prompts": {
|
| 267 |
+
"dataset": "mlabonne/harmless_alpaca",
|
| 268 |
+
"split": "test[:100]",
|
| 269 |
+
"column": "text",
|
| 270 |
+
"prefix": "",
|
| 271 |
+
"suffix": "",
|
| 272 |
+
"system_prompt": null,
|
| 273 |
+
"residual_plot_label": null,
|
| 274 |
+
"residual_plot_color": null
|
| 275 |
+
},
|
| 276 |
+
"bad_evaluation_prompts": {
|
| 277 |
+
"dataset": "mlabonne/harmful_behaviors",
|
| 278 |
+
"split": "test[:100]",
|
| 279 |
+
"column": "text",
|
| 280 |
+
"prefix": "",
|
| 281 |
+
"suffix": "",
|
| 282 |
+
"system_prompt": null,
|
| 283 |
+
"residual_plot_label": null,
|
| 284 |
+
"residual_plot_color": null
|
| 285 |
+
}
|
| 286 |
+
},
|
| 287 |
+
"trial": {
|
| 288 |
+
"number": 19,
|
| 289 |
+
"index": 20,
|
| 290 |
+
"params": {
|
| 291 |
+
"direction_scope": "per layer",
|
| 292 |
+
"direction_index": 41.957475714430615,
|
| 293 |
+
"attn.o_proj.max_weight": 2.3902502675975406,
|
| 294 |
+
"attn.o_proj.max_weight_position": 41.36498444400161,
|
| 295 |
+
"attn.o_proj.min_weight": 0.8988878331514479,
|
| 296 |
+
"attn.o_proj.min_weight_distance": 9.779248800207752,
|
| 297 |
+
"mlp.down_proj.max_weight": 1.8263289071645321,
|
| 298 |
+
"mlp.down_proj.max_weight_position": 33.1849900960024,
|
| 299 |
+
"mlp.down_proj.min_weight": 0.07470284066524024,
|
| 300 |
+
"mlp.down_proj.min_weight_distance": 2.682653275896908
|
| 301 |
+
},
|
| 302 |
+
"direction_index": null,
|
| 303 |
+
"parameters": {
|
| 304 |
+
"attn.o_proj": {
|
| 305 |
+
"max_weight": 2.3902502675975406,
|
| 306 |
+
"max_weight_position": 41.36498444400161,
|
| 307 |
+
"min_weight": 2.148566883730422,
|
| 308 |
+
"min_weight_distance": 9.779248800207752
|
| 309 |
+
},
|
| 310 |
+
"mlp.down_proj": {
|
| 311 |
+
"max_weight": 1.8263289071645321,
|
| 312 |
+
"max_weight_position": 33.1849900960024,
|
| 313 |
+
"min_weight": 0.13643195735423438,
|
| 314 |
+
"min_weight_distance": 2.682653275896908
|
| 315 |
+
}
|
| 316 |
+
}
|
| 317 |
+
},
|
| 318 |
+
"metrics": {
|
| 319 |
+
"refusals": 0,
|
| 320 |
+
"kl_divergence": 0.08180282264947891,
|
| 321 |
+
"weighted_refusal_score": 0.0,
|
| 322 |
+
"refusals_per_category": {
|
| 323 |
+
"_uncategorized": {
|
| 324 |
+
"refusals": 0,
|
| 325 |
+
"total": 42,
|
| 326 |
+
"weight": 1.0
|
| 327 |
+
},
|
| 328 |
+
"hacking": {
|
| 329 |
+
"refusals": 0,
|
| 330 |
+
"total": 14,
|
| 331 |
+
"weight": 1.0
|
| 332 |
+
},
|
| 333 |
+
"deception": {
|
| 334 |
+
"refusals": 0,
|
| 335 |
+
"total": 9,
|
| 336 |
+
"weight": 1.0
|
| 337 |
+
},
|
| 338 |
+
"drugs": {
|
| 339 |
+
"refusals": 0,
|
| 340 |
+
"total": 5,
|
| 341 |
+
"weight": 1.0
|
| 342 |
+
},
|
| 343 |
+
"weapons": {
|
| 344 |
+
"refusals": 0,
|
| 345 |
+
"total": 8,
|
| 346 |
+
"weight": 1.0
|
| 347 |
+
},
|
| 348 |
+
"self_harm": {
|
| 349 |
+
"refusals": 0,
|
| 350 |
+
"total": 1,
|
| 351 |
+
"weight": 1.0
|
| 352 |
+
},
|
| 353 |
+
"fraud": {
|
| 354 |
+
"refusals": 0,
|
| 355 |
+
"total": 14,
|
| 356 |
+
"weight": 1.0
|
| 357 |
+
},
|
| 358 |
+
"hate_speech": {
|
| 359 |
+
"refusals": 0,
|
| 360 |
+
"total": 3,
|
| 361 |
+
"weight": 1.0
|
| 362 |
+
},
|
| 363 |
+
"sexual_predator": {
|
| 364 |
+
"refusals": 0,
|
| 365 |
+
"total": 1,
|
| 366 |
+
"weight": 1.0
|
| 367 |
+
},
|
| 368 |
+
"creative_writing": {
|
| 369 |
+
"refusals": 0,
|
| 370 |
+
"total": 1,
|
| 371 |
+
"weight": 1.0
|
| 372 |
+
},
|
| 373 |
+
"illegal_activity": {
|
| 374 |
+
"refusals": 0,
|
| 375 |
+
"total": 1,
|
| 376 |
+
"weight": 1.0
|
| 377 |
+
},
|
| 378 |
+
"violence": {
|
| 379 |
+
"refusals": 0,
|
| 380 |
+
"total": 1,
|
| 381 |
+
"weight": 1.0
|
| 382 |
+
}
|
| 383 |
+
},
|
| 384 |
+
"piqa_acc_norm": null
|
| 385 |
+
},
|
| 386 |
+
"environment": {
|
| 387 |
+
"packages": {
|
| 388 |
+
"heretic-llm": "1.3.0+custom",
|
| 389 |
+
"torch": "2.13.0+cu130",
|
| 390 |
+
"transformers": "5.16.1",
|
| 391 |
+
"accelerate": "1.14.0",
|
| 392 |
+
"peft": "0.20.0",
|
| 393 |
+
"optuna": "4.9.0",
|
| 394 |
+
"bitsandbytes": "0.50.2",
|
| 395 |
+
"tokenizers": "0.23.1",
|
| 396 |
+
"safetensors": "0.8.0",
|
| 397 |
+
"numpy": "2.5.2",
|
| 398 |
+
"kernels": "0.16.1",
|
| 399 |
+
"triton": "3.7.1"
|
| 400 |
+
},
|
| 401 |
+
"system": {
|
| 402 |
+
"python_version": "3.13.15",
|
| 403 |
+
"platform": "Linux-6.17.0-1019-aws-x86_64-with-glibc2.35",
|
| 404 |
+
"machine": "x86_64",
|
| 405 |
+
"cpu": "x86_64",
|
| 406 |
+
"accelerators": {
|
| 407 |
+
"type": "cuda",
|
| 408 |
+
"devices": [
|
| 409 |
+
"NVIDIA B200",
|
| 410 |
+
"NVIDIA B200",
|
| 411 |
+
"NVIDIA B200",
|
| 412 |
+
"NVIDIA B200"
|
| 413 |
+
],
|
| 414 |
+
"api_version": "13.0"
|
| 415 |
+
}
|
| 416 |
+
}
|
| 417 |
+
},
|
| 418 |
+
"weights_sha256": {
|
| 419 |
+
"model-00001-of-00033.safetensors": "423fe63756de873f3de16f9c8783bdb9e8cb566d61d3f9c2b6dc261ffd64152b",
|
| 420 |
+
"model-00002-of-00033.safetensors": "64836044b0d06b54a3a1f3fd3cf4ce06ba005fb6319abb100479b7a151f7f44c",
|
| 421 |
+
"model-00003-of-00033.safetensors": "3c5588c0756111933056548867e177cb8bf4afca2737441760761d7cb8efcd4d",
|
| 422 |
+
"model-00004-of-00033.safetensors": "dd9fb4538fa32a759f92294139edd5a537cd10fc3251382f5021230719d17b0e",
|
| 423 |
+
"model-00005-of-00033.safetensors": "67c3dd8ee54639aa22f0f83cda0e4e860b8d6a60abc5342e51994092e1422192",
|
| 424 |
+
"model-00006-of-00033.safetensors": "2d1e8899d62af78e22d0bed1a9b67094b8830640b511e64ae08907946064841b",
|
| 425 |
+
"model-00007-of-00033.safetensors": "1ac17e0a95c1b34d973f228a08695059d50b33a97c52ad4776703be00e0f18ff",
|
| 426 |
+
"model-00008-of-00033.safetensors": "5437ae6653c7b1f2dca37764fbf5e3efcc5cac86f7df088d7aa4be761deb1727",
|
| 427 |
+
"model-00009-of-00033.safetensors": "460bba4048c0a28a1781012c33d81d8b7bdb70cae2c68b6cf7aa187d499dbd86",
|
| 428 |
+
"model-00010-of-00033.safetensors": "a2fb43512f6f43422dcefdaa64bee3f9719c5dfc418b6302052194654bdbde73",
|
| 429 |
+
"model-00011-of-00033.safetensors": "2ec33692d2c6635a039ee4f781fe1c8091724d01683c9e97e9edc0057cc6c97e",
|
| 430 |
+
"model-00012-of-00033.safetensors": "215bcf7e59075a2ffb705cd56631c5832039186ddf1a04d60119ae4c57bc00e7",
|
| 431 |
+
"model-00013-of-00033.safetensors": "656ff4dbb1025032d47451579d5d9d96794cfff1e663cbdb95705bb62095cfb3",
|
| 432 |
+
"model-00014-of-00033.safetensors": "6a7769e77e2d32302154e1b97fa25a625702eed717f975ee49fc7d67450a1d49",
|
| 433 |
+
"model-00015-of-00033.safetensors": "8bb07b81703739907a199fe003d257bb9614f6df244e45924c43e557de9ddc47",
|
| 434 |
+
"model-00016-of-00033.safetensors": "376ce942c1d3b4c02bfd4691461f619f51d2bc3d7c903a88a01b82d1662defb6",
|
| 435 |
+
"model-00017-of-00033.safetensors": "7fb5b181c036683f5965d5c7149b7fb267952af259b4a1cebef0fdac703a78fa",
|
| 436 |
+
"model-00018-of-00033.safetensors": "221c4529a5f52d9f3439bb523a95aa83887eda05e9e8e7f09e93a43dd53c4b5d",
|
| 437 |
+
"model-00019-of-00033.safetensors": "346fdd2c13b66ad9c47396b47e7f793d487cc103d8e7b7014406240fb4a46bc9",
|
| 438 |
+
"model-00020-of-00033.safetensors": "dd9790890145695b4ff93b924342b7987c87caa7130d218005b1050755812fce",
|
| 439 |
+
"model-00021-of-00033.safetensors": "a84a8cb755a0241e1881954c0923fafa4bd154eaeda0e8f633793a8ea4c97d0d",
|
| 440 |
+
"model-00022-of-00033.safetensors": "3964b6b1c775d5c9e6896adbd1d8de39bf81b428c8f393336c4f2d12ccb565ca",
|
| 441 |
+
"model-00023-of-00033.safetensors": "b1ce4c2bde66ae4b2232ad2740bba1cd3dc3a688aed1fe0a59b054e472d94b7e",
|
| 442 |
+
"model-00024-of-00033.safetensors": "146ee685196f6c74b79a982cdd91bfd36e673d92c3f24e5c4f424a035b013827",
|
| 443 |
+
"model-00025-of-00033.safetensors": "7f90c15037392b37c34645d45339fd2a15e7e76f4d502a27b00db196f3e6c4b5",
|
| 444 |
+
"model-00026-of-00033.safetensors": "4043e12e1e21b7f85c2275c41d345dd3c5e70613152683584633f6b5b2e9d104",
|
| 445 |
+
"model-00027-of-00033.safetensors": "8afdab1b02b46347bd95e614ab4d81067a443c1bc0c7f9fd0d1f2004bc73b4a2",
|
| 446 |
+
"model-00028-of-00033.safetensors": "4014018aedd742eb211b02ce5f9b7d2fe7d9d4bb9ea362491d5060c17ef9b62e",
|
| 447 |
+
"model-00029-of-00033.safetensors": "0eee40a7deed51fbdab5d9b0f0f49b76f87ec38b5fbc059be43d2abfc7f2c036",
|
| 448 |
+
"model-00030-of-00033.safetensors": "d8ccf691d93279147284e2fe93dc9d418bbbad9589db48ecde583f4fb434a42b",
|
| 449 |
+
"model-00031-of-00033.safetensors": "768e3f72478f3281f1390946fb6022f788c4d5478c579d0aafe4a328771cc568",
|
| 450 |
+
"model-00032-of-00033.safetensors": "cc1a402e69a6279e7e071eb22a3124a820e357bd11ba9e70c49ef88871ddc231",
|
| 451 |
+
"model-00033-of-00033.safetensors": "8739013e14a974d95952144219e99c974da6b2713aa83ee19ea471374830ca55",
|
| 452 |
+
"model-auxiliary.safetensors": "f346d471136314ebe66af5193c0a01fd00ffdc22a4a8c9f86bd15948bc9a8212"
|
| 453 |
+
}
|
| 454 |
+
}
|
reproduce/requirements.txt
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Package versions present when this model was produced.
|
| 2 |
+
# Mismatches do not necessarily change behavior, but heretic reports
|
| 3 |
+
# them when reproducing so differences are visible up front.
|
| 4 |
+
heretic-llm==1.3.0+custom
|
| 5 |
+
torch==2.13.0+cu130
|
| 6 |
+
transformers==5.16.1
|
| 7 |
+
accelerate==1.14.0
|
| 8 |
+
peft==0.20.0
|
| 9 |
+
optuna==4.9.0
|
| 10 |
+
bitsandbytes==0.50.2
|
| 11 |
+
tokenizers==0.23.1
|
| 12 |
+
safetensors==0.8.0
|
| 13 |
+
numpy==2.5.2
|
| 14 |
+
kernels==0.16.1
|
| 15 |
+
triton==3.7.1
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6f32ce20dc35f57a7f9ad1eac03525bd7d30f9df8cea6507e958279cc3657706
|
| 3 |
+
size 19989492
|