|
|
---
|
|
|
license: mit
|
|
|
language:
|
|
|
- en
|
|
|
- zh
|
|
|
base_model:
|
|
|
- zai-org/GLM-4-9B-0414
|
|
|
pipeline_tag: image-text-to-text
|
|
|
library_name: transformers
|
|
|
tags:
|
|
|
- reasoning
|
|
|
---
|
|
|
|
|
|
# GLM-4.1V-9B-Thinking
|
|
|
|
|
|
<div align="center">
|
|
|
<img src=https://raw.githubusercontent.com/zai-org/GLM-4.1V-Thinking/99c5eb6563236f0ff43605d91d107544da9863b2/resources/logo.svg width="40%"/>
|
|
|
</div>
|
|
|
<p align="center">
|
|
|
📖 View the GLM-4.1V-9B-Thinking <a href="https://arxiv.org/abs/2507.01006" target="_blank">paper</a>.
|
|
|
<br>
|
|
|
📍 Using GLM-4.1V-9B-Thinking API at <a href="https://www.bigmodel.cn/dev/api/visual-reasoning-model/GLM-4.1V-Thinking">Zhipu Foundation Model Open Platform</a>
|
|
|
</p>
|
|
|
|
|
|
|
|
|
## Model Introduction
|
|
|
|
|
|
Vision-Language Models (VLMs) have become foundational components of intelligent systems. As real-world AI tasks grow
|
|
|
increasingly complex, VLMs must evolve beyond basic multimodal perception to enhance their reasoning capabilities in
|
|
|
complex tasks. This involves improving accuracy, comprehensiveness, and intelligence, enabling applications such as
|
|
|
complex problem solving, long-context understanding, and multimodal agents.
|
|
|
|
|
|
Based on the [GLM-4-9B-0414](https://github.com/zai-org/GLM-4) foundation model, we present the new open-source VLM model
|
|
|
**GLM-4.1V-9B-Thinking**, designed to explore the upper limits of reasoning in vision-language models. By introducing
|
|
|
a "thinking paradigm" and leveraging reinforcement learning, the model significantly enhances its capabilities. It
|
|
|
achieves state-of-the-art performance among 10B-parameter VLMs, matching or even surpassing the 72B-parameter
|
|
|
Qwen-2.5-VL-72B on 18 benchmark tasks. We are also open-sourcing the base model GLM-4.1V-9B-Base to
|
|
|
support further research into the boundaries of VLM capabilities.
|
|
|
|
|
|
|
|
|
|
|
|
Compared to the previous generation models CogVLM2 and the GLM-4V series, **GLM-4.1V-Thinking** offers the
|
|
|
following improvements:
|
|
|
|
|
|
1. The first reasoning-focused model in the series, achieving world-leading performance not only in mathematics but also
|
|
|
across various sub-domains.
|
|
|
2. Supports **64k** context length.
|
|
|
3. Handles **arbitrary aspect ratios** and up to **4K** image resolution.
|
|
|
4. Provides an open-source version supporting both **Chinese and English bilingual** usage.
|
|
|
|
|
|
## Benchmark Performance
|
|
|
|
|
|
By incorporating the Chain-of-Thought reasoning paradigm, GLM-4.1V-9B-Thinking significantly improves answer accuracy,
|
|
|
richness, and interpretability. It comprehensively surpasses traditional non-reasoning visual models.
|
|
|
Out of 28 benchmark tasks, it achieved the best performance among 10B-level models on 23 tasks,
|
|
|
and even outperformed the 72B-parameter Qwen-2.5-VL-72B on 18 tasks.
|
|
|
|
|
|
|
|
|
|
|
|
## Quick Inference
|
|
|
|
|
|
This is a simple example of running single-image inference using the `transformers` library.
|
|
|
First, install the `transformers` library from source:
|
|
|
|
|
|
```
|
|
|
pip install transformers>=4.57.1
|
|
|
```
|
|
|
|
|
|
Then, run the following code:
|
|
|
|
|
|
```python
|
|
|
from transformers import AutoProcessor, Glm4vForConditionalGeneration
|
|
|
import torch
|
|
|
|
|
|
MODEL_PATH = "zai-org/GLM-4.1V-9B-Thinking"
|
|
|
messages = [
|
|
|
{
|
|
|
"role": "user",
|
|
|
"content": [
|
|
|
{
|
|
|
"type": "image",
|
|
|
"url": "https://upload.wikimedia.org/wikipedia/commons/f/fa/Grayscale_8bits_palette_sample_image.png"
|
|
|
},
|
|
|
{
|
|
|
"type": "text",
|
|
|
"text": "describe this image"
|
|
|
}
|
|
|
],
|
|
|
}
|
|
|
]
|
|
|
processor = AutoProcessor.from_pretrained(MODEL_PATH, use_fast=True)
|
|
|
model = Glm4vForConditionalGeneration.from_pretrained(
|
|
|
pretrained_model_name_or_path=MODEL_PATH,
|
|
|
torch_dtype=torch.bfloat16,
|
|
|
device_map="auto",
|
|
|
)
|
|
|
inputs = processor.apply_chat_template(
|
|
|
messages,
|
|
|
tokenize=True,
|
|
|
add_generation_prompt=True,
|
|
|
return_dict=True,
|
|
|
return_tensors="pt"
|
|
|
).to(model.device)
|
|
|
generated_ids = model.generate(**inputs, max_new_tokens=8192)
|
|
|
output_text = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
|
|
|
print(output_text)
|
|
|
```
|
|
|
|
|
|
For video reasoning, web demo deployment, and more code, please check
|
|
|
our [GitHub](https://github.com/zai-org/GLM-V). |
