Instructions to use UCSC-VLAA/openvision2-vit-huge-patch14-336-vision-only with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- OpenCLIP
How to use UCSC-VLAA/openvision2-vit-huge-patch14-336-vision-only with OpenCLIP:
import open_clip model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:UCSC-VLAA/openvision2-vit-huge-patch14-336-vision-only') tokenizer = open_clip.get_tokenizer('hf-hub:UCSC-VLAA/openvision2-vit-huge-patch14-336-vision-only') - Notebooks
- Google Colab
- Kaggle
Add model card for OpenVision 2
Browse filesThis PR adds a comprehensive model card for the OpenVision 2 model.
It includes:
- The `pipeline_tag: image-text-to-text` to ensure proper discoverability on the Hugging Face Hub.
- A clear description of the model based on its abstract.
- A link to its associated paper [OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning](https://huggingface.co/papers/2509.01644).
- Links to the project page (https://ucsc-vlaa.github.io/OpenVision2/) and the GitHub repository (https://github.com/UCSC-VLAA/OpenVision/blob/main/src/main_openvision2.py) for further details and code.
Please review and merge this PR if everything looks good!
README.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
pipeline_tag: image-text-to-text
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
# OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
|
| 6 |
+
|
| 7 |
+
This repository contains the OpenVision 2 model, a generative pretrained visual encoder for multimodal learning, as presented in the paper [OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning](https://huggingface.co/papers/2509.01644).
|
| 8 |
+
|
| 9 |
+
## Abstract
|
| 10 |
+
|
| 11 |
+
This paper provides a simplification on OpenVision's architecture and loss design for enhancing its training efficiency. Following the prior vision-language pretraining works CapPa and AIMv2, as well as modern multimodal designs like LLaVA, our changes are straightforward: we remove the text encoder (and therefore the contrastive loss), retaining only the captioning loss as a purely generative training signal. We name this new version OpenVision 2. The initial results are promising: despite this simplification, OpenVision 2 competitively matches the original model's performance on a broad set of multimodal benchmarks while substantially cutting both training time and memory consumption. For example, with ViT-L/14, it reduces training time by about 1.5x (from 83h to 57h), and memory usage by about 1.8x (from 24.5GB to 13.8GB, equivalently allowing the maximum batch size to grow from 2k to 8k). This superior training efficiency also allows us to scale far beyond the largest vision encoder used in OpenVision, reaching more than 1 billion parameters. We hold a strong belief that this lightweight, generative-only paradigm is compelling for future vision encoder development in multimodal foundation models.
|
| 12 |
+
|
| 13 |
+
## Project Page
|
| 14 |
+
|
| 15 |
+
[https://ucsc-vlaa.github.io/OpenVision2/](https://ucsc-vlaa.github.io/OpenVision2/)
|
| 16 |
+
|
| 17 |
+
## Code
|
| 18 |
+
|
| 19 |
+
The source code for OpenVision 2 can be found on the [GitHub repository](https://github.com/UCSC-VLAA/OpenVision/blob/main/src/main_openvision2.py).
|