--- license: apache-2.0 base_model: StarVLA/VLAct_Qwen3_Pretrain tags: - robotics - vision-language-action - vlact - starvla - qwen3-vl - fine-tuning - pi - flow-matching - robotwin - agilex ---

VLAct overview: representation-centric continued pre-training for vision-language-action models

# VLAct · Qwen3-VL-4B PI · RoboTwin 2.0 (Clean) [![Paper](https://img.shields.io/badge/Paper-PDF-b31b1b.svg)](https://starvla.github.io/VLAct/assets/VLAct.pdf) [![Project Page](https://img.shields.io/badge/Project-Page-1f6feb.svg)](https://starvla.github.io/VLAct/) [![Code](https://img.shields.io/badge/Code-GitHub-181717.svg)](https://github.com/starVLA/VLAct) [![Models](https://img.shields.io/badge/%F0%9F%A4%97-VLAct_Collection-ffcc4d.svg)](https://huggingface.co/collections/StarVLA/vlact-6a903c2e0c176179da425c96) [![Pretrain](https://img.shields.io/badge/%F0%9F%A4%97-VLAct_Pretrain-ffcc4d.svg)](https://huggingface.co/StarVLA/VLAct_Qwen3_Pretrain) This repository contains the **50K-step VLAct downstream fine-tuning checkpoint** for **RoboTwin 2.0 Clean**, introduced in *Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models*. It starts from [`StarVLA/VLAct_Qwen3_Pretrain`](https://huggingface.co/StarVLA/VLAct_Qwen3_Pretrain) and adapts the shared VLAct backbone with a randomly initialized **PI (`QwenPI_v4`)** action head for the target benchmark. > [!IMPORTANT] > This is a **StarVLA training / evaluation checkpoint**, not a standard > `transformers.AutoModel` package. Load it with the matching StarVLA framework > (`QwenPI_v4`) and the packaged `config.yaml` / `dataset_statistics.json`. Safe robot > deployment still requires embodiment-specific action mapping, normalization, camera calibration, > control-rate handling, workspace constraints, and independent safety systems. ## What is VLAct? VLAct is a representation-centric continued-pretraining recipe for vision-language-action models. It preserves the VLM prior, co-trains multiple continuous action heads on a shared latent, and shares action semantics across embodiments with a partially unified padded layout and wrap-aware joint loss. Downstream policies discard the pretraining heads, randomly initialize a target action head, and fine-tune from the VLAct backbone. The complete method, ablations, and evaluation protocols are documented in the [paper](https://starvla.github.io/VLAct/assets/VLAct.pdf) and [code repository](https://github.com/starVLA/VLAct). ## Checkpoint details | Item | Value | | --- | --- | | Framework | StarVLA `QwenPI_v4` | | Base VLM | [`StarVLA/Qwen3-VL-4B-Instruct-Action`](https://huggingface.co/StarVLA/Qwen3-VL-4B-Instruct-Action) | | Pretrained backbone | [`StarVLA/VLAct_Qwen3_Pretrain`](https://huggingface.co/StarVLA/VLAct_Qwen3_Pretrain) @ 100K | | Action head | PI / flow-matching, randomly initialized at fine-tuning start | | Action representation | Continuous absolute joint (`action_mode: abs`) | | Action dimensions | 14-D | | Action horizon | 32 steps (`future_action_window_size: 31`) | | Diffusion settings | `num_inference_timesteps: 4`, `repeated_diffusion_steps: 4` | | Dataset mix | `robotwin_wrap_32` | | Data root | `./playground/Datasets/RoboTwin-Clean` | | Training step | 50,000 | | Per-GPU VLA batch size | 16 | | Extra losses | `endpoint_wrap_loss_weight: 0.5`, wrap-aware angular joint loss | | Learning rates | Qwen-VL interface `1e-5`; action/base modules `1e-4` | | Seed | 42 | ## Fine-tuning data Fine-tuning uses RoboTwin 2.0 Clean (`robotwin_wrap_32`) with absolute-joint actions and a CoT object-grounding prompt. The PI flow-matching head is randomly initialized and trained on top of the VLAct backbone. ## Recommended use: download and evaluate ### 1. Install StarVLA / VLAct ```bash git clone https://github.com/starVLA/VLAct.git cd VLAct conda create -n vlact python=3.10 -y conda activate vlact # Install a CUDA-compatible PyTorch build first. python -m pip install -r requirements.txt python -m pip install flash-attn==2.7.4.post1 --no-build-isolation python -m pip install -e . ``` ### 2. Download the checkpoint Run from the VLAct repository root: ```bash hf download StarVLA/VLAct_Qwen3PI_Robotwin_Finetune \ --local-dir playground/Pretrained_models/VLAct-Qwen3VL4B-PI-Robotwin ``` The checkpoint path is then: ```text playground/Pretrained_models/VLAct-Qwen3VL4B-PI-Robotwin/checkpoints/steps_50000_pytorch_model.pt ``` Keep the downloaded directory structure unchanged. StarVLA resolves `config.yaml` and `dataset_statistics.json` from the run directory two levels above the checkpoint file. ### Action un-normalization The policy predicts actions normalized to roughly `[-1, 1]`. StarVLA maps them back to physical units using the `q01` / `q99` / `mask` statistics stored in `dataset_statistics.json`, and the `unnorm_key` selects which statistics block to use. This run packages a single key, `new_embodiment`. StarVLA resolves it automatically when only one key is present, so you normally do not need to set anything. If your evaluation config exposes an `unnorm_key` field (for example DOMINO's `examples/DOMINO/eval_files/deploy_policy.yml` or the eval config generated by the RoboTwin launcher), set it to `new_embodiment`. Changing the normalization statistics, camera ordering, state usage, action ordering, or execution horizon can materially change results. ### 3. Evaluate with the StarVLA / VLAct scripts Follow [`examples/Robotwin/README.md`](https://github.com/starVLA/VLAct/tree/main/examples/Robotwin) for evaluation setup. Reproduce training with: ```bash bash scripts/run_scripts/RoboTwin/train_robotwin_qwen3pi.sh ``` ## Loading the policy Reconstruct the policy with the matching StarVLA framework (`QwenPI_v4`) and the packaged configuration. The `.pt` file contains model parameters only; it does not package optimizer or scheduler state. This checkpoint is intended for **evaluation or further fine-tuning on the same embodiment / action contract**. Transferring it to a different robot, camera setup, or action space usually requires additional adaptation. ## Files ```text VLAct-Qwen3VL4B-PI-Robotwin/ ├── README.md ├── config.yaml ├── training_config.original.yaml ├── dataset_statistics.json ├── summary.jsonl └── checkpoints/ └── steps_50000_pytorch_model.pt ``` | File | Purpose | | --- | --- | | `checkpoints/steps_50000_pytorch_model.pt` | Fine-tuned PyTorch state dict for the downstream policy | | `config.yaml` | Portable resolved configuration using the public base-model ID | | `training_config.original.yaml` | Original resolved run configuration as produced by training; it records the internal base-model ID used at training time and is kept for provenance only | | `dataset_statistics.json` | Dataset statistics used by StarVLA normalization utilities | | `summary.jsonl` | Saved-checkpoint step history | Checkpoint SHA-256: ```text 287cb36f9851063220d0eba53f2d684755b4ffe8214d35e99fbd260bad45511e ``` ## Benchmark results VLAct's published **RoboTwin 2.0** results: | Setting | VLAct (OFT head) | Matched Qwen3-VL-OFT baseline | | --- | ---: | ---: | | RoboTwin 2.0 Base, Clean | **80.5%** | 61.7% | | RoboTwin 2.0 Data Scaling, Clean / Random | **92.5% / 90.8%** | 88.2% / 88.3% | > [!NOTE] > The released artifact head can differ from the head used in the paper's headline > table for the same benchmark. The numbers above are the published VLAct results for > this benchmark; they are **not** a re-evaluation of this specific checkpoint. > > The RoboTwin headline numbers above use the **OFT** head, while this checkpoint uses the **PI** > head. No separate RoboTwin success rate is reported for this PI artifact. This checkpoint is the VLAct-initialized **PI** RoboTwin Clean policy for head / architecture comparison. To reproduce the headline RoboTwin Base numbers, use [`VLAct_Qwen3OFT_Robotwin_Finetune`](https://huggingface.co/StarVLA/VLAct_Qwen3OFT_Robotwin_Finetune). ## Intended use and limitations This checkpoint is intended for research on VLA representation transfer and benchmark evaluation on **RoboTwin 2.0 Clean**. - It has not been validated as a universal zero-shot policy across arbitrary robots. - Safe deployment requires embodiment-specific action mapping, normalization, camera calibration, control-rate handling, workspace constraints, and independent safety systems. - Performance depends on evaluation protocol, simulator / real-robot setup, observation configuration, and action execution settings. - The model may inherit limitations and biases from its base VLM, the VLAct pretraining mixture, and the downstream fine-tuning data. ## Citation ```bibtex @article{yang2026beyond, title={Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models}, author={Yang, Senqiao and Wang, Chengyao and Chen, Yuxin and Wang, Zixuan and Tang, Longxiang and Gui, Haokun and Ye, Jinhui and Lu, Changsheng and Wu, Xiaoyang and Zhu, Mingkang and others}, journal={arXiv preprint arXiv:2608.27550}, year={2026} } ``` ## License and acknowledgements The checkpoint is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). The VLAct code repository is released separately under the MIT License. Users must also comply with the licenses and terms of the base model, the VLAct pretraining checkpoint, and the training / evaluation datasets. VLAct builds on [StarVLA](https://github.com/starVLA/starVLA), [LeRobot](https://github.com/huggingface/lerobot), [GR00T](https://github.com/NVIDIA/Isaac-GR00T), and [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL). For questions, email [yangsenqiao.ai@gmail.com](mailto:yangsenqiao.ai@gmail.com) or open an issue in the [VLAct repository](https://github.com/starVLA/VLAct/issues).