TinyVLA: a dual-rate VLA that beats OpenVLA-7B on LIBERO-Spatial at 0.64B params

Headline result: 86% success on LIBERO-Spatial (n=100) in dual-rate mode (semantic latent refreshed at 1 Hz), 79% full-rate — vs OpenVLA-7B's 84.7% — with 140M parameters on the online control path. Built and trained end-to-end in ~5 days on a single RTX A6000.

1. Architecture

                SLOW / SMART path (0.5–2 Hz, cached)
instruction ─────────────────────┐
camera → Qwen3.5-0.8B vision tower → Qwen3.5 language stack (first 12 of 24 layers)
                  │                       │
                  │            8 readout tokens → linear → semantic latent (8×512)
                  │                       │ cached between refreshes
──────────────────┼───────────────────────┼──────────────────────────
                  │       FAST / MOTOR path (10–20 Hz, ~140M online)
                  └── 64 spatial tokens per camera ─┐
proprio (padded 16d) ───────────────────────────────┤
embodiment-id embedding ────────────────────────────┤
                                                    ▼
                    flow-matching action expert (38M, 12 blocks,
                    interleaved cross-/self-attention, AdaLN-free)
                                                    ▼
                              50-step action chunk (padded 8d + validity mask)

• Single shared encoder: the Qwen3.5-0.8B native ViT (100.6M) feeds both paths.
• Slow path: image + instruction + 8 trailing pad-token "readout" slots through the truncated language stack (L/2 layer-skip a la SmolVLA); their hidden states → latent 8×512.
• Fast path: raw vision-tower spatial tokens (bypassing the LM) + proprio + embodiment token, cross-attended by the flow expert. 10 Euler steps at inference.
• Flow matching: linear-OT path, t ~ Beta(1.5, 1), velocity target, masked MSE.
• Totals: 643M stored / 39.5M–575M trainable depending on stage / ~140M online per tick.
• Implemented as an out-of-tree LeRobot policy plugin (--policy.type=tinyvla --policy.discover_packages_path=tinyvla, lerobot 0.6.1).

2. Training recipe

Stage	Data	Steps	Trainable	Result
1. Overfit smoke	svla_so100_stacking (56 eps)	5k	expert+proj (40M)	loss 1.885→0.099
2. Mixture pretrain	40 SO100/101 community datasets (~10.7K eps, weight 0.65, per-dataset proportional) + Bridge V2 10K eps (0.35)	40k, bs 64×2, lr 1e-4	39.5M (backbone frozen)	held-out RMSE@t0 0.30σ; stale-latent 2s == fresh
2b. Staleness aug	same	from step 20k	—	latent from frame t−k, k~U(0,2s), p=0.5
3. Spatial distillation	+ 132K teacher labels (Qwen3.5-4B, center-point → 32×32 grid), CE ×0.15 on cam0 tokens	20k, lr 5e-5	same + head	head top-1 20.8%, top-5 44.8% (random 0.1%)
4. LIBERO FT (ft4)	HuggingFaceVLA/libero (1693 eps, all suites)	40k, bs 48, lr 1e-4 / backbone 0.1×	575M (LM layers + vision unfrozen; vocab embeddings frozen)	loss 0.066

Ablation FTs: ft1 (all frozen) → 0% closed-loop; ft2 (vision unfrozen) → 60%; ft3 (= ft2 from distilled ckpt) → 61%; ft4 (LM+vision unfrozen) → 79/86%. Unfreezing the LM was the single largest gain; distillation was neutral on LIBERO (likely overwritten by vision unfreezing during FT — bake it into pretraining next time).

Cross-embodiment handling: canonical schema (cam0=scene/cam1=wrist via regex mapping, per-dataset mean/std normalization, action pad-to-8 + validity mask, state pad-to-16, learned embodiment-id token). Weighted mixture sampler with per-source retry on corrupt frames (IndexError/KeyError/decode RuntimeError).

3. Results

LIBERO — all four suites (dual-rate, latent @ 1 Hz)

Suite	TinyVLA	OpenVLA-7B (published)
Spatial	86% (n=100; full-rate 79%)	84.7%
Object	94% (n=50)	88.4%
Goal	86% (n=50)	79.2%
Long (libero_10)	70% (n=50)	53.7%
Average	84.0%	76.5%

Beats OpenVLA-7B on every suite at 1/11 the parameters and ~1/50 the online compute. XS-VLA (90.3 avg) remains ahead — their latent flow matching is future work for us.

LIBERO-Spatial (100 episodes, 10 per task, init states 0-9)

Mode	Success
Full-rate (latent every chunk)	79/100 = 79%
Dual-rate (latent @ 1 Hz)	86/100 = 86%

Dual-rate ≥ full-rate consistently across all evals (also 65% vs 58% at ft3, n=50) — a stable cached latent appears to act as temporal regularization.

Context (published numbers, their own eval protocols)

Model	Params	LIBERO-Spatial	4-suite avg
MolmoAct2-LIBERO	5B	98.4% (97.2% re-measured on our rig, n=500)	98.25%
π0	3.3B	98%	—
SmolVLA-2.25B	2.25B	93%	—
XS-VLA	0.26B	~93%	90.3%
MolmoAct-7B (v1)	7B	87.0%	86.6%
TinyVLA (this)	0.64B (140M online)	86% dual-rate	84.0%
OpenVLA	7B	84.7%	76.5%

MolmoAct2 context (from their cards/repo, no numeric tables published outside the LIBERO-LeRobot card): 5B params on Molmo2-ER backbone + flow expert; claims to beat π0.5; Molmo2-ER reportedly surpasses GPT-5 / Gemini-Robotics-ER-1.5 on 13 embodied-reasoning benchmarks; #1 VLA on the MolmoSpace leaderboard. Inference needs ~26GB fp32 (16GB bf16), cuda-graphs for ~2× speedup — H100-class serving vs our Raspberry-Pi-class 140M fast path. Notably they ship MolmoAct2-SO100_101 (absolute joint control) — a 5B competitor on our target embodiment; a head-to-head offline eval on SO101 held-out data is an obvious next test.

Honesty notes: our n=100 (±8pp) vs their 500; single suite; eval covers init states 0–9 per task (demos of same tasks/init states are in training data — this is the standard LIBERO protocol used by all listed models). Multi-suite evals (object/goal/long) and an identically-protocoled SmolVLA baseline run are in progress.

Dual-rate validation (the architectural claim)

• Offline: held-out chunk MSE with latent from a frame 2s earlier == fresh (0.772 vs 0.772).
• Closed-loop: 86% @ 1 Hz refresh vs 79% full-rate.
• Deployment implication (Raspberry Pi 5): the 0.5B slow path can run at 0.5–1 Hz (llama.cpp GGUF Q4 territory), only the 140M fast path needs real-time rates.

4. Pitfalls we hit (read before reproducing)

1. robosuite/LIBERO renders are 180°-rotated vs recorded datasets. A vertical flip alone leaves a horizontal MIRROR — the policy reaches to the wrong side and scores 0%. Use img[::-1, ::-1]. (lerobot's LiberoEnv.render() flips for visualization but _format_raw_obs feeds policies raw rotated frames — check your eval path.)
2. eef orientation antipodal ambiguity: env quat → rotvec gives −π-ish x-component, datasets store +π-ish. Canonicalize (flip antipodal representation when rotvec_x < 0) or your normalized state input is off by ~17σ.
3. HuggingFaceVLA/libero meta says fps=10 but frames are native 20 Hz (~161 frames/episode ≈ 8s demos). Do NOT action-repeat at eval.
4. GT-replay harness is the decisive convention test: replay dataset actions in the env (first episode per task ↔ init_state 0). 4/5 success = conventions right; policy-independent.
5. HuggingFaceVLA community_dataset v1/v3 are aggregates of v2.1 sub-datasets → convert with lerobot's convert_dataset_v21_to_v30.py; 4/44 sources have meta≠parquet row counts (broken at source) — validate len(reader.hf_dataset) == meta.total_frames.
6. Qwen3.5 custom forwards need mm_token_type_ids for M-RoPE; patchify must match the official processor bit-exact (verify!). Linear-attention needs fla-core + causal-conv1d or you get a 5× slower torch fallback.
7. Flow-matching loss can drop while conditioning is ignored — A/B with shuffled observations to verify the model actually uses them.

5. Repository layout

src/tinyvla/            # LeRobot plugin: configuration/modeling/processor + modules
scripts/train.py        # accelerate loop: weighted mixture, staleness aug, distill loss, resume
scripts/eval_libero.py  # closed-loop LIBERO eval (all convention fixes, dual-rate, video)
scripts/replay_libero.py, diag_obs_gap.py   # convention diagnostics
scripts/prepare_community_v3.py, label_spatial.py, eval_offline.py
configs/                # stage2/stage3/libero_ft* yaml
checkpoints/            # stage2_final, stage3_final, libero_ft4_final (main)
videos/                 # sample eval episodes

6. Quick start

from tinyvla.modeling_tinyvla import TinyVLAPolicy
policy = TinyVLAPolicy.from_pretrained("AlexWortega/tinyvla")  # ft4 checkpoint at root
# closed-loop LIBERO: python scripts/eval_libero.py --checkpoint <ckpt> \
#   --suite libero_spatial --episodes 100 --n-action-steps 10 --refresh-s 1.0

Trained 2026-08-14 → 2026-08-19 on 1× RTX A6000 (48GB), lerobot 0.6.1, torch 2.10, transformers 5.5.4.