xijiazhao/VLA-finetuned

VLA finetunes on UR10e multi-task manipulation

Four vision-language-action policies finetuned on the same real-robot dataset, so their representations can be compared under a common protocol. Each is shipped both as the LoRA adapter it was trained as and as a merged, directly-loadable checkpoint (except the full finetune, which is already a full model).

Dataset

bag100/ur10e-multitask — a real UR10e arm, not simulation.

episodes	285 train / 50 held-out val
frames	51,215 train / 9,201 val
tasks	7
rate	15 fps
cameras	3 × 180×320 (exterior_1 overhead = primary, wrist, exterior_2)
state	7 = 6 joint positions (rad) + gripper [0,1]
action	7 = 6 joint velocities (±0.9 rad/s) + gripper

Split is seed-0 and identical across every model here. Normalization statistics were computed on the train split alone.

Models

All four ran 20,000 optimizer steps on the same split.

folder	base	trainable	recipe
smolvla-ur10e-lora (+-merged)	SmolVLA 0.45 B	LoRA r=32 α=64 on expert q/v + projections	bs 32/GPU, lr 1e-4
smolvla-ur10e-full	SmolVLA 0.45 B	all 0.45 B	bs 16/GPU, lr 5e-5
xvla-ur10e-lora (+-merged)	X-VLA 0.89 B (lerobot/xvla-libero)	13.6 M (1.5 %) — LoRA r=32 α=64 dr=0.05 on the policy-transformer blocks, plus soft_prompt_hub trained in full	bs 16, eff. batch 64, lr 1e-4
groot-ur10e-lora (+-merged)	GR00T N1.5-3B	328 M / 2.72 B (10.8 %) — LoRA r=64 α=128 + projectors	bs 16/GPU
groot-ur10e-full	GR00T N1.5-3B	≈1.5 B / 2.72 B — vision tower + action head, Eagle LLM frozen	bs 8/GPU
xvla-ur10e-full	X-VLA 0.89 B	all 879.5 M	bs 16/GPU, eff. batch 64, lr 5e-5

Three of the seven are full finetunes (smolvla-ur10e-full, groot-ur10e-full, xvla-ur10e-full); the rest are LoRA. Every LoRA arm ships both the adapter and a pre-merged standalone checkpoint, so nothing here requires fetching a base model to run.

X-VLA is the one architecture present as a matched LoRA/full pair trained on identical data with the same effective batch, which makes it the cleanest comparison in the repo — once either arm has a held-out score.

X-VLA's soft prompts are its cross-embodiment mechanism; pure LoRA would have frozen the UR10e domain at its LIBERO value, so they are in modules_to_save. Its action_mode is auto (AutoActionSpace), not the base's ee6d.

Held-out evaluation

Chunked open-loop replay on the 50 held-out episodes. nMSE = per-dim MSE ÷ per-dim variance of the held-out actions, averaged: 1.0 = no better than predicting the dataset mean, 0 = perfect.

model	nMSE ↓	joints	gripper	MAE (rad/s)	grip-acc
smolvla-ur10e-full	0.2916	0.3300	0.0612	0.0440	96.2 %
groot-ur10e-full	0.3146	0.3599	0.0430	0.0407	97.6 %
groot-ur10e-lora	0.3613	0.4111	0.0625	0.0451	97.0 %
smolvla-ur10e-lora	0.3849	0.4361	0.0775	0.0535	95.9 %
xvla-ur10e-full	not yet measured	—	—	—	—
xvla-ur10e-lora	not yet measured	—	—	—	—
control: train-mean action	1.0004	1.0003	1.0012	0.1296	54.8 %
control: constant zero	1.2082	1.0151	2.3662	0.1417	45.2 %
control: un-finetuned base, zero-shot	1.6242	1.7199	1.0501	0.1717	43.8 %

The zero-shot control matters: an un-finetuned base checkpoint on these episodes scores worse than emitting zeros, so clearing 1.0 is informative rather than automatic.

Neither X-VLA arm has a held-out number yet, so both sit unscored above. Their training losses:

step	100	4K	8K	12K	16K	20K
xvla-ur10e-lora	1.187	0.215	0.146	0.124	0.106	0.115
xvla-ur10e-full	1.103	0.041	0.019	0.011	0.007	0.006

The LoRA arm flattens from ~14K steps and ends marginally above its own 16K minimum; the full arm reaches ~19× lower loss and was still descending at 20K. On 285 episodes that gap is equally consistent with a better fit and with memorization — training loss cannot separate them, and the open-loop replay that would has not been run for either. Treat it as a statement about optimization, not generalization.

No success rate was measured for any model here — these are open-loop trajectory metrics, not task success.

Loading

Merged checkpoints (LeRobot policies):

from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy
from lerobot.policies.factory import make_pre_post_processors

policy = SmolVLAPolicy.from_pretrained("smolvla-ur10e-lora-merged")
pre, post = make_pre_post_processors(
    policy_cfg=policy.config, pretrained_path="smolvla-ur10e-lora-merged",
    preprocessor_overrides={"device_processor": {"device": "cuda"}})
action = post(policy.select_action(pre(batch)))   # -> (B, 7)

XVLAPolicy for the X-VLA folders. The processor pipeline is not optional — normalization, key renaming and language tokenization all live in policy_preprocessor.json + policy_*_normalizer_processor.safetensors, not inside the policy. Calling select_action on a raw dict skips all of it.

GR00T folders need the Isaac-GR00T codebase; experiment_cfg/metadata.json carries the new_embodiment normalization statistics that Gr00tPolicy reads at inference.

Gotchas

1. config.json declares a stale observation.state width. SmolVLA says 6, X-VLA says 8; the real state is 7. The value is inherited from each base checkpoint and is inert at runtime — the saved 7-dim normalizer statistics are what the pipeline actually broadcasts against — but building an input tensor from config.input_features["observation.state"] will produce a confusing shape error. Trust the stats, not the config. The action shape (7) is correct everywhere.
2. X-VLA's preprocessor downloads facebook/bart-large for its tokenizer step at load time. Under HF_HUB_OFFLINE=1 with a cold cache it fails.
3. LeRobot's checkpoint writer omits the "type" discriminator that PreTrainedConfig.from_pretrained requires; it has been written back into every config.json here.
4. X-VLA needs a LeRobot revision that includes the xvla policy.

Both full finetunes beat their LoRA twins here, on a dataset (285 episodes) small enough that the opposite would not have been surprising. The 0.45 B SmolVLA also beats the 2.72 B GR00T on nMSE, so parameter count is not what orders this table.

Not included

Two finetunes on the same dataset are deliberately absent: π0.5 LoRA (open-loop nMSE 0.6404) and an OpenVLA-OFT LoRA (7.65 B, 20,000 steps, completed) — both exist and are archived, just not published here.