cagataydev/smolvla_tictactoe_vision_unfrozen

SmolVLA TicTacToe — Vision-Unfrozen (60K steps)

Fine-tuned SmolVLA on the HashtagRobotics tic-tac-toe SO-101 dataset with the vision backbone unfrozen (393M trainable ≈ 87% of the 450M total), trained end-to-end through the strands-robots training abstraction (create_trainer("lerobot_local", policy_type="smolvla")).

This is the counterpart to the default frozen-vision SmolVLA fine-tune. Unfreezing the SigLIP vision encoder allows the model to adapt to the tic-tac-toe scene's specific texture / lighting / geometric-board features that the SmolVLA base wasn't pretrained on.

When to use this checkpoint: OOD scenes where the base VLM's community-pretrained features don't cover the visual domain (tic-tac-toe board, custom lighting, specific object appearance). For in-distribution manipulation, the frozen-vision variant is faster and comparable.

Training summary

Metric	Value
Base model	lerobot/smolvla_base
Steps	60,000
Trainable params	393M / 450M (vision + expert)
Final loss	0.078
Wall time	~10h 40m on 1× GPU
Throughput	~1.6 step/s, ~13 samples/s
Peak GPU mem	8.85 GB
Samples seen	480K (~3.32 epochs)
Final LR	2.5e-6 (cosine schedule)
Dataset	195 episodes, ~144K frames @ 30 Hz, SO-101 (6-DoF)
Cameras	observation.images.top, observation.images.wrist
Framework	strands-robots training.create_trainer("lerobot_local")

Loss trajectory

Step	Loss
10K	~0.15
20K	~0.11
30K	~0.09
40K	~0.08
50K	~0.075
60K	0.078

Frozen vs unfrozen: what changed

SmolVLA's config defaults are freeze_vision_encoder=True + train_expert_only=True, which gives ~100M trainable (action-expert only). To train the full stack we override both to False:

extra={
    "policy_type": "smolvla",
    "policy.freeze_vision_encoder": False,   # MUST be Python bool, not "false" string
    "policy.train_expert_only":     False,
}

Verify you got the unfrozen path — grep the training log for num_learnable_params:

• 99,880,992 (~100M) → expert-only, vision frozen
• 392,904,096 (~393M) → vision-unfrozen ✓
• 450,046,176 (~450M) → total (frozen + trainable)

Speed impact: ~1.6 step/s (unfrozen) vs ~2.6 step/s (frozen).

How to train this yourself (via strands-robots)

The full training script — using the strands_robots.training abstraction, not raw lerobot.scripts.lerobot_train — is reproduced below.

# 1. Env
git clone https://github.com/strands-labs/robots.git
cd robots
pip install -e ".[lerobot,sim-mujoco]"

# 2. Kick training (single GPU, 60K steps, vision unfrozen)
python train_smolvla_tictactoe.py \
    --steps 60000 --batch 8 \
    --repo-id HashtagRobotics/tic-tac-toe-so101-block-a-clean-v1 \
    --base-model lerobot/smolvla_base \
    --out ./checkpoints \
    --streaming

train_smolvla_tictactoe.py (full source)

"""Train SmolVLA on HashtagRobotics/tic-tac-toe-so101-block-a-clean-v1 via
strands_robots training abstraction (NOT raw lerobot.scripts.lerobot_train).

Tests the strands_robots.training.create_trainer("lerobot_local", policy_type="smolvla")
DX end-to-end on a real public HF dataset.

Dataset: 195 episodes, ~144k frames @ 30fps, SO-101 (6-DoF joint state/action),
         two cameras: observation.images.top, observation.images.wrist.
Base:    lerobot/smolvla_base.
"""
from __future__ import annotations

import argparse
import sys
import time
from pathlib import Path

from strands_robots.training import TrainSpec, create_trainer


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--smoke", action="store_true", help="Tiny run (200 steps, batch=4)")
    ap.add_argument("--steps", type=int, default=20000)
    ap.add_argument("--batch", type=int, default=8)
    ap.add_argument("--lr", type=float, default=None)
    ap.add_argument("--method", default="full", choices=["full", "lora", "expert_only"])
    ap.add_argument("--out", default="./checkpoints")
    ap.add_argument("--dataset-root", default=None)
    ap.add_argument("--repo-id", default="HashtagRobotics/tic-tac-toe-so101-block-a-clean-v1")
    ap.add_argument("--base-model", default="lerobot/smolvla_base")
    ap.add_argument("--streaming", action="store_true",
                    help="Stream shards from HF instead of full download.")
    ap.add_argument("--save-freq", type=int, default=1000)
    ap.add_argument("--val-episodes", type=int, default=None)
    ap.add_argument("--seed", type=int, default=42)
    ap.add_argument("--validate-only", action="store_true")
    args = ap.parse_args()

    if args.smoke:
        args.steps, args.batch, args.save_freq = 200, 4, 100

    Path(args.out).mkdir(parents=True, exist_ok=True)

    # === strands_robots training abstraction ===
    trainer = create_trainer("lerobot_local", policy_type="smolvla")
    print(f"[strands_robots] trainer={type(trainer).__name__} provider={trainer.provider_name}")
    print(f"[strands_robots] hardware_floor: {trainer.hardware_floor}")

    spec = TrainSpec(
        dataset_repo_id=args.repo_id,
        dataset_root=args.dataset_root or "",
        base_model=args.base_model,
        output_dir=args.out,
        steps=args.steps,
        global_batch_size=args.batch,
        learning_rate=args.lr,
        method=args.method,
        save_freq=args.save_freq,
        val_episodes=args.val_episodes,
        num_gpus=1,
        seed=args.seed,
        streaming=args.streaming,
        extra={
            "policy_type": "smolvla",
            "job_name": "smolvla-tictactoe-60k-vision-unfrozen",
            # smolvla_base expects observation.images.camera{1,2,3} — dataset
            # has top/wrist, so rename them.
            "rename_map": {
                "observation.images.top":   "observation.images.camera1",
                "observation.images.wrist": "observation.images.camera2",
            },
            # === VISION UNFROZEN ===
            # Defaults: freeze_vision_encoder=True + train_expert_only=True → ~100M trainable.
            # Overriding both to Python False → ~393M trainable.
            # WARNING: in-process passthrough uses setattr(), so values MUST be Python
            # booleans, not strings. String "false" is truthy → freeze stays on.
            "policy.freeze_vision_encoder": False,
            "policy.train_expert_only":     False,
        },
    )

    # Preflight
    print("\n[validate] running preflight...")
    problems = trainer.validate(spec)
    if problems:
        print("[validate] PROBLEMS:")
        for p in problems:
            print("  •", p)
        if not args.validate_only:
            return 1
    else:
        print("[validate] ✅ no problems.")

    # Show argv-parity CLI (what the abstraction would call under the hood)
    try:
        cmd = trainer.build_command(spec)
        print("\n[argv-parity] equivalent lerobot CLI:")
        print("  " + " \\\n    ".join(cmd))
    except Exception as e:
        print(f"[argv-parity] build_command failed: {e}")

    if args.validate_only:
        return 0

    # Prepare + train
    print("\n[prepare]")
    trainer.prepare(spec)

    print(f"\n[train] launching in-process — steps={args.steps} batch={args.batch}")
    t0 = time.time()
    try:
        result = trainer.train(spec)
    except KeyboardInterrupt:
        print("\n[train] interrupted by user")
        return 130
    dt = time.time() - t0

    print(f"\n[train] finished in {dt/60:.1f} min")
    print(f"  status:         {result.status}")
    print(f"  job_id:         {result.job_id}")
    print(f"  checkpoint_dir: {result.checkpoint_dir}")
    print(f"  exported_model: {result.exported_model}")
    for k, v in (result.metrics or {}).items():
        print(f"    metric[{k}]: {v}")

    if result.status == "success" and result.checkpoint_dir:
        exported = trainer.export(spec, result.checkpoint_dir)
        print(f"\n[export] exported model → {exported}")
        print(f"[export] load with:")
        print(f"  create_policy('lerobot_local', pretrained_name_or_path='{exported}')")

    return 0 if result.status == "success" else 2


if __name__ == "__main__":
    sys.exit(main())

How to run this model (inference / eval)

Option A — via strands_robots.create_policy (recommended)

from strands_robots import Robot, create_policy

# Load this fine-tuned checkpoint
policy = create_policy(
    "lerobot_local",
    pretrained_name_or_path="cagataydev/smolvla_tictactoe_vision_unfrozen",
    policy_type="smolvla",
    device="cuda",
)

# Attach to an SO-101 in MuJoCo
sim = Robot("so101", mesh=False)
sim.add_camera(name="top", position=[0.5, 0, 0.4], target=[0.2, 0, 0.05])
sim.add_camera(name="wrist", parent_body="so101/gripper",
               position=[0.0, 0.0, 0.05], target=[0.05, 0.0, 0.0], fov=70.0)

sim.run_policy(
    robot_name="so101",
    policy=policy,
    instruction="place the block on the tic-tac-toe board",
    n_steps=300,
    control_frequency=30,
    # Feed cameras with the same rename map used at training time
    camera_key_map={
        "top":   "observation.images.camera1",
        "wrist": "observation.images.camera2",
    },
    video={"path": "rollout.mp4", "camera": "top", "fps": 30},
)

Option B — direct via lerobot

from lerobot.policies.factory import make_policy
from lerobot.configs.policies import PreTrainedConfig

cfg = PreTrainedConfig.from_pretrained("cagataydev/smolvla_tictactoe_vision_unfrozen")
policy = make_policy(cfg=cfg, ds_meta=None)   # or your dataset's meta
policy.eval().to("cuda")

# obs is a dict:
#   observation.state: (1, 6) float32   — SO-101 joint positions
#   observation.images.camera1: (1, 3, H, W) float32 in [0,1] — top cam
#   observation.images.camera2: (1, 3, H, W) float32 in [0,1] — wrist cam
# language: "place the block on the tic-tac-toe board"

action = policy.select_action(obs, task=language)

Option C — real SO-101 hardware (via strands-robots teleop-swap pattern)

from strands_robots import Robot, create_policy

# 1. Connect real SO-101 (via feetech STS3215 bus)
robot = Robot("so101", real=True, port="/dev/ttyACM0")
robot.calibrate()

# 2. Load policy
policy = create_policy(
    "lerobot_local",
    pretrained_name_or_path="cagataydev/smolvla_tictactoe_vision_unfrozen",
    policy_type="smolvla",
    device="cuda",
)

# 3. Attach cameras (Intel RealSense / USB webcams)
robot.add_camera("top", device=0)                 # top-mounted USB cam
robot.add_camera("wrist", device=1)               # wrist-mounted USB cam

# 4. Roll out
robot.run_policy(
    policy=policy,
    instruction="place the block on the tic-tac-toe board",
    control_frequency=30,
    duration=30.0,
    camera_key_map={
        "top":   "observation.images.camera1",
        "wrist": "observation.images.camera2",
    },
)

Camera contract (critical)

SmolVLA base expects observation keys named observation.images.camera{1,2,3}. The tic-tac-toe dataset ships them as top / wrist. Both training and inference MUST apply the same rename map:

Sim/dataset name	Model input key
observation.images.top	observation.images.camera1
observation.images.wrist	observation.images.camera2

If you skip this, you'll hit KeyError: 'observation.images.camera1' at model.forward.

Reproducibility

• Seed: 42
• Optimizer: AdamW, β=(0.9, 0.95), weight_decay=1e-10
• LR schedule: cosine, peak 1e-4 → min 2.5e-6 (default SmolVLA schedule)
• Precision: bf16
• Framework:
  • lerobot (via strands_robots[lerobot] extra)
  • strands_robots >= 0.6.0
  • torch >= 2.6.0
  • transformers >= 4.50
  • peft (only needed if --method lora)

Related checkpoints

• Frozen vision (100M trainable) — canonical HF recipe, faster, best for in-distribution scenes
• Vision unfrozen (this repo, 393M trainable) — better for OOD scenes like tic-tac-toe
• LoRA (r=64) — dramatically cheaper, comparable perf on niche data. Available via --method lora in the same script.

Dataset

HashtagRobotics/tic-tac-toe-so101-block-a-clean-v1

• 195 episodes, ~144K frames @ 30 Hz
• Robot: SO-101 (6-DoF)
• Cameras: top (0), wrist (1)
• Task: place colored blocks on tic-tac-toe board positions
• LeRobot v3 format

License

Apache-2.0 (matches base model)

Citation

@misc{cagatay2026smolvla_tictactoe_vision_unfrozen,
  title  = {SmolVLA Fine-tune on TicTacToe SO-101 (Vision Unfrozen)},
  author = {cagataydev},
  year   = {2026},
  url    = {https://huggingface.co/cagataydev/smolvla_tictactoe_vision_unfrozen}
}

@article{smolvla2025,
  title  = {SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics},
  author = {Hugging Face LeRobot Team},
  year   = {2025},
  url    = {https://arxiv.org/abs/2506.01844}
}

---

Trained end-to-end using strands-robots training.create_trainer("lerobot_local") on Thor (NVIDIA aarch64). Full training script above is self-contained and reproducible.