Cosmos-3 Tactile-MoT v2 β€” step-250 (interim) checkpoint

⚠️ Interim checkpoint. These are the weights at step 250 of a 3000-step full-SFT run, shared to bring up an evaluation pipeline in parallel with training β€” not a final model. The predicted video is coherent but still early/soft. Final weights will follow.

Joint video + tactile future prediction on the real 15.9 B NVIDIA Cosmos3-Nano backbone. Task: past video + past tactile β†’ future video + future tactile (30 Hz source / 10 Hz slow / 30 Hz fast; no action, no hand-pose).

What was trained (v2, full SFT)

Full SFT of the Cosmos3 generation pathway (~6.98 B, FSDP2/ZeRO-3) + a fresh tactile modality; the und/reasoner tower and the Wan VAE are frozen. On top of the v1 fixes (distinct per-view mRoPE, corrected VAE conditioning boundary, removed fast-path future leakage), v2 adds:

  1. Slow tactile = aligned 10 Hz sample (the 30 Hz sample at the tick time), not a within-block mean.
  2. Per-view identity embedding (head / wrist-L / wrist-R).
  3. English text conditioning via the official Cosmos3 prompt format (system + chat template + duration/resolution + <|vision_start|>); prompts are English translations of the vendor task.
  4. Official UniPC flow sampler for sampling; trainer uses fp32-master + bf16 compute.

Both the slow expert (gen pathway + tactile head) and the 30 Hz fast expert (codec + FastTactileExpert) are trained.

Files

  • model.safetensors β€” full model weights in bf16 (Cosmos3 backbone + tactile head + view embedding + tactile codec + fast expert + Wan VAE). ~32 GB.
  • config.json β€” the JointModelConfig used (set snapshot to your local Cosmos3-Nano base path).
  • keys.json β€” the exact state-dict keys (for a load sanity check).

How to load & evaluate

Requires the model code and the Cosmos3-Nano base snapshot (for architecture/config + frozen weights the checkpoint overrides).

import torch
from safetensors.torch import load_file
from tacwam.cosmos_tactile.joint_model import JointModelConfig, TactileCosmosModel

cfg = JointModelConfig(snapshot="/path/to/Cosmos3-Nano", train_mode="gen_sft", use_lora=False,
                       master_dtype="bfloat16", slow_ctx=8, slow_hor=8, num_views=3, use_fast_expert=True)
m = TactileCosmosModel(cfg).to("cuda").eval()
sd = load_file("model.safetensors")
missing, unexpected = m.load_state_dict(sd, strict=False)
assert len(unexpected) == 0 and len(missing) < 8, (len(missing), len(unexpected))  # only tiny non-persistent buffers may miss

Or with the repo's eval/coherence helper (handles .safetensors, single-file .pt, and DCP dirs):

python scripts/cosmos3/smoke_coherence.py --ckpt <dir-with-model.safetensors> --sampler unipc

Caveats

  • Interim (8% of the planned run); expect softness, especially in the close-up wrist views.
  • Sampling is rectified-flow; use --sampler unipc (the checkpoint's shipped scheduler).
  • The training corpus is egocentric human-subject vendor video; only weights are shared here.
Downloads last month
8
Safetensors
Model size
16B params
Tensor type
BF16
Β·
Video Preview
loading

Model tree for haohw/cosmos3-tactile-mot-v2-step250

Finetuned
(48)
this model