rayaanoidpr/grandtour-world-models

GrandTour World Models

Action-conditioned quadruped world models trained on the GrandTour dataset (ANYmal-D, hdr_front / depth). Two frameworks, one repo:

• LeWM-depth (lewm-*) — latent world model (JEPA + SIGReg) from rayaanoidPrime/quad-wm.
• V-JEPA (vjepa/) — frozen DINOv2 / V-JEPA 2.1 encoder + AdaLN predictor from rayaanoidPrime/vjepa-wm.

Layout

lewm-depth/                       original depth LeWM (5600 steps, no state head)
lewm-depth-grounded-s4551/        grounded arm, state head (ADR 0007)
lewm-depth-residual-s4551/        joint-residual action arm, state head
lewm-depth-residual-simft-s4551/  residual + sim fine-tune (1800 steps)
jepa-baseline-v2/                 V-JEPA-style baseline (quad-wm framework)
reports/                          eval reports (2728/2741/2742)
vjepa/                            V-JEPA checkpoints (see below)

V-JEPA (vjepa/)

run	encoder	predictor	steps	epoch-avg loss
gt_v0_12f_fps5_r224_dv2vits_AdaLN_d6_2roll_1n	DINOv2 ViT-S/14 (frozen)	AdaLN depth 6	1000	5.3 → 1.53
gt_v1_12f_fps5_r224_dv2vitl_AdaLN_d12c_2roll_1n	DINOv2 ViT-L/14 (frozen, 304M)	AdaLN depth 12 / embed 1024 / causal t-window 3	1600+	1.25 → 0.63
gt_v0_step2_vm2m, gt_v1_step2_vm2m	VM2M-style in-domain decoder heads
gt_v4/	V-JEPA 2.1 pretrain (model_only + jepa-model-only + probe)
bigrun/	V-JEPA 2.1 ViT-B/384, causal tubelet masking, velocity-consistency head		8000

V-JEPA copy-baseline (latent L2): gt_v0 model/copy 1.01 (1-step) → 0.88 (h=6); gt_v1 0.81 → 0.77. Big run (step 8000) held-out: feature cosine 0.890 / 0.854 (ICE-1 / SPX-2), velocity RMSE true-vs-shuffled 0.061/0.097 and 0.052/0.092. Original cards kept at vjepa/README-vjepa-grandtour.md and vjepa/bigrun/README.md; full bigrun log in vjepa/bigrun/big_run_log.txt.

LeWM-depth key results

Real data (GrandTour), EV5 rollout error ε_k (σ-normalized, lower better, k=1, linear probe):

run	model	persistence	encoded_floor
lewm-depth (original)	3.30	6.41	3.26
grounded	2.07	6.52	1.56
residual	2.12	6.52	1.58
residual-simft	2.23	6.56	1.65

Latents probe well for lin_vel/joint_pos/joint_vel (r²≈0.93–0.98) but poorly for base_pos (r²≈0.18); contacts are not represented. In the MuJoCo EV3 CEM-planning eval the scripted controller is still better; only the sim-fine-tuned arm avoids flat-terrain falls and its Δ_s2r is closest to zero. See reports/report-2741.md (all arms, protocol-v2) and report-2742.md (sim-lewm-v1 planning).

Usage

LeWM (quad-wm):

git clone https://github.com/rayaanoidPrime/quad-wm
CONFIG=configs/jepa-wm/lewm_depth_residual_simft.yaml RUN_ROOT=<dir-with-the-runs> \
  uv run --extra sim quadwm sim-eval --sim-eval-config configs/eval/sim_lewm.yaml

LeWM checkpoints are plain dicts {model, optimizer, epoch, config}.

V-JEPA (vjepa-wm, branch exp-v2):

GT_RUN=gt_v1_12f_fps5_r224_dv2vitl_AdaLN_d12c_2roll_1n python grandtour/scripts/eval_copy_baseline.py

Caveats

• LeWM arms are single-seed (s4551); protocol wants ≥3 seeds for significance.
• lewm-depth (original) has no state head, so no planning_state_head readout.
• V-JEPA runs come from different points in the vjepa-wm project; check each yaml for exact architecture/training settings.

Licenses

GrandTour data: CC BY-SA 4.0 (Frey et al., arXiv:2602.18164). V-JEPA weights derive from DINOv2 (Apache-2.0) and the jepa-wms codebase (CC-BY-NC 4.0). LeWM from the quad-wm codebase.