DINO-WM checkpoints for ACWM-Phys

DINO-WM world models trained on four environments of ACWM-Phys: Pour Water, Cloth Move, Push Sand and Stack Cube. The code, settings and results are in the branch hanbin5/acwm-phys of jongmin-s-multicore/thread0.

All runs use 224 px frames, frameskip 5, batch size 24, predictor learning rate 5e-5 and seed 0. The encoder is frozen DINOv2 ViT-S/14 and is not stored here.

Files

Each run is in outputs/<run>/ with checkpoints/, hydra.yaml and, for the five base runs, epoch_logs.jsonl. A numbered checkpoint is checkpoints/model_<epoch>.pth. In the base runs the last epoch is stored once, as model_latest.pth.

Run Environment Checkpoints Training
train_pw_h1f5 Pour Water epochs 1, 2, 3, 4, 5 and model_latest.pth (epoch 6) num_hist 1
train_pw_h3f5 Pour Water epochs 1, 2, 3, 4, 5 and model_latest.pth (epoch 6) num_hist 3
train_cloth_h1f5 Cloth Move epochs 5, 10 and model_latest.pth (epoch 15) num_hist 1, 61,875 steps
train_sand_h1f5 Push Sand epochs 6, 12 and model_latest.pth (epoch 18) num_hist 1, 65,016 steps
train_stack_h1f5 Stack Cube epochs 5, 10, 15, 20 and model_latest.pth (epoch 25) num_hist 1, 62,150 steps
train_cloth_h1f5_long Cloth Move epochs 30, 60, 120 continued from train_cloth_h1f5 epoch 15
train_sand_h1f5_long Push Sand epochs 36, 72, 144 continued from train_sand_h1f5 epoch 18
train_stack_h1f5_long Stack Cube epochs 50, 100, 200 continued from train_stack_h1f5 epoch 25
train_cloth_h1f5_oodmix Cloth Move epochs 5, 10, 15 trained from the start with OoD episodes added
train_sand_h1f5_oodmix Push Sand epochs 6, 12, 18 trained from the start with OoD episodes added
train_stack_h1f5_oodmix Stack Cube epochs 10, 20, 25 trained from the start with OoD episodes added
train_pw_h1f5_oodmix Pour Water epochs 2, 4, 6 trained from the start with OoD episodes added
train_pw_h1f5_long Pour Water epochs 12, 24, 48 continued from train_pw_h1f5 epoch 6

Notes on the last eight runs:

  • *_long: only the predictor, the action encoder and the proprio encoder were trained further, on DINOv2 features computed once. The decoder is the one of the base run.

  • *_oodmix: the training set is the training episodes of the base run plus part of ood_test. split.json lists the added episodes (train) and the episodes left for testing (held_out). The predictor and the two encoders were trained from random weights. The decoder is the one of the base run.

    Run Added from ood_test Left for testing Rule (--extra_train)
    train_cloth_h1f5_oodmix 25 smaller + 25 larger 25 + 25 smaller:even,larger:even
    train_sand_h1f5_oodmix 50 50 all:even
    train_stack_h1f5_oodmix 136 135 all:even
    train_pw_h1f5_oodmix 26 (14 low water, 12 high water) 24 (12 low, 12 high) all:mod4=0.1 (episode number mod 4 is 0 or 1)
  • They were trained with repro/colab/predictor_long.py; repro/colab/ckpt_tools.py wrote the checkpoints in the format above. Their hydra.yaml is a copy of the base run's file.

Use

Set up the repository as its README says. Then download into $DINO_TRAIN and run the evaluation with the run name.

source repro/env.sh
huggingface-cli download hanbin5/dino-wm-acwm-phys --local-dir $DINO_TRAIN
python repro/eval/eval_acwm_phys.py $DINO_TRAIN train_cloth_h1f5 $DINO_WORK/results/acwm_phys/cloth_h1f5

hydra.yaml is the training config. Three absolute paths in it were replaced with the variables that repro/env.sh sets: ckpt_base_path is $DINO_TRAIN and data_path starts with $DATASET_DIR.

The checkpoints are in the upstream DINO-WM format: torch.save of the module objects (predictor, decoder, action_encoder, proprio_encoder) and, in the base runs, the optimizers. They are pickles. Load them with the code of the branch above, and only from a source you trust.

Sources

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for hanbin5/dino-wm-acwm-phys