DINO-WM checkpoints for ACWM-Phys
DINO-WM world models trained on four environments of ACWM-Phys: Pour Water, Cloth Move, Push Sand and Stack Cube.
The code, settings and results are in the branch hanbin5/acwm-phys of jongmin-s-multicore/thread0.
All runs use 224 px frames, frameskip 5, batch size 24, predictor learning rate 5e-5 and seed 0. The encoder is frozen DINOv2 ViT-S/14 and is not stored here.
Files
Each run is in outputs/<run>/ with checkpoints/, hydra.yaml and, for the five base runs, epoch_logs.jsonl.
A numbered checkpoint is checkpoints/model_<epoch>.pth. In the base runs the last epoch is stored once, as model_latest.pth.
| Run | Environment | Checkpoints | Training |
|---|---|---|---|
train_pw_h1f5 |
Pour Water | epochs 1, 2, 3, 4, 5 and model_latest.pth (epoch 6) |
num_hist 1 |
train_pw_h3f5 |
Pour Water | epochs 1, 2, 3, 4, 5 and model_latest.pth (epoch 6) |
num_hist 3 |
train_cloth_h1f5 |
Cloth Move | epochs 5, 10 and model_latest.pth (epoch 15) |
num_hist 1, 61,875 steps |
train_sand_h1f5 |
Push Sand | epochs 6, 12 and model_latest.pth (epoch 18) |
num_hist 1, 65,016 steps |
train_stack_h1f5 |
Stack Cube | epochs 5, 10, 15, 20 and model_latest.pth (epoch 25) |
num_hist 1, 62,150 steps |
train_cloth_h1f5_long |
Cloth Move | epochs 30, 60, 120 | continued from train_cloth_h1f5 epoch 15 |
train_sand_h1f5_long |
Push Sand | epochs 36, 72, 144 | continued from train_sand_h1f5 epoch 18 |
train_stack_h1f5_long |
Stack Cube | epochs 50, 100, 200 | continued from train_stack_h1f5 epoch 25 |
train_cloth_h1f5_oodmix |
Cloth Move | epochs 5, 10, 15 | trained from the start with OoD episodes added |
train_sand_h1f5_oodmix |
Push Sand | epochs 6, 12, 18 | trained from the start with OoD episodes added |
train_stack_h1f5_oodmix |
Stack Cube | epochs 10, 20, 25 | trained from the start with OoD episodes added |
train_pw_h1f5_oodmix |
Pour Water | epochs 2, 4, 6 | trained from the start with OoD episodes added |
train_pw_h1f5_long |
Pour Water | epochs 12, 24, 48 | continued from train_pw_h1f5 epoch 6 |
Notes on the last eight runs:
*_long: only the predictor, the action encoder and the proprio encoder were trained further, on DINOv2 features computed once. The decoder is the one of the base run.*_oodmix: the training set is the training episodes of the base run plus part ofood_test.split.jsonlists the added episodes (train) and the episodes left for testing (held_out). The predictor and the two encoders were trained from random weights. The decoder is the one of the base run.Run Added from ood_testLeft for testing Rule ( --extra_train)train_cloth_h1f5_oodmix25 smaller+ 25larger25 + 25 smaller:even,larger:eventrain_sand_h1f5_oodmix50 50 all:eventrain_stack_h1f5_oodmix136 135 all:eventrain_pw_h1f5_oodmix26 (14 low water, 12 high water) 24 (12 low, 12 high) all:mod4=0.1(episode number mod 4 is 0 or 1)They were trained with
repro/colab/predictor_long.py;repro/colab/ckpt_tools.pywrote the checkpoints in the format above. Theirhydra.yamlis a copy of the base run's file.
Use
Set up the repository as its README says. Then download into $DINO_TRAIN and run the evaluation with the run name.
source repro/env.sh
huggingface-cli download hanbin5/dino-wm-acwm-phys --local-dir $DINO_TRAIN
python repro/eval/eval_acwm_phys.py $DINO_TRAIN train_cloth_h1f5 $DINO_WORK/results/acwm_phys/cloth_h1f5
hydra.yaml is the training config. Three absolute paths in it were replaced with the variables that repro/env.sh sets: ckpt_base_path is $DINO_TRAIN and data_path starts with $DATASET_DIR.
The checkpoints are in the upstream DINO-WM format: torch.save of the module objects (predictor, decoder, action_encoder, proprio_encoder) and, in the base runs, the optimizers. They are pickles. Load them with the code of the branch above, and only from a source you trust.
Sources
- Data: ACWM-Phys (paper, dataset
t1an/ACWM-Phys, CC BY 4.0). - Code: DINO-WM (
gaoyuezhou/dino_wm, MIT) with the changes in the branch above.