Skip to content
lucrbviPublic

About

Play DOOM inside a small world model

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

miniworld

A small Latent World Model of DOOM (1993). The model predicts the next frame's latent tokens from past frames and actions, then decodes them to pixels. It runs fast enough that you can play inside it in real time.

Install

uv sync

Download pretrained weights

uv run download_weights.py

This pulls the world model (and action policy, if available) from the Hugging Face Hub into ./checkpoints/world-model/ and ./checkpoints/action-policy/.

Train

uv run train.py --mode wm              # encoder + predictor
uv run train.py --mode decoder         # pixel decoder on a frozen WM
uv run train.py --mode action_policy   # forward dynamics action policy
uv run train.py --mode all             # all three, in order

Training data is downloaded automatically from Hugging Face. Checkpoints are written to ./checkpoints/world-model, ./checkpoints/decoder and ./checkpoints/action-policy, and resume from the latest one when you rerun.

The decoder and action_policy modes need a trained world model to start from. Either run --mode wm first (or --mode all), or pass --wm-checkpoint <path> to point at one checkpoint explicitly.

Play inside the world model

Drive the world model with your keyboard and play a pseudo-DOOM game. First download the weights (see above), then run:

uv run dream.py

Let the policy play

The learned action policy plays in the real DOOM engine while you watch.

uv run autoplay.py

This uses ./checkpoints/world-model and ./checkpoints/action-policy by default.

Pass --wad path/to/doom.wad to play the real game (for example DOOM.WAD). Without it, VizDoom falls back to its bundled FreeDoom resources, which work but look slightly off and have different level names. You can also add --record out.mp4 to save a video, or --headless to skip the window.

Architecture

The encoder is a ViT that maps frames to patch tokens of shape [B, T, n_patches+1, D]. The predictor is a causal transformer over time, conditioned on actions, trained in latent space with MSE, short rollouts and the LeJEPA SIGReg objective. The decoder is a simple convolutionnal neural networks with Pixel Shuffling to generate pixels. The action policy is a MLP that predict the next action vector.

References

About

Play DOOM inside a small world model

Resources

Stars

2 stars

Watchers

0 watching

Forks

Contributors

Languages