| <div align="center"> | |
| <h2>Understanding Physical Dynamics with Counterfactual World Modeling</h2> | |
| [**Rahul Venkatesh***](https://rahulvenkk.github.io/)<sup>1</sup> · [**Honglin Chen***](https://web.stanford.edu/~honglinc/)<sup>1*</sup> · [**Kevin Feigelis***](https://neuroscience.stanford.edu/people/kevin-t-feigelis)<sup>1</sup> · [**Daniel M. Bear**](https://twitter.com/recursus?lang=en)<sup>1</sup> · [**Khaled Jedoui**](https://web.stanford.edu/~thekej/)<sup>1</sup> · [**Klemen Kotar**](https://klemenkotar.github.io/)<sup>1</sup> · [**Felix Binder**](https://ac.felixbinder.net/)<sup>2</sup> · [**Wanhee Lee**](https://www.linkedin.com/in/wanhee-lee-31102820b/)<sup>1</sup> · [**Sherry Liu**](https://neuroailab.github.io/cwm-physics/)<sup>1</sup> · [**Kevin A. Smith**](https://www.mit.edu/~k2smith/)<sup>3</sup> · [**Judith E. Fan**](https://cogtoolslab.github.io/)<sup>1</sup> · [**Daniel L. K. Yamins**](https://stanford.edu/~yamins/)<sup>1</sup> | |
| (* equal contribution) | |
| <sup>1</sup>Stanford    <sup>2</sup>UCSD    <sup>3</sup>MIT | |
| <a href="https://arxiv.org/abs/2312.06721"><img src='https://img.shields.io/badge/arXiv-CWM-red' alt='Paper PDF'></a> | |
| <a href='https://neuroailab.github.io/cwm-physics/'><img src='https://img.shields.io/badge/Project_Page-CWM-green' alt='Project Page'></a> | |
| <a href='https://neuroailab.github.io/cwm-physics/'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue'></a> | |
| <a href='https://neuroailab.github.io/cwm-physics/'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Colab-yellow'></a> | |
| </div> | |
| This work presents the Counterfactual World Modeling (CWM) framework. CWM is capable of counterfactual prediction and extraction of vision structures useful for understanding physical dynamics. | |
|  | |
| ## 📣 News | |
| - 2024-06-01: Release [project page](https://neuroailab.github.io) and [codes](https://github.com/rahulvenkk/cwm_release.git) | |
| ## 🔨 Installation | |
| ``` | |
| git clone https://github.com/rahulvenkk/cwm_release.git | |
| pip install -e . | |
| ``` | |
| ## ✨ Usage | |
| To download and use a pre-trianed model run the following | |
| ``` | |
| from cwm.model.model_factory import model_factory | |
| model = model_factory.load_model('vitbase_8x8patch_3frames_1tube') | |
| ``` | |
| This will automatically initialize the appropriate model class and download the specified weights to your `$CACHE` directory. | |
| ## 🔄 Pre-training | |
| To train the model run the following script | |
| ``` | |
| ./scripts/pretrain/3frame_patch8x8_mr0.90_gpu.sh | |
| ``` | |