README.md

metadata

title: Multiverse
emoji: 🐟
colorFrom: blue
colorTo: blue
sdk: docker
app_file: server.py
pinned: true
short_description: AI Multiplayer World Model
app_port: 8080
disable_embedding: false

Multiverse: The First AI Multiplayer World Model

🌐 Enigma-AI website - 📚 Technical Blog - 🤗 Model on Huggingface - 🤗 Datasets on Huggingface - 𝕏 Multiverse Tweet

Two human players driving cars in Multiverse

---

Installation

git clone https://github.com/EnigmaLabsAI/multiverse
cd multiverse
pip install -r requirements.txt

Running the model

python src/play.py --compile

Note on Apple Silicon you must enable CPU fallback for MPS backend with PYTORCH_ENABLE_MPS_FALLBACK=1 python src/play.py

When running this command, you will be prompted with the controls. Press enter to start:

Then the game will be start:

• To control the silver car at the top screen use the arrow keys.
• To control the blue car at the bottom use the WASD keys.

---

Training

Multiverse comprised two models:

• Denoiser - a world model that simulates a game
• Upsampler - a model which takes the frames from the denoiser and increases their resolution

Denoiser training

1. Download the dataset

Download the Denoiser's training set from 🤗 Huggingface.

2. Process data for training

Run the command:

python src/process_denoiser_files.py <folder_with_dataset_files_from_step_one> <folder_to_store_processed_data>

3. Edit training configuration

Edit config/env/racing.yaml and set:

• path_data_low_res to <folder_to_store_processed_data>/low_res
• path_data_full_res to <folder_to_store_processed_data>/full_res

Edit config/training.yaml to train the denoiser:

train_model: denoiser

4. Launch training run

You can then launch a training run with python src/main.py.

Upsampler training

1. Download the dataset

Download the Upsampler's training set from 🤗 Huggingface.

2. Process data for training

Run the command:

python src/process_upsampler_files.py <folder_with_dataset_files_from_step_one> <folder_to_store_processed_data>

3. Edit training configuration

Edit config/env/racing.yaml and set:

• path_data_low_res to <folder_to_store_processed_data>/low_res
• path_data_full_res to <folder_to_store_processed_data>/full_res

Edit config/training.yaml to train the denoiser:

train_model: upsampler

4. Launch training run

You can then launch a training run with python src/main.py.

---

Datasets

1. We've collected over 4 hours of multiplayer (1v1) footage from Gran Turismo 4 at a resolution of 48x64 (per players): 🤗 Huggingface link.

2. A sparse sampling of full resolution, cropped frames, are availabe in order to train the upsampler at a resolution of 350x530: 🤗 Huggingface link.

The datasets contain a variety of situations: acceleration, braking, overtakes, crashes, and expert driving for both players. You can read about the data collection mechanism here

Note: The full resolution dataset is only for upsampler training and is not fit for world model training.

---

Outside resources

• DIAMOND - https://github.com/eloialonso/diamond
• AI-MarioKart64 - https://github.com/Dere-Wah/AI-MarioKart64

---

Cloud Gaming Server

This project now includes a WebSocket-based cloud gaming server that allows you to play the game through a web browser.

Using Docker (Recommended for GPU Servers)

The easiest way to deploy the cloud gaming server on a machine with an NVIDIA GPU is using Docker:

# Build the Docker image
docker build -t ai-game-multiverse .

# Run the container with GPU support
docker run --gpus all -p 8080:8080 ai-game-multiverse

Then access the web interface at http://yourserver:8080

Features

• Web-based interface accessible from any modern browser
• Real-time streaming of AI-generated game frames
• Keyboard and mouse controls
• Multiple scene selection
• WebSocket communication for low-latency interaction

Usage

1. Access the web interface at http://yourserver:8080
2. Click "Connect" to establish a WebSocket connection
3. Select a scene from the dropdown
4. Click "Start Stream" to begin streaming frames
5. Use WASD keys for movement, Space for jump, Shift for attack
6. Mouse controls camera view (click on game area to capture mouse)

Note: The server requires an NVIDIA GPU for optimal performance with the AI models. Without a suitable GPU, it will fall back to using simple placeholder frames.