TheoVincent/Atari_EauDeQN

Sparse model parameters on 10 Atari games

The sparse model parameters were obtained with EauDeQN and PolyPruneQN leading to EauDeDQN and PolyPruneDQN in an online scenario, and leading to EauDeCQL and PolyPruneCQL in an offline scenario 🎮 While PolyPruneQN applies a fixed polynomial pruning schedule to reach a final sparsity level of 95%, EauDeQN prunes the network parameters at the agents learning pace 🪡

We also release the model parameters for the dense approach using DQN and CQL 🏋️ The online training were made for 40M frames and the offline training were made for 50 $\times$ 62 500 gradient steps ⏱️. We used the CNN architecture, where the number of neurons of the first linear layer is reported as "Feature Size":

Training type	Feature Size:	32 (Small)	512 (Medium)	2048 (Large)
Online	EauDeDQN	✅	✅	✅
Online	PolyPruneDQN	✅	✅	✅
Online	DQN (dense)	✅	✅	✅
Offline	EauDeCQL	✅		✅
Offline	PolyPruneCQL	✅		✅
Offline	CQL (dense)	✅		✅

5 seeds are available for each configuration which makes a total of 750 available models 📈.

The evaluate.ipynb notebook contains a minimal example to evaluate to model parameters 🧑‍🏫 It uses JAX 🚀 The hyperparameters used during training are reported in online_config.json and offline_config.json 🔧

The training code is available soon ⏳

Model sparsity & performances

EauDeDQN and EauDeCQL achieve high sparsity while keeping performances high. Published at RLDM✨ List of Atari games BeamRider, MsPacman, Qbert, Pong, Enduro, SpaceInvaders, Assault, CrazyClimber, Boxing, VideoPinball.

The episodic returns and lenghts are available in the evaluations folder 🔬

User installation

Python 3.11 is recommended. Create a Python virtual environment, activate it, update pip and install the package and its dependencies in editable mode:

python3 -m venv env
source env/bin/activate
pip install --upgrade pip setuptools wheel
pip install -r requirements.txt

Citing Eau De Q-Network

@inproceedings{Vincent_CRLDM_2025,
    title={Eau De $ Q $-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning},
    author={Vincent, Th{\'e}o and Faust, Tim and Tripathi, Yogesh and Peters, Jan and D'Eramo, Carlo},
    booktitle={Conference on Reinforcement Learning and Decision Making (RLDM)},
    year={2025}
}