---
abstract: 'We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark.'
author:
- |
    Richard Y. Chen\
    OpenAI\
    `richardchen@openai.com`\
    Szymon Sidor\
    OpenAI\
    `szymon@openai.com`\
    Pieter Abbeel\
    OpenAI\
    University of California, Berkeley\
    `pabbeel@cs.berkeley.edu`\
    John Schulman\
    OpenAI\
    `joschu@openai.com`\
bibliography:
- 'ref.bib'
title: 'UCB Exploration via $\boldsymbol{Q}$-Ensembles'
---

Introduction
============

Deep reinforcement learning seeks to learn mappings from high-dimensional observations to actions. Deep $Q$-learning (@mnih2015human) is a leading technique that has been used successfully, especially for video game benchmarks. However, fundamental challenges remain, for example, improving sample efficiency and ensuring convergence to high quality solutions. Provably optimal solutions exist in the bandit setting and for small MDPs, and at the core of these solutions are exploration schemes. However these provably optimal exploration techniques do not extend to deep RL in a straightforward way.

Bootstrapped DQN (@osband2016deep) is a previous attempt at adapting a theoretically verified approach to deep RL. In particular, it draws inspiration from *posterior sampling for reinforcement 