We propose Bayesian Deep Q-Network  (BDQN), a  practical Thompson sampling based Reinforcement Learning (RL) Algorithm. Thompson sampling allows for targeted exploration in high dimensions through posterior sampling but is usually computationally expensive. We address this limitation by introducing uncertainty only at the output layer of the network through a Bayesian Linear Regression (BLR) model, which can be trained with fast closed-form updates and its samples can be drawn efficiently through the Gaussian distribution. We apply our method to a wide range of Atari Arcade Learning Environments. Since BDQN carries out more efficient exploration, it is able to reach higher rewards substantially faster than a key baseline, DDQN.
Using Bayesian regression to estimate the posterior over Q-functions and deploy Thompson Sampling as a targeted exploration strategy with efficient trade-off the exploration and exploitation The authors propose a new algorithm for exploration in Deep RL where they apply Bayesian linear regression with features from the last layer of a DQN network to estimate the Q function for each action. The authors describe how to use Bayesian neural networks with Thompson sampling for efficient exploration in q-learning and propose an approach that outperforms epsilon-greedy exploration approaches.