successful exploration in some problems, a major drawback is that this exploration bonus does not depend on the rewards, so the exploration may focus on irrelevant aspects of the environment, which are unrelated to reward.

Bayesian reinforcement learning {#section:bayesian}
-------------------------------

Earlier works on Bayesian reinforcement learning include @dearden1998bayesian [@dearden1999model]. @dearden1998bayesian studied Bayesian $Q$-learning in the model-free setting and learned the distribution of $Q^*$-values through Bayesian updates. The prior and posterior specification relied on several simplifying assumptions, some of which are not compatible with the MDP setting. @dearden1999model took a model-based approach that updates the posterior distribution of the MDP. The algorithm samples from the MDP posterior multiple times and solving the $Q^*$ values at every step. This approach is only feasible for RL problems with very small state space and action space. @strens2000bayesian proposed posterior sampling for reinforcement learning (PSRL). PSRL instead takes a single sample of the MDP from the posterior in each episode and solves the $Q^*$ values. Recent works including @osband2013more and @osband2016posterior established near-optimal Bayesian regret bounds for episodic RL. @sorg2012variance models the environment and constructs exploration bonus from variance of model parameters. These methods are experimented on low dimensional problems only, because the computational cost 