of these methods is intractable for high dimensional RL.

Bootstrapped DQN {#section:bootstrap}
----------------

Inspired by PSRL, but wanting to reduce computational cost, prior work developed approximate methods. @osband2014generalization proposed randomized least-square value iteration for linearly-parameterized value functions. Bootstrapped DQN @osband2016deep applies to $Q$-functions parameterized by deep neural networks. Bootstrapped DQN (@osband2016deep) maintains a $Q$-ensemble, represented by a multi-head neural net structure to parameterize $K \in \mathbb{N}_+$ $Q$-functions. This multi-head structure shares the convolution layers but includes multiple “heads”, each of which defines a $Q$-function $Q_k$.

Bootstrapped DQN diversifies the $Q$-ensemble through two mechanisms. The first mechanism is independent initialization. The second mechanism applies different samples to train each $Q$-function. These $Q$-functions can be trained simultaneously by combining their loss functions with the help of a random mask $m_{\tau} \in \mathbb{R}_+^K$ $$L = \sum\nolimits_{\tau \in B_{\mathrm{mini}} }\sum\nolimits_{k=1}^K  m_{\tau}^k \cdot ( Q^k(s, a; \theta)  - y_{\tau}^{Q_k})^2,$$ where $y_{\tau}^{Q_k}$ is the target of the $k$th $Q$-function. Thus, the transition $\tau$ updates $Q_k$ only if $m_{\tau}^k$ is nonzero. To avoid the overestimation issue in DQN, bootstrapped DQN calculates the target value $y_{\tau}^{Q_k}$ using the approach of Double DQN (@van2016deep), such that the current $Q_k(\cdot; \theta_t)$ network determines the optimal action and the target network 