$e_t(s_t,a) = {\theta ^*_a}^T x(s_t)+\eta_t$ does not hold. Thus, we propose learning a set of features $\phi(s_t)$ that are realizable, i.e., $e(s_t,a) = {\theta ^*_a}^T \phi(s_t)$, using neural networks (using the activations of the last layer as features). A practical challenge here is that the features must be fixed over time when used by the contextual bandit, while the activations change during optimization. We therefore follow a batch-updates framework [@levine2017shallow; @riquelmedeep], where every few steps we learn a new contextual bandit model that uses the last layer activations of the AEN as features.

**Input:** $\epsilon,\beta,\ell,\lambda,C,L,N$ Initialize AEN and DQN with random weights $\omega, \theta$ respectively, and set target networks $Q^-,E^-$ with a copy of $\theta,\omega$ Define $\phi(s) \leftarrow \mbox{LastLayerActivations}(E(s))$ Initialize Replay Memory D to capacity N $a_t$ = ACT$(s_t,Q,E^-,V^{-1},\epsilon,\ell,\beta)$ Execute action $a_t$ and observe $\{r_t, e_t, s_{t+1}\}$ Store transition $\{s_t,a_t,r_t,e_t,s_{t+1}\}$ in D Sample transitions $\{s_j,a_j,r_j,e_j,s_{j+1}\}_{j=1}^m \in 
    D$ $y_j = \text{Targets}\left(s_{j+1},r_j,\gamma,Q^-,E^-,V^{-1},\beta,\ell\right)$ $\theta= \theta - \nabla_\theta \sum_j \left( y_j - Q(s_j,a_j;\theta)\right)^2$ $\omega = \omega - \nabla_\omega \sum_j\left( e_j- \text{E}(s_j,a_j;\omega)\right)^2$ If [$\left(t \text{ mod } C\right) = 0$]{} : $Q^-\gets Q$ If [$\left(t \text{ mod } L\right) = 0$]{} : $E^-,V^{-1} \gets $AENUpdate($E,\lambda,D$)

$A'\leftarrow \{a: E(s)_a - \sqrt{\beta \phi(s)^TV_a^{-1}\phi(s)}  < \ell  \}$ With 