\mathbb{E}_{s' \sim\mathcal{P}(\cdot, s, a),\, a' \sim \pi(s')} \left [\gamma Q^\pi(s', a')\right].\notag$$ A policy $\pi^*$ maximizing $Q^\pi$ for every $s\in \mathcal{S},\, a\in\mathcal{A}$ yields the optimal $Q$ function, denoted $Q^*$, whose corresponding Bellman equation is $$Q^*(s, a) =\mathcal{R}(s, a) + \mathbb{E}_{s' \sim\mathcal{P}(\cdot, s, a)} \left [\gamma \max_{a'} Q^*(s', a')\right]. \label{eq:bellman}$$ In words, equation  means that the current expected reward equals the sum of the immediate reward and the best (over choice of action) possible expected reward of the next state.

In $Q$ learning [@watkins1992q], the $Q$ function is iteratively updated, in order to have the Bellman equation satisfied: $$Q(s, a) \leftarrow Q(s, a)  + \alpha \left[\mathcal{R}(s, a) + \gamma \max_{a'} Q(s', a') - Q(s, a)  \right].\notag$$

Deep $Q$ network (DQN, [@mnih2015human]) is arguably the first major milestone in utilizing deep neural networks for reinforcement learning. It is an instance of Fitted $Q$ Iteration [@ernst2005tree], where the $Q$ function is represented by a neural network (DQN - deep $Q$ network), parametrized by $\theta$. The net is trained to minimize the squared difference between the left and right hand sides of the Bellman Equation $$L_{DQN}(\theta) = \left( \left(\mathcal{R}(s, a) + \gamma \max_{a'} Q(s', a'; \hat{\theta})\right) - Q(s, a; \theta)\right)^2, \label{eq:loss}$$ where $Q(\cdot, \cdot; \hat{\theta})$ 