and moves to state $s_{t+1}$, which is sampled from $\mathcal{P}(\cdot, s_t, a_t)$.

The agent’s goal is to learn a policy that maximizes the expected reward $$\arg\max_\pi \mathbb{E}[R_T] = \arg\max_\pi \mathbb{E}\left[\sum_{t=1}^T\gamma^t r_t  \biggm\lvert a_t \sim \pi(s_t), \, s_{t+1} \sim\mathcal{P}(\cdot, s_t, a_t)\right] ,$$ where $T$ is the length of the episode.

Rather than supervised learning, in which ground truth labels are known during training, such knowledge is absent in RL. Instead, the reward signal is the driving force of the learning. This weaker form of supervision signal makes RL systems take longer to train comparing to supervised learning algorithms. However, it is applicable to many scenarios where supervised learning is not an available approach. In recent years, RL has been an active research area in the machine learning community, and some dramatic successes demonstrated its potential, e.g., [@silver2016mastering].

### $Q$ learning and Deep $Q$ Learning {#sec:dqn}

In reinforcement learning, the $Q$ value function $Q^\pi:\mathcal{S}\times \mathcal{A} \rightarrow \mathbb{R}$ of a policy $\pi$ corresponds to the expected reward given the current state and the current chosen action, where the agent follows $\pi$ after taking the action. By definition, the $Q$ function satisfies a recursive relation, known as Bellman equation[@sutton2018reinforcement]: $$Q^\pi(s, a) =\mathcal{R}(s, a) + 