all $i$. Then the Q function is updated: $$\begin{aligned}
    &Q (x_{i,k},m_k,u_{i,k}) \leftarrow Q (x_{i,k},m_k,u_{i,k}) + \eta \delta_{i,k},\forall i,\\
    &\delta_{i,k} = l_{i,k} + \gamma \min_u Q(x_{i,k+1},m_{k+1},u) - Q(x_{i,k},m_k,u_{i,k}).\end{aligned}$$

#### Example

In this example, we consider $M=50$ agents in the system. Only one agent is infected in the beginning. The run-time cost is the same as in the example in the *distributed optimal control* section, i.e., $l_{i,k} = x_{i,k+1} + \alpha \exp (\frac{1}{u_{i,k}-1})$ where $\alpha$ is chosen to be $1$. For simplicity, the action space is discretized to be $\{0, 1/M, 10/M\}$, called as low, medium, and high. Hence the Q function can be stored as a $2\times M\times 3$ matrix. In the learning algorithm, the learning rate is set to $\eta = 1$. The exploration rate is set to decay in different episodes, i.e., $\epsilon = 0.5(1-E/\max E)$ where $E$ denotes the current episode and the maximum episode is $\max E = 200$. The Q function is initialized to be $10$ for all entries. Three different cases are considered. For each case, we illustrate the Q function learned after 200 episodes as well as the system trajectories for episodes $10, 20,\ldots, 200$, blue for earlier episodes and red for later episodes. 