Solutions to Markov Decision Processes (MDP) are often very sensitive to state transition probabilities. As the estimation of these probabilities is often inaccurate in practice, it is important to understand when and how Reinforcement Learning (RL) agents generalize when transition probabilities change. Here we present a new methodology to evaluate such generalization of RL agents under small shifts in the transition probabilities. Specifically, we evaluate agents in new environments (MDPs) in the vicinity of the training MDP created by adding quantifiable, parametric noise into the transition function of the training MDP. We refer to this process as Noise Injection, and the resulting environments as $\delta$-environments. This process allows us to create controlled variations of the same environment with the level of the noise serving as a metric of distance between environments. Conventional wisdom suggests that training and testing on the same MDP should yield the best results. However, we report several cases of the opposite—when targeting a specific environment, training the agent in an alternative noise setting can yield superior outcomes. We showcase this phenomenon across $60$ different variations of ATARI games, including PacMan, Pong, and Breakout.

§ INTRODUCTION

Markov Decision Processes (MDPs) are a well-established formulation to model and solve sequential decision-making problems [6, 28]. A MDP is typically described by a State Space $\mathcal{S}$, an Action Space $\mathcal{A}$, a Transition Function $\mathcal{T}$ that specifies the transition probabilities between states given an action, and a Reward Function $\mathcal{R}$ denoting reward for every state-action pair [9]. In practice, these parameters and transition probabilities are assumed known or estimable with reasonable precision [5, 15]. However, several works have showcased the difficulty in accurately estimating transition probabilities [1, 18, 13], and their large impact on the solution which often deteriorates as transition probabilities are changed [35, 26, 32, 24].

In stark contrast to these works, here we report several cases of the opposite phenomenon—we show that a shift in the transition probabilities between the training and testing environments can actually help under certain conditions. This observation is in contrast to conventional wisdom which suggests that the best approach to perform well on a target MDP is to train an RL agent on the same target MDP.

ATARI games modified with Noise Injection. (a) In the original Target Environment($\mathcal{M}_T$), when the agent (PacMan) moves right, PacMan moves right with probability $1.0$. Noise Injection allows us to create multiple worlds in the vicinity of this environment by adding controlled Gaussian noise ($\delta$) to the original Transition Function ($T$). When the agent takes the action right in these $\delta-$environments, with a low probability the game may transition to a state which would not be possible in non-noisy PacMan. For brevity, we refer to these transitions as non-standard transitions which are $0$ probability in the original Target, but are now possible. Experiments with noise injection are presented on three ATARI games—(b) PacMan, (c) Pong, and (d) Breakout. (e) We compare two agents with these environments—a Learnability agent trained and tested on the same target environment ($\mathcal{M}_\delta)$, and a Generalization agent trained on a different MDP ($\mathcal{M}_T$) and tested on $\mathcal{M}_\delta$.

We explored zero-shot policy transfer where a policy trained in one environment is tested on a different environment. We extend past works which focus on uncertainty in the transition probabilities [26, 24, 15], and propose a novel framework for studying policy transfer in environments with controlled, quantifiable distribution shifts in the transition probabilities.

Our framework introduces these shifts by pre-computing the transition function of an MDP, and adding small Gaussian noise to its entries, as shown in Fig. <ref>(a). For brevity,  we refer to this approach as Noise Injection and the resulting new MDPs as $\delta$-environments. Starting with an environment ($\mathcal{M}_T$), noise is sampled and added to it to obtain the new MDP ($\mathcal{M}_\delta$). Noise addition introduces several non-standard transitions, which had $0$ probability in the original MDP as shown in Fig. <ref>(a). Multiple such environments can be created by sampling noise, and the distance between these worlds can be quantified by the added noise. This approach allows us to create multiple worlds in the vicinity of a target MDP, with quantitative control over the variations in the transition probabilities. An increase in the standard deviation of the Gaussian noise results in increasingly perturbed MDPs. We report experiments with Noise Injection on multiple domains across three ATARI games—PacMan, Pong, and Breakout as shown in Fig. <ref> (b), (c), and (d), respectively.

To study policy transfer we define two agents as shown in Fig. <ref>(e). Firstly, a Learnability Agent which is trained and tested on the same target MDP ($\mathcal{M}_\delta$), and a Generalization Agent which is trained on a different environment($\mathcal{M}_T$) but tested on the target MDP ($\mathcal{M}_\delta$). Conventional wisdom suggests that agents trained on $\mathcal{M}_\delta$ will outperform agents trained on $\mathcal{M}_T$ when evaluated on $\mathcal{M}_\delta$. However, our study across $60$ MDPs built on ATARI games reveals a surprising finding—there are several cases where the Generalization Agent ($\mathcal{G}_T$) outperformed the Learnability Agent ($\mathcal{L}_\delta$).

We validated that this finding extends beyond our setup of noise injection and $\delta$-environments, and also holds true for target MDPs differing in terms of the stochasticity of game elements. This includes variations in the distribution of the Ghost for PacMan, and the computer bar in the Pong. For brevity, we refer to these as semantic variations in MDPs. More details on their construction are provided in Sec. <ref>. We observed the same phenomenon in semantic variations as well—at times, training an agent on a different MDP resulted in better performance than training on the target MDP itself.

Finally, to better understand this phenomenon we analyzed the exploration patterns of the Learnability ($\mathcal{L}_\delta$) and Generalization Agent ($\mathcal{G}_T$) agents, and the corresponding policies learned by them. Our analyses revealed that $\mathcal{L}_\delta$ agents outperformed $\mathcal{G}_T$ agents only in scenarios where they were able to explore a unique, significantly larger set of State-Action pairs as compared to the $\mathcal{G}_T$ agents. In contrast, when when there were no differences in their exploration, the performance of the $\mathcal{G}_T$ aligned or exceeded that of the $\mathcal{L}_\delta$ agent.

§ PRELIMINARIES: REINFORCEMENT LEARNING

Similar to [9], our work considers Reinforcement Learning (RL) as a group of algorithms designed to solve problems formulated as Markov Decision Processes (MDPs). A Markov Decision Process is characterized by the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \lambda)$, representing the collection of potential world states ($\mathcal{S}$), space of actions ($\mathcal{A}$), the transition function ($\mathcal{T} : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{P}(\mathcal{S})$), the reward function ($\mathcal{R} : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R}$), and a discount factor ($0 < \lambda \leq 1$). The objective is to identify policies ($\pi : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R}$) that maximize cumulative rewards.

Q-learning [34] and SARSA [29] are two algorithms to learn such policies.
Both Q-Learning and SARSA algorithms update the Q-values of state-action pairs, but they differ in their approaches. Q-Learning focuses on the maximum expected future rewards, and updates Q-values using the formula:

\begin{equation}
Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right]
\end{equation}

where $\alpha$ is the learning rate, $\gamma$ is the discount factor, and $s, s', a, a', r$ represent the current state, next state, current action, next action, and immediate reward, respectively.

On the other hand, SARSA updates Q-values based on the actual policy's actions with the formula:

\begin{equation}
Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma Q(s', a') - Q(s, a) \right]
\end{equation}

Here the update incorporates both immediate rewards and the Q-value of the actual next action taken.

Agents need to balance two critical aspects: exploration and exploitation. Exploration involves trying potentially less optimal actions to understand the environment better. Conversely, exploitation means choosing actions known to yield high rewards. We report results with the Boltzmann and the $\epsilon$-greedy exploration strategies.
Boltzmann exploration determines the probability of selecting an action as follows:

\begin{equation}
Pr_q(a) = \frac{e^{Q(s,a)/\tau}}{\sum_{a'}e^{Q(s,a')/\tau}}
\end{equation}

The constant $\tau$ is referred to as the temperature. On the other hand, the $\epsilon$-greedy strategy is simpler and more direct—the agent selects a random action with probability $\epsilon$, and the action with the highest Q-value with probability $1-\epsilon$.

Variations for PacMan.(a)
Game dynamics when the agent picks the action right in a standard, non-noisy MDP for the v3 Grid. Ghosts follow a uniform distribution over possible moves and move up or right with an equal probability of $0.5$. This is referred to as a RandomGhost. (b) Grid variations for PacMan—v2, v3, and v4. These grids vary in grid size, positions of walls, and positions of food pellets. v2, v3, and v4 are designed to be increasingly hard. (c) Semantic variations involves changing the distribution of stochastic game elements. Here, a FollowingGhost is depicted which has a higher probability of taking a move which brings it closer to the PacMan ($0.8$). (d) Noise injected MDP generated by adding gaussian noise to the standard Transition Function. Alongside states reachable by the ghost taking a legal move, non-standard transitions now become possible which result in the game reaching states otherwise unreachable—ghost and PacMan arrive to the right hand corners of the grid in a single step with a non-zero probability.

§ RELATED WORKS

§.§ Generalization in Reinforcement Learning

Generalization in Reinforcement Learning (RL) aims to learn policies that perform well in novel, unseen environments at the time of deployment [19, 25, 27, 14, 7]. Recent years have seen several generalization benchmarks, which includes variations in the state space [16], dynamics [12], observation [37], reward function [3], and new game levels [17], among others. Broadly, these works can be divided into two categories. Firstly, Procedural Content Generation which involves creation of diverse and dynamic environments based on algorithms driven by a seed value. Examples of this include OpenAI Procgen benchmark [10] and the Distracting Control Suite [31]. Secondly, recent work on Controllable Environments [19] allows explicit manipulation of individual components as parameters of the environment, providing more granular control. CausalWorld [2], RWRL [12], Alchemy [33], Meta-world [36] are examples of such parametric worlds.
A major drawback in these benchmarks is the difficulty in concretely quantifying the distribution shifts between the train and test MDPs. In contrast, our methodology allows for a concrete and parametric definition for this distribution shift, and allows explicit modification of transition probabilities.

§.§ Inspiration from controlled analysis of generalization behaviour in visual recognition

We draw inspiration from work studying generalization under controlled, quantifiable distribution shifts. This includes shifts in 3D rotations [23, 22], category-viewpoint combinations [20], incongruent scene context [8], novel light and viewpoint combinations [30], object materials [21], and non-canonical viewpoints [4], among others. Inspired by these, we characterize generalization in RL through carefully constructed environments with a consistent metric defining the distribution shifts between environments.

§ GENERATING MDPS FOR INVESTIGATING GENERALIZATION

2*2.5cmATARI Game     2*Grid Variations     2*Noise Injected Variations     2*Semantic Variation     2*Total

4*PacMan         $\delta = 0$ (No Noise)    RandomGhost

v2, v3, v4     $\delta \sim \mathcal{N}(0,0.1)$     FollowingGhost ($p=0.3,0.6$)      33

$\delta \sim \mathcal{N}(0,0.5)$     TeleportingGhost ($p=0.5,0.2$)

4*Pong         $\delta = 0$ (No Noise)      RandomPaddle

p1, p2      $\delta \sim \mathcal{N}(0,0.1) $     FollowingPaddle ($p=0.3,0.6$)     18

$\delta \sim \mathcal{N}(0,0.5)$

4*Breakout          $\delta = 0$ (No Noise)

b1, b2, b3     $\delta \sim \mathcal{N}(0,0.1)$     -     9

$\delta \sim \mathcal{N}(0,0.5)$

Oveview of experimental protocol. Our experiments include multiple variations of three ATARI games—PacMan, Pong, and Breakout. For each game, we have multiple grid variations of increasing difficulty. When introducing variations in these grids with noise injection, we report results for two levels of added noise—a low-noise setting: $\delta \sim \mathcal{N}(0,0.1)$, and a high-noise setting: $\delta \sim \mathcal{N}(0,0.5)$. Furthermore, for each grid we introduce further variations by modifying the distribution of the stochastic game element (ghost in PacMan, and the computer paddle in Pong). In all, we report results on $60$ MDPs across these games.

We created $60$ different MDPs across three ATARI games (PacMan, Pong, and Breakout) by varying grid layouts, distributions defining the stochasticity of different game elements, and modifying transition probabilities using Noise Injection (See Fig. <ref> and Table. <ref>). Here we outline these variations.

§.§ Domains

We implemented all three ATARI games from scratch, building on the Berkeley PacMan Projects [11]. PacMan was modelled as an MDP characterized by the tuple $(\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \lambda)$. Below we define these:

State ($s$) and State Space ($\mathcal{S}$): We represented a grid of size $M \times N$ as a matrix of the same shape with the entries corresponding to the game element occupying the position in the grid—p (PacMan), g (Ghost), f (Food), w (Wall), or e (Empty). States were computed by unrolling this matrix into a vector of length $M*N$. The state space $\mathcal{S}$ refers to the set of all possible states.

Action Space ($\mathcal{A}(s)$): Set of legal actions PacMan could take in state $s$. PacMan can move Left, Right, Up, or Down but not enter walls. Thus, when the PacMan is at the top left position the set of legal actions was only Right, Down.

Transition Matrix ($\mathcal{T}(s_i, a, s_j)$): Probability of moving to state $s_j$ if the agent took action $a$ at state $s_i$ (see Fig. <ref>(a)).

Reward Function ($\mathcal{R}(s)$): PacMan received +20 for eating a food pellet, -1 for every time step, -200 when it was killed, and +500 for finishing the game.  [9].

Game Stochasticity: The motion of PacMan is deterministic—a Left action (if legal) will ensure that PacMan moves left. However, ghosts move stochastically according to a pre-fixed distribution. For instance, a RandomGhost moves in all directions with equal probability (accounting for walls). Thus, the game is non-deterministic.

MDPs for Pong and Breakout are defined analogously. For additional details, please refer to Supplementary Sec. <ref>.

§.§ Grid Variations of ATARI games

We build on past work [7], and train RL agents on the v2, v3, v4 grids for PacMan as depicted in Fig. <ref>(b). For Pong and Breakout we report results on two (p1 and p2) and three grids (b1, b2, and b3) respectively (see Supplementary Fig. <ref> and Fig. <ref>). These grids are designed to be increasingly hard for RL agents, and confirm that our findings are not an artefact of a specific ATARI game or grid.

§.§ Noise Injection: Generating new, controlled environments in the vicinity of an MDP

We generate controlled variations of a target MDP by explicitly computing its Transition Function and then adding sampled noise to it. Below we outline this process:

Explicit enumeration of all states:
We first enumerate all possible states by representing the game as a tree, and recursively rolling out all possible actions and moves by the agent and stochastic game elements. For PacMan, these correspond to the PacMan and the ghosts. For Pong, these correspond to the two paddles. For Breakout, the single paddle.

Explicit computation of Transition Function:
With all states enumerated, $\mathcal{T}(s_i,a,s_j)$ can be computed explicitly. As the movement of all elements is independent, $\mathcal{T}(s_i,a,s_j)$ was computed by multiplying the probability of each element moving from its grid-position in $s_i$ to its grid-position in $s_j$.

Creating $\delta-$environments: The new transition function for a $\delta$-enrivonment is denoted $\mathcal{T}_\delta = \mathcal{T} + \delta$, with $\delta \sim \mathcal{N}(\mu, \sigma)$ sampled i.i.d. before every game to ensure randomization. Then, $\mathcal{T}_\delta$ is normalized to ensure all probabilities $p_{i,j}$ for all possible transitions from a given state $s_{i}$ and action $a$ to any successor state $s_{j}$ sum to $1$:

\begin{equation}
\mathcal{T}_\delta(s_{j},a,s_{i}) = \frac{|\mathcal{S}| p_{i,j} + \delta_{i,j}}{|\mathcal{S}| + \sum_{j}\delta_{i,j}}
\label{eq:norm}
\end{equation}

$|\mathcal{S}|$ denotes the number of states, and guarantees the probability of legal successors does not approach $0$ as the state space grows. We investigated two settings—(i) Low-Noise with $\delta \sim \mathcal{N}(0,0.1)$, where some non-standard transitions previously impossible without noise are now possible with a low probability. (ii) High-Noise with $\delta \sim \mathcal{N}(0,0.5)$, where non-standard transitions are possible with higher probability.

§.§ Semantic variations of ATARI games

PacMan Variations: We modified PacMan by changing the distribution of the ghost in three different ways:

* RandomGhost: Let $l_s$ denote possible legal moves for the ghost given state $s$. A RandomGhost samples move $m \sim Unif(l_s)$. Thus, ghost movement is randomly sampled from legal moves with equal probability.

* FollowingGhost ($p$): Ghost follows PacMan with probability $p$—picks the legal move which minimizes the Manhattan Distance between the ghost and PacMan. The ghost picks one of the remaining legal moves with probability ($1-p$)/$(|l_s|-1$), where $|l_s|$ denotes total legal moves. We report results for $p=0.3,0.6$.

* TeleportingGhost ($p$): The ghost picks one of Left, Right, Up, Down with probability $p/4$ each. If the chosen action is not permissible (i.e., it runs into a wall), the ghost teleports to a random position on the grid. With probability $1-p$, the ghost behaves like a RandomGhost.

Pong Variations:
We focus on the single-player Pong—the RL agent controlled one paddle and aimed to outperform a computer-controlled paddle. Variations were introduced by altering the distribution of the computer-controlled paddle:

* RandomPaddle: Next move $m \sim Unif(l_s)$, where $l_s$ denotes set of legal moves.

* FollowingPaddle $(p)$: Computer-controlled paddle follows the ball with probability $p$ (minimizing manhattan distance), and functions as RandomPaddle with probability $1-p$. Setting $p=1$ guarantees the computer never loses. We report results for $p=0.3,0.6$.

§ EXPERIMENTAL DETAILS

§.§ Evaluation metric

We compared the mean reward curve of Learnability and Generalization agents. An agent $\mathcal{G}_T$ is said to generalize well with respect to $\mathcal{M}_\delta$, if its mean reward is as good as the corresponding learnability agent $\mathcal{L}_\delta$.

§.§ Learning algorithm and Evaluation details

Agents are trained with both tabular Q-Learning [34] and SARSA Q-learning [29], using Boltzmann or $\epsilon$-greedy exploration strategies. In particular, we trained agents for $1000$ episodes and averaged results over $500$ trained agents. After every $10$ training episodes, agents were evaluated using $10$ testing episodes. We report the mean reward curves at convergence. Hyperparameters were inherited from past work [9] and are available in the supplement in Sec. <ref>.

Generalization agents can outperform Learnability agents. Results for PacMan v4 grid reporting mean reward as a function of episode number. (a) SARSA agent trained with a Boltzmann exploration strategy. For Target MDPs generated with both high (solid line) and low (line with `x' markers) level noise injection, the Generalization Agent (red) beats the Learnability Agent (green). (b) The same result holds for a SARSA agent trained with the $\epsilon-$greedy exploration strategy. This finding also holds for Q-Learning agents trained with (c) Boltzmann and (d) $\epsilon-$greedy exploration strategies. Noise added to the transition function is sampled $\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings.

Generalization can outperform Learnability across multiple variations of PacMan.
(a) Agents trained on the PacMan v2 grid with the Ghost dynamics set to the RandomGhost setting—the ghost picks a move from all possible legal moves with an equal probability. (b) Agents trained on v2 with a DirectionalGhost with $p=0.3$—ghost has probability of $0.3$ to move in a pre-specified direction (here right), and a probability of $0.7$ of picking one of the remaining moves (each remaining move equally likely). (c) DirectionalGhost with $p=0.6$. (d),(e),(f) Variations with the v3 grid with RandomGhost, DirectionGhost ($p=0.3$) and DirectionalGhost ($p=0.6$) respectively. (g),(h),(i) Same variations with the v4 grid. All experiments are shown for SARSA agents trained with the $\epsilon-$greedy exploration strategy. Generalization agents consistently beat Learnability Agents across multiple Target MDPs which vary in the grid complexity and game stochasticity (ghost behaviour).

Mean Reward Curves for Learnability and Generalization Agents for Pong and Breakout. Performance of SARSA agents trained with a $\epsilon-$greedy exploration strategy on (a) Pong p1 grid, (b) Pong p2 grid, (c) Breakout b1 grid, and (d) Breakout b2 grid. The Generalization Agent beats the Learnability Agent for Target MDPs created with both high and low noise for all grids.

Rewards across semantic variations of PacMan and Pong. (a) Given the target PacMan MDP with the v2 grid and TeleportingGhost, the Generalization trained on the RandomGhost outperformed the Learnability agent that was trained and tested on the same Target MDP (TeleportingGhost). (b) This finding extends to TeleportingGhost and RandomGhost MDPs with the PacMan v3 Grid as well. (c) For the Pong p1 grid, Generalization agents trained on an MDP with DirectionalPaddle performed better on the RandomPaddle MDP during testing, as compared to the Learnability Agent trained and tested on the RandomPaddle MDP. (d) The same finding extends to the p2 grid as well.

Exploration patterns predict the reward gap between $\mathcal{L}_\delta$ and $\mathcal{G}_T$. (a) Reward for agents trained on PacMan v3, where $\mathcal{G}_T$ outperforms $\mathcal{L}_\delta$. (b) The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Each cell corresponds to one S-A Pair, and the color denotes which agents visited this S-A pair. In this case, a negligible fraction of S-A pairs were visited only by one agent (pink). (c) Rewards for agents trained on PacMan v3. Here, $\mathcal{G}_T$ performs worse than $\mathcal{L}_\delta$. (d) Corresponding exploration shows a large fraction of pairs were only visited by the $\mathcal{L}_\delta$ agent. (e) $D_{LG}$ averaged over PacMan grids where $\mathcal{G}_T$ outperformed $\mathcal{L}_\delta$ and vice-versa. (f) $D_{LG}$ averaged over Pong grids.

§ RESULTS

We report findings from the Generalization and Learnability agents trained with the multiple variations of PacMan, Pong, and Breakout as described in Sec. <ref> and Table <ref>.

§.§ There exist several MDPs where Generalization Agents outperform the Learnability Agents.

The mean reward increases with training, as expected (Fig. <ref>). Also, as intuitively expected, agents achieved higher performance under conditions of low noise compared to high noise (Fig. <ref>, compare solid lines versus lines with `x' markers). Less intuitive was the relationship between Generalization and Learnability agents. Fig. <ref>, shows mean reward curves for multiple RL agents trained to solve the PacMan v4 Grid with RandomGhosts, including Q-Learning and SARSA agents trained with the Boltzmann and $\epsilon-$greedy exploration strategies. The Learnability agent was trained on the noise injected target MDP, while the generalization agent is trained on the non-noisy MDP, and then both agents evaluated on the Target noise-injected environment. Intriguingly, the Generalization agents (Fig. <ref>, red) consistently outperformed the Learnability agents (Fig. <ref>, green). This  gap continued until convergence at $1000$ episodes, was observed across both low and high noise levels, and was apparent for SARSA and Q-Learning, for Boltzman and $\epsilon-$greedy exploration strategies.

To assess whether this observation was dependent on the target MDP, we replicated these findings on multiple PacMan grids and stochastic variations. These results are presented in Fig. <ref> for $9$ different target MDPs for Generalization and Learnability agents trained with SARSA and the $\epsilon-$greedy strategy. In several cases, the Generalization agents beat the Learnability agents, for both low and high levels of noise. Corresponding results for agents trained with SARSA + Boltzmann exploration strategy, and for Q-Learning with both $\epsilon-$greedy and Boltzmann exploration strategies are in Supplementary Figures  <ref>,  <ref> and  <ref>.

We also extended these findings to two additional ATARI games to evaluate if these extend to other games. Fig. <ref> reports these findings for Pong and Breakout. We report results for multiple grids for each, for agents trained with SARSA and $\epsilon-$greedy strategy. These experiments confirmed that Generalization agents can outperform Learnability agents consistently across several ATARI environments. Corresponding agents trained with Q-Learning and different exploration strategies are reported in the Supplementary Figs. <ref>,  <ref>,   <ref>, <ref>, <ref>, <ref>, and  <ref>. In these plots we compare generalization and learnability agents of 18 variations for Pong and 9 variations for Breakout.

In sum, there exist several MDPs where it is better to train on a different MDP than the target MDP. To the best of our knowledge, these results provide the first evidence suggesting that training on a different MDP can enable more efficient policy learning than training on the target environment. In doing so, our finding fundamentally challenges our understanding of the generalization capabilities of RL agents under shifts in transition probabilities.

§.§ Generalization agents also thrive in semantically meaningful $\delta$-environments

Results presented so far focused on MDPs generated by noise injection. We further confirmed that analogous trends also hold true for semantic variations of the two ATARI Games (see Sec. <ref>).

We denote these alternate semantic  environments as $\mathcal{M}_{T'}$ (semantic noise), as opposed to $\mathcal{M}_\delta$ in the case of noise injection.

For PacMan, Learnability agents were trained and tested using TeleportingGhosts ($\mathcal{M}_{T'}$), while the Generalization agents were trained with PacMan with RandomGhosts ($\mathcal{M}_{T}$) and then tested on TeleportingGhosts ($\mathcal{M}_{T'}$) (Fig. <ref>a, b).
Even under these semantic noise conditions, Generalization agents outperformed Learnability agents. These results provide compelling evidence suggesting these findings extend beyond the vicinity of MDPs.

In the case of Pong, we report analogous results with $\mathcal{M}_{T'}$ set to FollowingPaddle, and $\mathcal{M}_{T}$ set to RandomPaddle. As shown in Fig. <ref>(c) and (d), Generalization agents also outperformed Learnability agents (by a smaller margin) in both the p1 and p2 grids. These results are for SARSA agents trained with $\epsilon-$greedy exploration strategy. Analogous results for Pong with Q-Learning and other exploration strategies are reported in Supplementary Figs. <ref>,  <ref>,   <ref>, <ref>, <ref>,  <ref>,   <ref>, <ref>

§.§ The exploration patterns of state-action pairs can predict differences between Generalization and Learning agents

So far, we have presented comprehensive analyses with several examples of our main finding—Generalization Agents beating Learnability Agents. Here, we present a potential explanation for this counter-intuitive phenomenon.

Noise Injection modifies MDPs such that non-standard transitions become increasingly probable as noise is added. Intuitively, $\mathcal{L}_\delta$ trained on the noisy environment should be able to explore significantly more states and acquire knowledge about these low-probability transitions. This would enable it to beat $\mathcal{G}_T$, as conventional wisdom suggests. This motivated us to compare the exploration patterns for $\mathcal{L}_\delta$ and $\mathcal{G}_T$ to explain the gap in their performance.

We enumerate all State-Action Pairs, and divide them into three groups—(i) Fraction of state-action (S-A) pairs explored by both agents ($P_{LG}$), (ii) pairs explored only by the Learnability Agent ($P_L$), and (iii) pairs explored only by the Generalization agent ($P_G)$. Thus, $P_{LG} + P_L + P_G = 1$. We denote the feature $D_{LG} = P_L + P_G$, which measures the divergence in the exploration patterns of these two agents.

In Fig. <ref> we visualize $D_{LG}$ for grids where $\mathcal{G}_T$ outperformed $\mathcal{L}_\delta$  agents, and where it did not. Fig. <ref>(a) shows an agent trained with Q-Learning and Boltzmann exploration strategy for the PacMan v3 grid with RandomGhost stochasticity. Here, $\mathcal{G}_T$ beats the $\mathcal{L}_\delta$ agent. The corresponding panel Fig. <ref>(b) depicts $D_{LG}$ visually—each cell of this grid represents an S-A pair, and its color denotes which of the two agents explored the particular S-A pair. We refer to this as the exploration grid for these agents. Here, the exploration grid shows that most S-A pairs were explored by both agents, with no significant differences in their exploration patterns. In contrast, Fig. <ref>(c, d) report these numbers for PacMan v2 which are starkly different. Here, the $\mathcal{G}_T$ agent performs worse than the $\mathcal{L}_\delta$ agent, and the exploration grid reveals there is a very high fraction of S-A pairs explored only by the $\mathcal{L}_\delta$ agent. These figures visually depict the relationship between $D_{LG}$ and the Reward Gap between the two agents which we denote as $R_{LG} = R_L - R_G$.

Fig. <ref>(e) reports the mean value of $D_{LG}$ across all PacMan MDPs created with noise injection. On average, $D_{LG}$ is significantly higher in MDPs where $\mathcal{L}_\delta$ outperformed $\mathcal{G}_T$, than when it did not (two-sided t-test, $p<0.05$). The same result holds true for Pong MDPs as reported in Fig. <ref>(f). Exploration grids and additional results for variations of PacMan, Pong, and Breakout can be found in the supplement in Sec. <ref>.

Next, we conducted a correlation analysis to better understand the impact of exploration patterns on the reward gap between these two agents. The Spearman correlation coefficient between $D_{LG}$ and $R_{LG}$ was found to be $0.43 (p<0.005)$ for PacMan and $0.26 (p<0.005)$ for Pong indicating a moderate positive monotonic relationship between the two variables. Combined, these analyses show that the Reward Gap between these agents is driven by the differences in their exploration patterns under noisy transition probabilities. $\mathcal{L}_\delta$ agents can beat $\mathcal{G}_T$ agents, but only when they successfully explore a large, unique set of State-Action pairs which the $\mathcal{G}_T$ agent is not able to. In the absence of a substantial difference in their exploration patterns, Generalization agents perform better, or as well as, Learnability agents.

§ DISCUSSION

Here, we propose a new methodology to evaluate robustness of RL agents by generating new MDPs in the vicinity of the target with a metric to quantify the distance between these environments. This approach led us to discover a striking phenomenon—sometimes, training an agent on alternative MDPs can lead to outcomes even better than training on the target MDP itself. This phenomenon occurred across multiple algorithms and exploration strategies (Fig. <ref>), grid layouts and game stochasticity (Fig. <ref>), and multiple ATARI games (Fig. <ref>). We also showed that this phenomenon extends beyond Noise Injected environments, and can also occur when semantic changes are introduced in game elements(Fig. <ref>). Finally, we showed that this reward gap between the two agents appears to be driven by the exploration patterns of the agents under different transition probabilities (Fig. <ref>).

This phenomenon fundamentally challenges conventional wisdom regarding generalization under shifts in transition probabilities. Real-world applications often require modeling the environment stochasticity, and understanding the susceptibilities of RL agents under small errors in estimating or modeling this stochasticity is imperative to build robust agents that can be deployed with confidence. Thus, we strongly believe that the tools and analyses presented in this work can benefit the RL, Planning, and Robotics communities by shedding light on the generalization behaviour of RL agents under subtle shifts in transition probabilities.

[1]
Mohammed Abbad, Jerzy A Filar, and Tomasz R Bielecki.
Algorithms for singularly perturbed limiting average markov control problems.
In 29th IEEE Conference on Decision and Control, pages 1402–1407. IEEE, 1990.

[2]
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wüthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer.
Causalworld: A robotic manipulation benchmark for causal structure and transfer learning.
CoRR, abs/2010.04296, 2020.

[3]
Victor Bapst, Alvaro Sanchez-Gonzalez, Carl Doersch, Kimberly L. Stachenfeld, Pushmeet Kohli, Peter W. Battaglia, and Jessica B. Hamrick.
Structured agents for physical construction.
CoRR, abs/1904.03177, 2019.

[4]
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.
Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.
Advances in neural information processing systems, 32, 2019.

[5]
Nicole Bäuerle and Alexander Glauner.
Distributionally robust markov decision processes and their connection to risk measures.
Mathematics of Operations Research, 47(3):1757–1780, 2022.

[6]
Dimitri Bertsekas.
Dynamic programming and optimal control: Volume I, volume 4.
Athena scientific, 2012.

[7]
A. Biedenkapp, H. F. Bozkurt, T. Eimer, F. Hutter, and M. Lindauer.
Dynamic algorithm configuration: Foundation of a new meta-algorithmic framework.
In Proceedings of the Twenty-fourth European Conference on Artificial Intelligence (ECAI'20), June 2020.

[8]
Philipp Bomatter, Mengmi Zhang, Dimitar Karev, Spandan Madan, Claire Tseng, and Gabriel Kreiman.
When pigs fly: Contextual reasoning in synthetic and natural scenes, 2021.

[9]
Thomas Cederborg, Ishaan Grover, Charles Lee Isbell, and Andrea Lockerd Thomaz.
Policy shaping with human teachers.
In International Joint Conference on Artificial Intelligence, 2015.

[10]
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman.
Leveraging procedural generation to benchmark reinforcement learning, 2020.

[11]
John DeNero, Dan Klein, and Pieter Abbeel.
Cs188: Berkeley pacman projects.
http://ai.berkeley.edu/home.html (Spring 2014), 2014.

[12]
Gabriel Dulac-Arnold, Daniel J. Mankowitz, and Todd Hester.
Challenges of real-world reinforcement learning.
CoRR, abs/1904.12901, 2019.

[13]
Eugene A Feinberg and Adam Shwartz.
Handbook of Markov decision processes: methods and applications, volume 40.
Springer Science & Business Media, 2012.

[14]
Angelos Filos, Panagiotis Tigas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal.
Can autonomous vehicles identify, recover from, and adapt to distribution shifts?
CoRR, abs/2006.14911, 2020.

[15]
Vineet Goyal and Julien Grand-Clement.
Robust markov decision processes: Beyond rectangularity.
Mathematics of Operations Research, 48(1):203–226, 2023.

[16]
Danijar Hafner.
Benchmarking the spectrum of agent capabilities.
CoRR, abs/2109.06780, 2021.

[17]
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi.
Illuminating generalization in deep reinforcement learning through procedural level generation.
arXiv preprint arXiv:1806.10729, 2018.

[18]
Suresh Kalyanasundaram, Edwin KP Chong, and Ness B Shroff.
Markov decision processes with uncertain transition rates: Sensitivity and robust control.
In Proceedings of the 41st IEEE Conference on Decision and Control, 2002., volume 4, pages 3799–3804. IEEE, 2002.

[19]
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel.
A survey of generalisation in deep reinforcement learning.
CoRR, abs/2111.09794, 2021.

[20]
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix.
When and how convolutional neural networks generalize to out-of-distribution category–viewpoint combinations.
Nature Machine Intelligence, 4(2):146–153, 2022.

[21]
Spandan Madan, Li You, Mengmi Zhang, Hanspeter Pfister, and Gabriel Kreiman.
What makes domain generalization hard?, 2022.

[22]
Spandan Madan, Tomotake Sasaki, Hanspeter Pfister, Tzu-Mao Li, and Xavier Boix.
Adversarial examples within the training distribution: A widespread challenge, 2023.

[23]
Shanka Subhra Mondal, Zack Dulberg, and Jonathan Cohen.
Generalization to out-of-distribution transformations, 2022.

[24]
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters.
Robust reinforcement learning: A review of foundations and recent advances.
Machine Learning and Knowledge Extraction, 4(1):276–315, 2022.

[25]
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters.
Robust reinforcement learning: A review of foundations and recent advances.
Machine Learning and Knowledge Extraction, 4(1):276–315, 2022.

[26]
Arnab Nilim and Laurent El Ghaoui.
Robust control of markov decision processes with uncertain transition matrices.
Operations Research, 53(5):780–798, 2005.

[27]
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang.
Solving rubik's cube with a robot hand.
CoRR, abs/1910.07113, 2019.

[28]
Martin L Puterman.
Markov decision processes: discrete stochastic dynamic programming.
John Wiley & Sons, 2014.

[29]
Gavin Adrian Rummery and Mahesan Niranjan.
On-line q-learning using connectionist systems.

[30]
Akira Sakai, Taro Sunagawa, Spandan Madan, Kanata Suzuki, Takashi Katoh, Hiromichi Kobashi, Hanspeter Pfister, Pawan Sinha, Xavier Boix, and Tomotake Sasaki.
Three approaches to facilitate invariant neurons and generalization to out-of-distribution orientations and illuminations.
Neural Networks, 155:119–143, 2022.

[31]
Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski.
The distracting control suite – a challenging benchmark for reinforcement learning from pixels, 2021.

[32]
Marnix Suilen, Thiago D Simão, David Parker, and Nils Jansen.
Robust anytime learning of markov decision processes.
Advances in Neural Information Processing Systems, 35:28790–28802, 2022.

[33]
Jane X. Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, H. Francis Song, Gavin Buttimore, David P. Reichert, Neil C. Rabinowitz, Loic Matthey, Demis Hassabis, Alexander Lerchner, and Matthew M. Botvinick.
Alchemy: A structured task distribution for meta-reinforcement learning.
CoRR, abs/2102.02926, 2021.

[34]
Christopher J. C. H. Watkins and Peter Dayan.
Machine Learning, 8(3):279–292, May 1992.

[35]
Huan Xu and Shie Mannor.
Distributionally robust markov decision processes.
Advances in Neural Information Processing Systems, 23, 2010.

[36]
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
CoRR, abs/1910.10897, 2019.

[37]
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martín.
robosuite: A modular simulation framework and benchmark for robot learning.
CoRR, abs/2009.12293, 2020.

§ DOMAINS

We present details for the ATARI PacMan, Pong and Breakout domains.

§.§ PacMan

PacMan is set in a two-dimensional grid that contains food, walls, ghosts, and the PacMan character. The game concludes with a +500 reward when all food pellets are consumed, while encountering a ghost results in a -500 penalty and game over. Each consumed food pellet awards +10 points, and PacMan incurs a -1 penalty for every time step. The available actions for PacMan are moving Up, Down, Right, or Left. The game's state includes the location of PacMan, the position and direction of any ghosts, and the distribution of food pellets. In this iteration of the game, ghosts move according to some distributions.

§.§ Pong

In this one-player version of Pong, the player competes against a computer-controlled paddle. The game is set on a two-dimensional grid, with the player controlling one paddle and the computer controlling the other. The game concludes with a +500 reward when the ball reaches the grid boundaries on the computer controlled paddle side, while if the grid boundary is reached on the agent's side, a -500 penalty is applied and game over. The agent incurs a -1 penalty for every time step. The available actions for the paddles are moving Right and Left or to Stop. The game's state includes the location of the ball and the position and direction of any paddle. In this iteration of the game, the computer controlled paddle moves according to some distribution. Visualizations of the grids are presented in <ref>.

§.§ Breakout

In this version of Breakout, the agent competes against a wall of bricks using a horizontally-moving paddle and a ball. The game is set on a two-dimensional grid, with the agent controlling the paddle located at the bottom of the screen. The objective is to break bricks by hitting them with the ball, which bounces back after each hit. The game concludes with a +500 reward when all bricks are destroyed, but if the ball passes the paddle and reaches the bottom grid boundary, a -500 penalty is applied, resulting in game over. Each hit brick awards +10 points, and the agent incurs a -1 penalty for every time step. The available actions for the agent's paddle are moving Right or Left, or choosing to Stop. The game's state includes the position of the ball, the location of the paddle, and the configuration and status of the bricks. Visualizations of the grids are presented in <ref>.

§ TRAINING PARAMETERS

In our experiments, parameters for Q-Learning and SARSA are inherited by [9]. In particular, $\mathcal{T}=1.5$ $\alpha=0.05$, and $\lambda=0.9$.

§ ADDITIONAL GRAPHS SHOWING THE GENERALIZATION AGENT OUTPERFORMING THE LEARNABILITY AGENT IN NON-SEMANTIC VARIATIONS

This section presents supplementary results showing the Generalization Agent and Learnability Agent behavior for Non-Semantic variations of grids throughout the analyzed domains.

§.§ PacMan

Additional results showing the Generalization Agent and Learnability Agent behaviour in Pacman for grids v2, v3, v4, are presented in the Supplementary figures. In particular, results for SARSA Agent with Boltzmann exploration strategy are presented in <ref>. <ref>, <ref> show Q-learning Agent with Boltzmann and $\epsilon$-greedy exploration strategies respectively.

§.§ Pong

Similarly, for Pong grids p1, p2 results are presented in the Supplementary figures <ref> for SARSA Agent and  <ref>, <ref> for Q-learning Agent.

§.§ Breakout

Analogously, for Breakout grids b1, b2, b3 results are presented in the Supplementary figures <ref>, <ref> for SARSA Agent and  <ref>, <ref> for Q-learning Agent.

§ ADDITIONAL GRAPHS SHOWING THE GENERALIZATION AGENT OUTPERFORMING THE LEARNABILITY AGENT IN SEMANTIC VARIATIONS

In this section we present supplementary results showing the Generalization Agent and Learnability Agent behavior for Semantic variations of grids throughout Pacman and Pong.

§.§ PacMan

The behavior of the Generalization and Learnabilty Agents under semantic variations of PacMan on grids v2, v3, v4 are presented in Supplementary figures <ref> for SARSA Agent and <ref> and <ref> for Q-learning Agent.

§.§ Pong

Similarly, for Pong grids p1, p2 results are presented in the Supplementary figures. In particular, semantic variations featuring Directional Ghost $p=0.3$ are presented in <ref>, <ref> for SARSA Agent and <ref>, <ref> for Q-learning Agent. While semantic variations featuring Directional Ghost $p=0.6$ are shown in <ref>, <ref> for SARSA Agent and <ref>, <ref> for Q-learning.

§ ADDITIONAL GRAPHS SHOWING PATTERNS OF STATE-ACTION PAIRS

This section shows the supplementary results for the exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$) throughout the analyzed domains.

§.§ PacMan

Results of the exploration grid for PacMan v2, v3, v4 are shown in Supplementary figures. In particular, for non-semantic grid variations, <ref> and <ref> report grid exploration graphs for Q-learning Agent and <ref> and <ref> for SARSA Agent. Additionally, for semantic games variations, <ref> and <ref> report grid exploration graphs for Q-learning Agent and <ref> and <ref> for SARSA Agent.

§.§ Pong

Similarly, for pong p1 and p2, <ref>, <ref>, <ref>, and <ref> report grid exploration graphs for non-semantic variations of Q-learning Agent and SARSA Agent respectively, while <ref>, <ref>, <ref>, and <ref> for semantic variations.

§.§ Breakout

For Breakout grids b1,b2, and b3, exploration graphs for non-semantic variations of Q-learning Agent and SARSA Agent are reported in Supplementary figures <ref>, <ref>, <ref>, and <ref>.

Grid variations for Pong.

Grid variations for Breakout.

SARSA Agent with Boltzmann exploration strategy: Results for PacMan v2, v3, v4 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

Q-learning Agent with Boltzmann exploration strategy: Results for PacMan v2, v3, v4 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: Results for PacMan v2, v3, v4 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

SARSA Agent with Boltzmann exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

Q-learning Agent with Boltzmann exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

SARSA Agent with Boltzmann exploration strategy: Results for Breakout b1, b2, b3 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: Results for Breakout b1, b2, b3 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

Q-learning Agent with Boltzmann exploration strategy: Results for Breakout b1, b2, b3 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the non-noisy version of the environment and tested on different level of noise ($\delta \sim \mathcal{N}(0,0.1)$ in Low-Noise and $\delta \sim \mathcal{N}(0,0.5)$ in High-Noise settings).

SARSA Agent with Boltzmann exploration strategy: Results for PacMan v2, v3, v4 grids reporting mean reward as a function of episode number. The agent is trained on the Random Ghost environment and tested on the Teleporting Ghost variation ($p=0.2$, $p=0.5$)

Q-learning Agent with Boltzmann exploration strategy: Results for PacMan v2, v3, v4 grids reporting mean reward as a function of episode number. The agent is trained on the Random Ghost environment and tested on the Teleporting Ghost variation ($p=0.2$, $p=0.5$)

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: Results for PacMan v2, v3, v4 grids reporting mean reward as a function of episode number. The agent is trained on the Random Ghost environment and tested on the Teleporting Ghost variation ($p=0.2$, $p=0.5$)

SARSA Agent with Boltzmann exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

Q-learning Agent with Boltzmann exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

SARSA Agent with Boltzmann exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

Q-learning Agent with Boltzmann exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: Results for Pong p1, p2 grids reporting mean reward as a function of episode number. The agent is trained on the Directional Ghost ($p=0.3$) environment and tested on the Random Ghost variation.

Q-learning Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Up, or Down.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Up, or Down.

SARSA Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Up, or Down.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Up, or Down.

Q-learning Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on Teleporting Ghost variation ($p=0.2$, $p=0.5$) and tested in different environments (reported in the headings). Rows in the right figure represents agent's actions Left, Right, Up, or Down.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on Teleporting Ghost variation ($p=0.2$, $p=0.5$) and tested in different environments (reported in the headings). Rows in the right figure represents agent's actions Left, Right, Up, or Down.

SARSA Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on Teleporting Ghost variation ($p=0.2$, $p=0.5$) and tested in different environments (reported in the headings). Rows in the right figure represents agent's actions Left, Right, Up, or Down.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for PacMan v2, v3, v4 grids, the agent is trained on Teleporting Ghost variation ($p=0.2$, $p=0.5$) and tested in different environments (reported in the headings). Rows in the right figure represents agent's actions Left, Right, Up, or Down.

Q-learning Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

SARSA Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

Q-learning Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on Directional paddle ($p=0.3$ top, $p=0.6$, bottom) variation and tested in the Random paddle environment. Rows in the right figure represents agent's actions Left, Right, Stop.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on Directional Paddle ($p=0.3$ top, $p=0.6$, bottom) variation and tested in the Random Paddle environment. Rows in the right figure represents agent's actions Left, Right, Stop.

SARSA Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on Directional Paddle ($p=0.3$ top, $p=0.6$, bottom) variation and tested in the Random Paddle environment. Rows in the right figure represents agent's actions Left, Right, Stop.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Pong p1, p2 grids, the agent is trained on Directional Paddle ($p=0.3$ top, $p=0.6$, bottom) variation and tested in the Random Paddle environment. Rows in the right figure represents agent's actions Left, Right, Stop.

Q-learning Agent with Boltzmann exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Breakout b1, b2, b3 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

Q-learning Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Breakout b1, b2, b3 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

SARSA Agent with Boltzmann exploration strategy:  The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Breakout b1, b2, b3 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.

SARSA Agent with $\epsilon\text{-}$greedy exploration strategy: The exploration grid visualizing the difference in State-Action (S-A) pairs explored by these agents ($D_{LG}$). Results for Breakout b1, b2, b3 grids, the agent is trained on non-noisy variations of different environments (reported in the headings) and tested in the Low-Noise regime. Rows in the right figure represents agent's actions Left, Right, Stop.


21st International Conference on Few-Body Problems in Physics

11institutetext: Laboratório de Física Teórica e Computacional, Universidade
Cruzeiro do Sul, Rua Galvão Bueno 868, 01506-000 São Paulo, SP, Brazil
22institutetext: Instituto de Física Teórica, Universidade Estadual Paulista,
Rua Bento Teobaldo Ferraz 271, 01140-070 São Paulo, SP, Brazil
33institutetext: Instituto de Física, Universidad de Antioquia, Calle 70, no.
52-21, Medellín, Colombia

# Contemporary continuum QCD approaches to excited hadrons

Bruno El-Bennich 1122<EMAIL_ADDRESS>Eduardo Rojas 33
<EMAIL_ADDRESS>
###### Abstract

Amongst the bound states produced by the strong interaction, radially excited
meson and nucleon states offer an important phenomenological window into the
long-range behavior of the coupling constant in Quantum Chromodynamics. We
here report on some technical details related to the computation of the bound
state’s eigenvalue spectrum in the framework of Bethe-Salpeter and Faddeev
equations.

##  1 Introduction: excited states as an eigenvalue problem

A great deal of research activity in hadron physics is concerned with hadron
structure and revolves around two fundamental questions: what constituents are
the hadrons made of and how does Quantum Chromodynamics (QCD), the strong
interaction component of the Standard Model, produce them? These are simple
questions which, however, may not entail simple answers. To understand the
measurable content of QCD, spectroscopy is a valuable and time-honored tool —
suffice it to mention the inestimable progress made in the computation of
atomic or molecular spectra and subsequent comparison with experiments that
lead to a deeper understanding of Quantum Electrodynamics.

The same is true for QCD: if confinement is related to the analytic properties
of QCD’s Schwinger functions, then light-quark confinement ought to be
understood by mapping out the infrared behavior of the theory’s universal
$\beta$ function. Obviously, this cannot be possibly achieved in perturbation
theory. A nonperturbative continuum approach to QCD is provided by Dyson-
Schwinger equations (DSE) which relate the theory’s $\beta$ function to
experimental observables Bashir:2012fs . Therefore, comparison between DSE
predictions embedded in bound-state calculations and observations of the
hadron mass spectrum as well as of elastic and transition form factors can be
used to study the long-range behavior of QCD’s interaction. As the properties
of excited hadron states are considerably more sensitive to the long-range
behavior of the strong interaction than those of ground states Qin:2011xq ;
Rojas:2013tza ; Rojas:2014aka , excited mesons and nucleons Aznauryan:2012ba
are an important source of information and complement our understanding of the
strong interaction in light mesons ElBennich:2008qa ; Chang:2009ae ;
ElBennich:2012ij ; daSilva:2012gf  and in heavy-light mesons with disparate
energy scales ElBennich:2009vx ; ElBennich:2010ha ; ElBennich:2011py ;
ElBennich:2012tp .

Of course, the properties of these hadrons can only be understood, and the
functional behavior of the $\beta$ function be inferred therefrom, by studying
the quark’s DSE in conjunction with quark-antiquark or three-quark bound-state
equations, the Bethe-Salpeter and Faddeev equations, respectively. Both are
treated as eigenvalue problems which we here exemplify with the relativistic
bound-state equation of pseudoscalar $J^{P}=0^{-}$ mesons. The homogeneous
Bethe-Salpeter equation for this $q\bar{q}$ bound state with relative momentum
$p$ and total momentum $P$ can be generally written as,

$\Gamma_{0^{-}}(p,P)=\int^{\Lambda}_{k}\\!\\!\mathcal{K}(p,k,P)\left[S(k+\eta_{+}P)\,\Gamma_{0^{-}}(k,P)\,S(k-\eta_{-}P)\right]\
,$ (1)

where $S(k\pm\eta_{\pm}P)$ are dressed quark propagators with
$\eta_{+}+\eta_{-}=1$ (NB: in a Poincaré invariant calculation numerical
results are independent of the momentum partition parameter $\eta_{\pm}$). For
the sake of simplicity, we omit flavor and Dirac indices, as the following
discussion is independent of them. In the same spirit, we restrict ourselves
to the rainbow-ladder truncation of the interaction kernel,

$\mathcal{K}(p,k,P)=-\frac{Z_{2}^{2}\,\mathcal{G}(q^{2})}{q^{2}}\,\frac{\lambda^{a}}{2}\gamma_{\mu}T_{\mu\nu}(q)\,\frac{\lambda^{a}}{2}\gamma_{\nu}\
,$ (2)

with the transverse projection operator
$T_{\mu\nu}(q):=g_{\mu\nu}-q_{\mu}q_{\nu}/q^{2}$, $q=p-k$, $Z_{2}$ is the
wave-function renormalization constant and $\lambda^{a}$ are the SU(3) color
matrices in the fundamental representation. Various model ansätze Maris:1997hd
; Maris:1997tm ; Maris:1999nt ; Qin:2011dd  have been proposed for the
effective interaction, $\mathcal{G}(q^{2})$, which emulates the combined
effect of the gluon and quark-gluon vertex dressing functions and most recent
efforts extend these models to include important transverse components of this
vertex beyond the leading truncation Binosi:2014aea . Most importantly, Eq.
(2) satisfies the axialvector Ward-Takahashi identity Bender:1996bb  and
therefore ensures a massless pion in the chiral limit. As we shall see below,
Eqs. (1) and (2) define an eigenvalue problem with physical solutions at the
mass-shell points, $P^{2}=-m^{2}$, where $m$ is the bound-state mass.

The Bethe-Salpeter equation’s Poincaré-invariant solutions can be cast in the
form,

$\Gamma_{0^{-}}\,(p,P)=\gamma_{5}\left[\,i\,\mathbb{I}_{D}E_{0^{-}}(p,P)+/\\!\\!\\!PF_{0^{-}}(p,P)+\
/\\!\\!\\!p(p\cdot
P)\,G_{0^{-}}(p,P)+\sigma_{\mu\nu}\,p_{\mu}P_{\nu}\,H_{0^{-}}(p,P)\,\right]\
,$ (3)

Note that this Euclidean-metric basis,
$\mathcal{A}^{\alpha}(p,P)=\gamma_{5}\big{\\{}i\,\mathbb{I}_{D}$,
$/\\!\\!\\!P$, ${/\\!\\!\\!p}(p\cdot P)$,
$\sigma_{\mu\nu}p_{\mu}P_{\nu}\big{\\}}$, is nonorthogonal with respect to the
Dirac trace. The functions $\mathcal{F}_{0^{-}}^{\alpha}(p,P)=$
$\big{\\{}E_{0^{-}}(p,P),F_{0^{-}}(p,P),G_{0^{-}}(p,P),H_{0^{-}}(p,P)\big{\\}}$
are Lorentz-invariant scalar amplitudes and are extracted from the Bethe-
Salpeter amplitude (3) with appropriate projectors, $P^{a}(p,P)$,111 see,
e.g., Ref. Cobos-Martinez:2010raa  for the derivation of the change of basis
coefficients $P^{\alpha\beta}(p,P)$.

$\tfrac{1}{4}\,P^{\alpha\beta}(p,P)\,\mathrm{Tr}_{\mathrm{D}}\left[\mathcal{A}^{\beta}(p,P)\mathcal{A}^{\gamma}(p,P)\right]=\delta_{\alpha\gamma}\
;\quad P^{\alpha}(p,P)=P^{\alpha\beta}(p,P)\,\mathcal{A}^{\beta}(p,P)\ ,$ (4)

where $\alpha,\beta=1,...,4$ and the projection is given by:

$\mathcal{F}_{0^{-}}^{\alpha}(p,P)=\tfrac{1}{4}\,\mathrm{Tr}_{\mathrm{D}}\Big{[}P^{\alpha}(p,P)\,\Gamma_{0^{-}}(p,P)\Big{]}\
.$ (5)

Using the Bethe-Salpeter equation, this leads to the eigenvalue problem,

$\lambda(P^{2})\,\mathcal{F}_{0^{-}}^{\alpha}(p,P)=\int^{\Lambda}_{k}\\!\\!\mathcal{K}^{\alpha\beta}(p,k,P)\,\mathcal{F}_{0^{-}}^{\beta}(k,P)\
,$ (6)

where $\mathcal{K}^{\alpha\beta}(p,k,P)$ stems from the projection of Eq. (1)
using Eqs. (2) and (5):

$\mathcal{K}^{\alpha\beta}(p,k,P)=-\frac{Z_{2}^{2}}{4}\frac{\mathcal{G}(q^{2})}{q^{2}}\,T_{\mu\nu}(q)\,\mathrm{tr}_{\mathrm{CD}}\\!\
\Big{[}P^{\alpha}(p,P)\,\gamma_{\mu}\,\lambda^{a}S(k_{+})\,\mathcal{A}^{\beta}(k,P)\,S(k_{-})\,\gamma_{\nu}\,\lambda^{a}\Big{]}\
.$ (7)

In Eq. (6), $\lambda(P^{2})$ is a scalar function and the eigenvalue equation
has a solution for every value of $P^{2}$. Typically, iterative eigenvalue
algorithms are employed as only under simplifying assumptions an inversion of
the Bethe-Salpeter problem is possible Lucha:2012ky ; Frederico:2013vga ;
Bhagwat:2007rj . To elucidate the iterative procedure, we simplify Eq. (6)
(see also Ref. Krassnigg:2003wy ), so that:

$\lambda(P^{2})\,|\Phi\rangle=\mathcal{K}(P^{2})\,|\Phi\rangle\ .$ (8)

The kernel, $\mathcal{K}(P^{2})$, has a complete set of real eigenvectors
$\phi_{i}$ with eigenvalues $\lambda_{i}(P^{2})$ which are ordered as
$\lambda_{0}(P^{2})>\lambda_{1}(P^{2})>\lambda_{2}(P^{2})>....>\lambda_{i}(P^{2})$.
Thus, any solution can be written as a linear superposition,

$|\Phi\rangle=\sum_{i=1}^{\infty}a_{i}\,|\,\phi_{i}\rangle\ ,$ (9)

where $a_{i}$ are real constants and the vector must be normalized. To begin
the iterative process, one may “guess” as solution, such as in Eq. (9), for a
given value of $P^{2}$ and $n$ successive actions of the kernel lead to,

$|\,\phi_{n}\rangle:=\,\mathcal{K}^{n}(P^{2})\,|\Phi\rangle=\sum_{i=1}^{\infty}\,\lambda^{n}_{i}a_{i}\,|\,\phi_{i}\rangle=\lambda_{0}^{n}\left[a_{0}\,|\,\phi_{0}\rangle+\sum_{i=1}^{\infty}\left(\frac{\lambda_{i}}{\lambda_{0}}\right)^{n}\\!a_{i}\,|\,\phi_{i}\rangle\right]\
.$ (10)

Since $\lambda_{0}>\lambda_{i}$, the coefficients of $|\,\phi_{i}\rangle$
converge to zero for sufficiently large values of $n$ and therefore the
amplitude $|\,\phi_{n}\rangle$ converges to the ground state amplitude
$|\,\phi_{0}\rangle$:

$|\,\phi_{n}\rangle\overset{n\to\infty}{=}\lambda_{0}^{n}\,a_{0}\,|\,\phi_{0}\rangle\,\simeq\,\lambda_{0}\,\mathcal{K}^{n-1}(P^{2})\,|\Phi\rangle\
.$ (11)

This is the most basic method of computing the largest eigenvalue
$\lambda_{0}(P^{2})$ and its associated eigenvector to any required accuracy
and is referred to as power or von Mises iteration. The trajectory of this
eigenvalue function for a range of $P^{2}$ values yields the ground-state
meson mass, that is one finds $\lambda_{0}(P^{2})=1$ when $P^{2}=-m_{0}^{2}$.

##  2 Gram-Schmidt orthogonalization

A common procedure to extract the wave function of excited states from the
spectrum of the interaction kernel is based on the Gram-Schmidt
orthogonalization process. As mentioned in Section 1, the homogeneous Bethe-
Salpeter equation is an eigenvalue problem, where the largest eigenvalue the
interaction kernel produces corresponds to the ground state. However, the
eigenvalue spectrum is not limited to the ground state and excited states with
smaller eigenvalues can be determined with the same iterative methods
discussed above. As sketched, e.g. in Ref. Krassnigg:2003wy , one chooses a
mass that is larger than the ground state mass, $P^{2}=-m^{2}<-m_{0}^{2}$, and
applies the iterative procedure outlined in Section 1:

  * •

find the largest eigenvalue, $\lambda_{0}$, and the associated eigenvector for
$m^{2}>m_{0}^{2}$ and $\lambda_{0}(m^{2})>\lambda_{0}(m_{0}^{2})$. This is the
unphysical “ground state” at the mass scale $P^{2}=-m^{2}$.

  * •

make again a guess for the Bethe-Salpeter amplitude now projecting out the
eigenvector that pertains to the largest eigenvalue $\lambda_{0}(m^{2})$ in a
first iteration.

  * •

use the eigenvector obtained in the previous step as input for the Bethe-
Salpeter amplitude and project out again the eigenvector associated with
$\lambda_{0}(m^{2})$ in a second iteration; the resulting eigenvector must be
projected as before in a third iteration and so on.

  * •

the iteration converges after the $n$th projection which yields an eigenvalue,
$\lambda_{1}(m^{2})$, with an eigenvector orthogonal to the one associated
with the “ground state” eigenvalue $\lambda_{0}(m^{2})>\lambda_{1}(m^{2})$.

  * •

by varying $P^{2}$ one obtains the mass evolution of the second largest
eigenvalue $\lambda_{1}(P^{2})$, which is exemplified in Figure 1. The
solution for $\lambda_{1}(P^{2})=1$ yields the mass of the first excited
state, $P^{2}=-m_{1}^{2}$ where $m_{1}>m_{0}$, at which the eigenvector is the
state’s Bethe-Salpeter amplitude.

The normalized projection within this Gram-Schmidt procedure is effected by,

$|\tilde{\Phi}\rangle=|\Phi\rangle-\frac{\langle\phi_{0}\,|\,\Phi\rangle}{\langle\phi_{0}\,|\,\phi_{0}\rangle}\
|\phi_{0}\rangle\ ,$ (12)

where $|\Phi\rangle$ is the initial guess for the Bethe-Salpeter amplitude in
Eq. (3) and $|\,\phi_{0}\rangle$ is the vector of the ground-state Bethe-
Salpeter amplitude. In projecting out this ground-state, one must define a
norm in Euclidean space via an inner product, as is evident from Eq. (12).
Within Quantum Field Theory in rainbow-ladder approximation, this product is
defined as:

$\langle\Psi\,|\,\Phi\rangle:=\mathrm{tr}_{\mathrm{CD}}\\!\
\\!\int_{k}^{\Lambda}\\!\overline{\Psi}(k,-P)\,S(k+\eta_{+}P)\,\Phi(k,p)\,S(k-\eta_{-}P)\
.$ (13)

Orthogonality is thus defined by the condition, $\langle\Psi|\Phi\rangle=0$,
which expresses the vanishing of the overlap amplitude at $P^{2}$. Note that
Eq. (13) is not valid for computations beyond the leading approximation, i.e.
the rainbow-ladder truncation.

Figure 1: First two eigenvalue trajectories of the quark-diquark kernel used
in Refs. Cloet:2008re ; Segovia:2015hra  as a function of $\sqrt{-P^{2}}$.
Note that the eigenvector solutions are here differently normalized so that
the eigenvalues are $\lambda_{i}=1/g_{s}^{2}$ instead of $\lambda_{i}=1$ for
$P^{2}=-m_{i}^{2}$; see Ref. Cloet:2008re  for details. The intersections of
the solid horizontal line with the trajectories locates the mass-pole position
of the nucleon and its first excited state identified with the Roper.

##  3 Krylov subspace and Arnoldi iteration

After momentum discretization and Chebyshev expansion of Eq. (6), the
numerical kernel of the Bethe-Salpeter equation (or the Faddeev equation for
the three-body problem) is a non-symmetric matrix of large dimensions and with
some of its eigenvalues close to zero. In general, non-symmetric matrices have
eigenvalues and eigenvectors which are very sensitive to small changes in the
matrix elements due to the lack of symmetries on which traditional methods
rely to ensure numerical stability Press:1992zz . This is particularly true
for radial excitations, where angular dependence encoded in higher Chebyshev
moments contributes in a nontrivial manner. Therefore, we make use of the
numerical ARPACK library Arpack  which is designed to compute a few
eigenvalues and corresponding eigenvectors of a general $n\times n$ matrix by
means of the implicitly restarted Arnoldi method (IRAM). The strength of the
Arnoldi method Arnoldi:1951  lies in the application of the stabilized
orthogonalization algorithm in the Krylov space of a given matrix $K$ to find
its eigenvectors, as will be explained shortly. The most important feature of
this algorithm is its ability to decouple the calculation of the eigenvectors
corresponding to the eigenvalues with the largest absolute value from the
eigenvectors whose eigenvalues have an absolute value close to zero. This is
highly desirable as the calculation of the latter are plagued by numerical
instabilities.

In algebra, the order-$r$ Krylov subspace is generated by an operator whose
representation is given by an $n\times n$ matrix $K=K_{ij}$, and a vector
$\Phi$ of dimension $n$. It corresponds to the linear subspace spanned by
actions of $K$ on $\Phi$:

$\mathcal{S}_{r}\,:=\,\left\\{\Phi,K\Phi,K^{2}\Phi,K^{3}\Phi,....,K^{r-1}\Phi\right\\}\
.$ (14)

The power iteration of $K$ yields the sequence in Eq. (14) and converges to
the eigenvector associated with the largest eigenvalue $\lambda_{0}$. However,
hereby one only makes very limited use of the stored information, as only the
final vector, $K^{r-1}\Phi$, is kept. On the other hand, the basis for the
Krylov subspace is derived from the Cayley-Hamilton theorem which implies that
the inverse of a matrix can be expressed as a linear combination of its
powers. Thus, Krylov subspaces play an important role in contemporary
iterative methods to obtain one or few eigenvalues of large sparse matrices or
to solve large systems of linear equations. Instead of heavy matrix
operations, these methods rely on successive multiplications of vectors by the
matrix, thus forming a Krylov subspace, and then employ the resulting vectors.

The vectors of the Krylov space are initially not ortogonal and usually become
almost linearly dependent due to the properties of the matrix power iteration.
Nonetheless, an orthogonal matrix can be constructed from the basis vectors by
means of the Gram-Schmidt process described in Section 2. This method proves
to be unstable but its shortcoming can be overcome with the Arnoldi iteration
Arnoldi:1951  which uses the stabilized Gram-Schmidt process and can be
applied to general, possibly non-Hermitian matrices. The Arnoldi method
generalizes the Gram-Schmidt process by computing the eigenvalues of the
orthogonal projection of $K$ onto the Krylov subspace, where the projection is
represented by the upper Hessenberg matrix $H_{r}$ Arpack . For Hermitian
(symmetric) matrices, the Arnoldi iteration is analogous to the Lanczos
iteration.

##  4 A note on orthogonality

In the context of Bethe-Salpeter and Faddeev equations, orthogonality is
defined by Eq. (13). In vectorial form, using the shorthand $x=k^{2}$ and
$y=p^{2}$, Eq. (6) can be written as,

$\lambda(y)\,\bm{F}_{R}(y)=\int\\!dx\ \bm{\mathcal{K}}(x,y)\cdot\bm{F}_{R}(x)\
,$ (15)

which in a numerical treatment is evaluated as the sum,

$\lambda(x_{j})\,\bm{F}_{R}(x_{j})=\sum_{i}w_{i}\,\bm{\mathcal{K}}(x_{j},x_{i})\cdot\bm{F}_{R}(x_{i})\
,$ (16)

and similarly,

$\lambda(x_{j})\,\bm{F}_{L}(x_{j})=\sum_{i}w_{i}\,\bm{F}_{L}(x_{i})\cdot\bm{\mathcal{K}}(x_{j},x_{i})\
,$ (17)

where $x_{i}$ are the nodes of a given quadrature with weights $w_{i}$. Since
the integral in Eq. (13) is over a charge-conjugate Bethe-Salpeter amplitude,
$\bar{\Psi}(k,-P):=C\,\Psi^{T}(-k,-P)C^{T}$, we introduce in Eqs. (16) and
(17) “left” and “right” eigenvectors, $\bm{F}_{L}$ and $\bm{F}_{R}$,
respectively. Multiplying Eq. (16) from the left by
$\sum_{j}w_{j}\,\bm{F}_{L}^{\prime}(x_{j})$ we obtain,

$\lambda\sum_{j}w_{j}\,\bm{F}_{L}^{\prime}(x_{j})\cdot\,\bm{F}_{R}(x_{j})\ =\
\sum_{i,j}w_{i}\,w_{j}\,\bm{F}_{L}^{\prime}(x_{j})\cdot\,\bm{\mathcal{K}}(x_{j},x_{i})\,\cdot\bm{F}_{R}(x_{i})\
=\
\lambda^{\prime}\sum_{i}w_{i}\,\bm{F}_{L}^{\prime}(x_{i})\cdot\bm{F}_{R}(x_{i})\
,$ (18)

where $\bm{F}_{L}^{\prime}(x_{j})$ is the eigenvector associated with the
eigenvalue $\lambda^{\prime}$. For $\lambda\neq\lambda^{\prime}$ this relation
implies,

$\displaystyle\int\\!dx\ \bm{F}_{L}^{\prime}(x)\cdot\bm{F}_{R}(x)=0\ ,\hskip
5.69046pt$ (19)

The left eigenvector is given by the Bethe-Salpeter wave function,
$S(k-\eta_{-}P)\,\overline{\Phi}_{\lambda^{\prime}}(k,-P)S(k+\eta_{+}P)$, and
the right eigenvector is the Bethe-Salpeter amplitude Blank:2011qk . We thus
deduce from Eq. (19),

$\displaystyle\mathrm{tr}_{\mathrm{CD}}\\!\
\int_{k}\,S(k-\eta_{-}P)\,\overline{\Phi}_{\lambda^{\prime}}(k,-P)\,S(k+\eta_{+}P)\,\Phi_{\lambda}(k,P)=0,\hskip
5.69046pt\text{for}\ \lambda^{\prime}\neq\lambda\ ,$ (20)

and because the trace is cyclic this orthogonality condition is equivalent to
that described in the paragraph below Eq. (13). Moreover, we verify that the
Bethe-Salpeter equation spectrum of Eq. (16) is equal to that in Eq. (17). We
stress that Eq. (19) is generally valid and no assumption was made about the
kernel structure or about the eigenvectors.

##  5 Examples in hadron physics

The meson and nucleon resonance structure has been the object of a long
history of studies and we here limit ourselves to the approaches based on the
combined Dyson-Schwinger and Bethe-Salpeter (or Faddeev) equations, in
particular their application to excited mesons and nucleons to compute their
masses, weak decay constants and/or electromagnetic form factors making use of
the techniques described in Sections 1–4.

Seminal studies on the ground state spectrum of light pseudoscalar mesons
established that the $\pi(1300)$ can be described as the first radially
excited state of the Goldstone boson and furthermore that the decay constant
of excited states, $P_{n}$, vanishes identically in the chiral limit
Holl:2004fr ,

$f_{P_{n}}^{\hat{m}=0}(\mu)\equiv 0\ ,\ n\geq 1\ ,$ (21)

where $\hat{m}$ is the renormalization-group invariant current-quark mass.
Electromagnetic properties of ground and excited state pseudoscalar mesons,
making use of the the Gram-Schmidt process illustrated in Section 2, were
studied in Ref. Holl:2005vu . A first approach to computing the eigenvalues of
unflavored light and heavy mesons and their corresponding Bethe-Salpeter
amplitudes with the implicitly restarted Arnoldi factorization, as implemented
in the ARPACK Arpack  library, is expounded in Refs. Blank:2010bp ;
Blank:2011qk  and has successively been applied to the light Hilger:2015ora
and heavy Hilger:2014nma ; Hilger:2015ora  quarkonia spectrum for
pseudoscalar, vector and tensor states. A recent detailed analysis of the
Maris-Tandy interaction Maris:1999nt  parameter space for ground and radially
excited states is presented in Ref. Hilger:2015ora , where the authors
conclude that the preferred inverse effective range of the Maris-Tandy
interaction in rainbow-ladder truncation, $\omega$, is shorter for heavy
quarkonia than for the light quarkonium spectrum: $\omega_{Q\bar{Q}}=0.7$ GeV
vs. $\omega_{q\bar{q}}=0.5$ GeV.

Table 1: Masses and decay constants for flavor singlet and nonsinglet $J^{P}=0^{-}$ mesons; see Section 5. | Model 1 [GeV] | Model 2 [GeV] | Reference
---|---|---|---
$m_{\pi}$ | 0.138 | 0.153 | 0.139 Beringer:1900zz
$f_{\pi}$ | 0.139 | 0.189 | 0.1304 Beringer:1900zz
$m_{\pi(1300)}$ | 0.990 | 1.414 |  $1.30\pm 0.10$ Beringer:1900zz
$f_{\pi(1300)}$ | $-1.1\times 10^{-3}$ | $-8.3\times 10^{-4}$ |
$m_{K}$ | 0.493 | 0.541 | 0.493 Beringer:1900zz
$f_{K}$ | 0.164 | 0.214 | 0.156 Beringer:1900zz
$m_{K(1460)}$ | 1.158 | 1.580 | 1.460 Beringer:1900zz
$f_{K(1460)}$ | $-0.018$ | $-0.017$ |
$m_{\bar{s}s}$ | 1.287 | 1.702 |
$f_{\bar{s}s}$ | $-0.0214$ | $-0.0216$ |
$m_{\eta_{c}(1S)}$ | 3.065 | 3.210 | 2.984 Beringer:1900zz
$f_{\eta_{c}(1S)}$ | 0.389 | 0.464 | 0.395 Davies:2010ip
$m_{\eta_{c}(2S)}$ | 3.402 | 3.784 | 3.639 Beringer:1900zz
$f_{\eta_{c}(2S)}$ | 0.089 | 0.105 |

The same library has then been employed to study radial excitations of flavor-
singlet and flavored pseudoscalar mesons within the framework of the rainbow-
ladder truncation Rojas:2014aka ; El-Bennich:2013yna ; Rojas:2014tya . A
summary of the theoretical values for the pseudoscalar’s ground- and excited-
state masses and weak decay constants for flavor-singlet and nonsinglet
$J^{P}=0^{-}$ mesons is reproduced in Table 1 for two parameter sets of the
interaction introduced by Qin et al. Qin:2011dd . Namely, Model 1 and Model 2
correspond to the interaction parameters, $\omega=0.4\leavevmode\nobreak\
\text{GeV}$, $\omega D=(0.8\leavevmode\nobreak\ \text{GeV})^{3}$ and
$\omega=0.6\leavevmode\nobreak\ \text{GeV}$, $\omega
D=(1.1\leavevmode\nobreak\ \text{GeV})^{3}$; see Ref. Rojas:2014aka  for
details and Ref. Rojas:2014tya  for a discussion of the effective interaction.
Note that we explored a range of parameter combinations for $\omega D\neq$
const., yet could not find a unique set which produces theoretical observables
that compare well with all data in Table 1. We therefore observe that we are
unable to consistently and simultaneously describe both ground and excited
state observables with the given interaction Qin:2011xq  in the rainbow-ladder
truncation — whereas one parameter set successfully reproduces experimental
ground state masses and decay constants but not the experimental data on
excited states, the second parameter set only provides a reasonable
description of the excited pseudoscalar mass spectrum.

Figure 2: Left panel: Lowest Chebyshev moment, ${}^{0}E_{P_{1}}(p^{2})$,
associated with the leading Dirac structure $E_{P_{1}}(p^{2})$ of the
pseudoscalar’s Bethe-Salpeter amplitude for the first radial excitations
$P_{1}=\pi(1300)$, $(\bar{s}s)_{n=1}$ and $\eta_{c}(2S)$. Right panel: First
three normalized Chebyshev moments, ${}^{i}S_{1}(p^{2})$, $i=0,1,2$, of the
leading $S$-wave component in the nucleon’s first excited-state Faddeev
amplitude.

As it becomes clear from the left graph of Fig. 2, for $p^{2}\gtrsim 1$ GeV2
the amplitude’s lowest Chebyshev projection, ${}^{0}E_{P_{1}}(p^{2})$
associated with the leading covariant $i\gamma_{5}$ of the pseudoscalar’s
Bethe-Salpeter amplitude, becomes negative definite in case of the first
radial excitations, $\pi(1300)$ and $(\bar{s}s)_{n=1}$, whereas it remains
positive for the ground states. It is remarkable that the behavior of the
Bethe-Salpeter amplitudes parallels the pattern of wave functions in quantum
mechanics, namely, the number of zeros can be associated with a principal
quantum number $n$. As just mentioned, the ground state amplitude has no zeros
and can thus be associated with $n=0$. The amplitude of the next highest mass
mesons possesses one zero and one assigns the quantum number $n=1$ and so on.
An analogous behavior is found for the nucleon’s first excited state Faddeev
amplitude for which we plot the first three Chebyshev moments of its leading
$S$-wave component. Indeed, very recently a range of properties of the
proton’s radial excitation was predicted which strongly suggests that the
nucleon’s first radial excitation is the Roper resonance Segovia:2015hra . In
particular, in Fig. 3 of Ref. Segovia:2015hra  one can appreciate that the
overlap amplitude between the nucleon and its first excited state described by
the nucleon-Roper transition form factor $F_{1}^{*}(Q^{2})$ vanishes at
$Q^{2}=0$ and therefore satisfies orthogonality.

We thank the organizers of the 21 International Conference on Few-Body
Problems in Physics in Chicago for the opportunity to present our work at this
successful event. The work of B. E. is supported in parts by a State of São
Paulo Research Foundation (FAPESP) grant, an MCTI/CNPq/Universal grant and by
the Brazilian agency CNPq with a visiting fellowship at the Institute for
Theoretical Physics, State University of São Paulo (IFT-Unesp). The work of E.
R. is supported by “Patrimonio Autónomo Fondo Nacional de Financiamiento para
la Ciencia, la Tecnología y la Innovación, Francisco José de Caldas” and by
“Sostenibilidad-UDEA 2014-2015”.

## References

  * (1) A. Bashir, L. Chang, I. C. Cloët, B. El-Bennich, Y. X. Liu, C. D. Roberts and P. C. Tandy, Commun. Theor. Phys. 58, 79 (2012).
  * (2) S. X. Qin, L. Chang, Y. X. Liu, C. D. Roberts and D. J. Wilson, Phys. Rev. C 85, 035202 (2012).
  * (3) E. Rojas, J. P. B. C. de Melo, B. El-Bennich, O. Oliveira and T. Frederico, JHEP 1310, 193 (2013).
  * (4) E. Rojas, B. El-Bennich and J. P. B. C. de Melo, Phys. Rev. D 90, 074025 (2014).
  * (5) I. G. Aznauryan et al., Int. J. Mod. Phys. E 22, 1330015 (2013).
  * (6) B. El-Bennich, J. P. B. C. de Melo, B. Loiseau, J.-P. Dedonder and T. Frederico, Braz. J. Phys. 38, 465 (2008).
  * (7) L. Chang, I. C. Cloët, B. El-Bennich, T. Klähn and C. D. Roberts, Chin. Phys. C 33, 1189 (2009).
  * (8) E. O. da Silva, J. P. B. C. de Melo, B. El-Bennich and V. S. Filho, Phys. Rev. C 86, 038202 (2012).
  * (9) B. El-Bennich, J. P. B. C. de Melo and T. Frederico, Few Body Syst. 54, 1851 (2013).
  * (10) B. El-Bennich, M. A. Ivanov and C. D. Roberts, Nucl. Phys. Proc. Suppl. 199, 184 (2010).
  * (11) B. El-Bennich, M. A. Ivanov and C. D. Roberts, Phys. Rev. C 83, 025205 (2011).
  * (12) B. El-Bennich, G. Krein, L. Chang, C. D. Roberts and D. J. Wilson, Phys. Rev. D 85, 031502 (2012).
  * (13) B. El-Bennich, C. D. Roberts and M. A. Ivanov, POS(QCD-TNT-II) 018, arXiv:1202.0454 [nucl-th] (2012).
  * (14) P. Maris, C. D. Roberts and P. C. Tandy, Phys. Lett. B 420, 267 (1998).
  * (15) P. Maris and C. D. Roberts, Phys. Rev. C 56, 3369 (1997).
  * (16) P. Maris and P. C. Tandy, Phys. Rev. C 60, 055214 (1999).
  * (17) S. x. Qin, L. Chang, Y. x. Liu, C. D. Roberts and D. J. Wilson, Phys. Rev. C 84, 042202 (2011).
  * (18) D. Binosi, L. Chang, J. Papavassiliou and C. D. Roberts, Phys. Lett. B 742, 183 (2015).
  * (19) A. Bender, C. D. Roberts and L. Von Smekal, Phys. Lett. B 380, 7 (1996).
  * (20) J. J. Cobos-Martínez, Static and dynamic properties of the pion from continuum model ling of strong QCD, PhD Thesis, Durham University (2010).
  * (21) W. Lucha and F. F. Schöberl, Phys. Rev. D 87, no. 1, 016009 (2013).
  * (22) T. Frederico, G. Salmè and M. Viviani, Phys. Rev. D 89, 016010 (2014).
  * (23) M. S. Bhagwat, A. Höll, A. Krassnigg, C. D. Roberts and S. V. Wright, Few Body Syst. 40, 209 (2007).
  * (24) A. Krassnigg and C. D. Roberts, Fizika B 13, 143 (2004).
  * (25) I. C. Cloët, G. Eichmann, B. El-Bennich, T. Klähn and C. D. Roberts, Few Body Syst. 46, 1 (2009).
  * (26) J. Segovia, B. El-Bennich, E. Rojas, I. C. Cloët, C. D. Roberts, S. S. Xu and H. S. Zong, Phys. Rev. Lett. 115, no. 17, 171801 (2015).
  * (27) W. H. Press, S. A. Teukolsky, W. T. Vetterling and B. P. Flannery, ISBN-9780521430647.
  * (28) R. B. Lehoucq, D. C. Sorensen, C. Yang, ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods, Siam (1998).
  * (29) W. E. Arnoldi, Quarterly of Applied Mathematics, vol. 9, pp. 17-29 (1951).
  * (30) M. Blank, arXiv:1106.4843 [hep-ph] (2011).
  * (31) A. Höll, A. Krassnigg and C. D. Roberts, Phys. Rev. C 70, 042203 (2004).
  * (32) A. Höll, A. Krassnigg, P. Maris, C. D. Roberts and S. V. Wright, Phys. Rev. C 71, 065204 (2005).
  * (33) M. Blank and A. Krassnigg, Comput. Phys. Commun. 182, 1391 (2011).
  * (34) T. Hilger, C. Popovici, M. Gomez-Rocha and A. Krassnigg, Phys. Rev. D 91, no. 3, 034013 (2015).
  * (35) T. Hilger, M. Gomez-Rocha and A. Krassnigg, arXiv:1508.07183 [hep-ph].
  * (36) B. El-Bennich, E. Rojas, M. A. Paracha and J. P. B. C. de Melo, AIP Conf. Proc. 1625, 80 (2014).
  * (37) E. Rojas, B. El-Bennich, J. P. B. C. De Melo and M. A. Paracha, Few Body Syst. 56, no. 6-9, 639 (2015).
  * (38) J. Beringer et al. [Particle Data Group Collaboration], Phys. Rev. D 86, 010001 (2012).
  * (39) C. T. H. Davies, C. McNeile, E. Follana, G. P. Lepage, H. Na and J. Shigemitsu, Phys. Rev. D 82, 114504 (2010).