CHIRPs: Change-Induced Regret Proxy metrics for Lifelong Reinforcement Learning
John Birkbeck Adam Sobey Federico Cerutti Katherine Heseltine Hurley Flynn Timothy J. Norman


Exploring the dynamic rotational profile of the hotter solar atmosphere: A multi-wavelength approach using SDO/AIA data
    [
    
=======================================================================================================================







§ ABSTRACT 
Reinforcement learning agents can achieve superhuman performance in static tasks but are costly to train and fragile to task changes. This limits their deployment in real-world scenarios where training experience is expensive or the context changes through factors like sensor degradation, environmental processes or changing mission priorities. Lifelong reinforcement learning aims to improve sample efficiency and adaptability by studying how agents perform in evolving problems. The difficulty that these changes pose to an agent is rarely measured directly, however. Agent performances can be compared across a change, but this is often prohibitively expensive. We propose Change-Induced Regret Proxy (CHIRP) metrics, a class of metrics for approximating a change's difficulty while avoiding the high costs of using trained agents. A relationship between a CHIRP metric and agent performance is identified in two environments, a simple grid world and MetaWorld's suite of robotic arm tasks. We demonstrate two uses for these metrics: for learning, an agent that clusters MDPs based on a CHIRP metric achieves 17% higher average returns than three existing agents in a sequence of MetaWorld tasks. We also show how a CHIRP can be calibrated to compare the difficulty of changes across distinctly different environments.












§ THE VALUE OF MEASURING CHANGE


Humans are experts at adaptation. When we detect changes in the world, we can predict their consequences and replan our behaviours accordingly. In contrast, Reinforcement Learning (RL) agents perform poorly when change occurs; their adaptation is a product of trial and error rather than anticipation. While RL agents have outperformed humans in controlled conditions, they require vast amounts of experience to do so, and lose their ability rapidly as conditions begin to vary <cit.>.

These weaknesses mean RL agents are inapplicable in problems with a wide variety of potential conditions or tasks. Consider an aerial drone designed to monitor a hazardous area. The drone could suffer sensor or actuator damage, be exposed to dangerous weather and adversaries, or be assigned a new mission. This variety precludes pre-training against all possibilities and instead requires agents to achieve human-like adaptation from small samples of experience.

In Lifelong Reinforcement Learning (LRL), agents are exposed to change to test whether they exhibit fast adaptation and fast remembering <cit.>, avoid interference <cit.>, or incrementally learn upon prior knowledge <cit.>. However, we rarely analyze the relationship between change and agent performance directly. Understanding this relationship could help agents achieve these qualities through proactive behavioural adjustments that mitigate a change's potential impact.

Metrics calculated from agent performance, such as regret, could be used for this purpose but require agents to be trained for each change to be measured; this is prohibitively expensive for more than a handful of examples and requires the designer to be aware of all the possible changes. To our knowledge, no performance-based metric has been used to quantify the difficulty of MDP change. This is likely due to the high computational costs involved <cit.>. 

Model-based metrics measure differences in Markov Decision Process components, avoiding the need for trained agents, but are unsuitable for other reasons. In <cit.>, restricted Boltzmann machines are trained to represent MDPs, with the MDP distance defined as a difference between model weights. This becomes prohibitively expensive as the number of measurements needed increases. Other metrics require a calculation over every state in both MDPs, such as <cit.>'s method for finite MDPs, and are also inapplicable for continuous state or action spaces. 

Although variation in these components is the cause of degraded agent performance, the link between model- and performance-based metrics has not yet been investigated, likely due to restrictive assumptions like finite state-action spaces or their sometimes considerable calculation costs <cit.>. 

Metrics between MDPs are rarely used for lifelong reinforcement learning. Arguably the closest to the CHIRP concept is Lipshitz Lifelong Reinforcement Learning <cit.>, in which a pseudometric is defined on the transitions of finite MDPs to be used for lifelong learning. However, no method for efficiently estimating this metric for complex MDPs is discussed.

This paper introduces Change-Induced Regret Proxies (CHIRPs), a class of metrics that use the difference between Markov Decision Process components to approximate their impact on agent performance; an example CHIRP is constructed from the Wasserstein distance between MDP transition distributions and a relationship between it and agent performance is identified in a simple gridworld and MetaWorld's suite of robotic arm tasks. The value of CHIRPs in lifelong reinforcement learning is demonstrated by an agent that reuses policies across MDPs clustered by CHIRP value and an example calibration of a CHIRP across environments is used to allow benchmark-agnostic difficulty comparisons for the first time.



§ PRELIMINARIES


The standard Reinforcement Learning (RL) problem is defined as a Markov Decision Process (MDP) ℳ = {𝒮, 𝒜, ℛ, 𝒫, γ} with state space 𝒮, action space 𝒜, reward space ℛ: 𝒮×𝒜×𝒮ℝ, transition probability density function p(s', r | s, a) ∈ [0, 1] and discount rate γ∈ [0, 1] <cit.>.

An RL agent is tasked with learning an action policy π: 𝒮→𝒜 which maximises the returns, the discounted sum of future rewards G_t = ∑_k=0^∞γ^k R_t+k+1 from the current timestep t.

Lifelong Reinforcement Learning modifies the RL MDP by allowing its components[We treat the discount rate γ as a static component due to its common treatment as a tunable hyper-parameter rather than a pre-defined aspect of the problem.] to vary in time: ℳ_L(t) = {𝒮(t), 𝒜(t), ℛ(t), 𝒫(t), γ}. These components may change discretely, continuously, or as a semi-continuous mixture. As ℳ_L(t) can always be represented as a sequence of static MDPs (in the extreme case as a unique MDP per timestep), we use static MDPs ℳ_i to denote versions of ℳ_L(t) as it evolves through time.

In episodic RL, a policy π's performance in MDP ℳ_a can be judged by its expected returns over s ∈𝒮^0_a, the set of possible initial states: 𝔼[G_t |𝒮^0_a, π]. In Lifelong RL, the expected returns of a policy also depend upon the current components of ℳ_L(t); for clarity below, we omit 𝒮^0_a as part of the definition of ℳ_a and include ℳ_a as an explicit dependence of the expectation: 𝔼[G_t |ℳ_a, π].



§ DEFINING A TARGET FOR PROXY MEASUREMENT: SCALED OPTIMAL POLICY REGRET



A natural choice for measuring the impact of an MDP change is regret, the drop in an agent's returns <cit.>. However, comparing returns is unsuitable when the two MDPs have different minimum and maximum returns. Returns and rewards bounds can change independently; a transition function change can block high-reward states from being visited, lowering the maximum returns without changing the reward function.

To avoid this issue, we propose Scaled Optimal Policy Regret (SOPR) to account for the returns' scales in each MDP:

    SOPR(ℳ_i, ℳ_j) = 
       ∑_π^+_i ∈Π^+_i[𝔼[G_t |ℳ_j, π^+_j ] - 𝔼[G_t |ℳ_j, π^+_i ]/𝔼[G_t |ℳ_j, π^+_j ] - 𝔼[G_t |ℳ_j, π^-_j ]],

where π^+_j is an optimal returns-maximising policy of ℳ_j and π^-_j is an optimal returns-minimising policy of ℳ_j. Note that SOPR(ℳ_i, ℳ_j) ∈ [0, 1] for any pair of MDPs where SOPR is calculable.



  §.§.§ When is SOPR calculable?


By inspection of (<ref>), SOPR(ℳ_i, ℳ_j) is defined if and only if the four expectations are defined. It is reasonable to assume three of the four terms are defined as they are expectations of policies in their respective MDPs. This leaves only the expectation that `crosses' the MDPs, 𝔼[G_t |ℳ_j, π^+_i ] to be analyzed. 

Equation (<ref>) is the standard definition <cit.> of expected returns for a single state s_j ∈𝒮_j. For 𝔼[G_t |ℳ_j, π^+_i ] to be defined, (<ref>) must be defined for all initial states of ℳ_j. This implies that 𝒫_i(s', r | s, a) must be defined ∀ a ∈π^+_i(a|s), and therefore that 𝒜_i ⊆𝒜_j. Additionally, π^+_i(a|s) must be defined ∀ s' ∈𝒮_j, therefore s' ∈𝒮_j  s' ∈𝒮_i, i.e. 𝒮_i ⊆𝒮_j. 

Less formally, SOPR(ℳ_i, ℳ_j) is calculable so long as ℳ_i's optimal policies are executable in every state of ℳ_j.


    𝔼[G_t |s_j, π^+_i ] = 
       ∑_aπ^+_i(a|s_j)∑_s_j', r𝒫_i(s_j', r | s, a)[r + γ𝔼[G_t+1 | s_j', π^+_i ]].




§ CHANGE-INDUCED REGRET PROXY (CHIRP) METRICS


A desirable proxy for SOPR would have the following properties:

    
  * Positively correlated with SOPR: as the measured proxy distance between MDPs grows, the drop in performance should increase.
    
  * Monotonicity with SOPR: It is undesirable to have a single value of the proxy metric map to multiple SOPR values. A monotonic relationship between them would avoid this.
    
  * Computational efficiency: There is little value in an estimator of performance-based metrics with comparable computational costs. 
    
  * Captures change across MDP components: In lifelong RL we may encounter changes in any MDP component.


With these requirements in mind, we modify <cit.>'s existing metric for use as an example CHIRP. In their work, a distance d'(s_i, s_j) between states in MDPs ℳ_i and ℳ_j is defined as the maximum Wasserstein distance between transitions. This is aggregated over all state pairs to define a distance between MDPs,

    d(ℳ_i, ℳ_j) = min_k=1, …, |𝒮_1|, 
     t=1, …, |𝒮_2|∑_k=1^|𝒮_1|∑_t=1^|𝒮_2|γ_ktd'(s_k, s_t),

with γ_kt a constraint not discussed here. This is intractable for continuous state and action spaces and is difficult to estimate from samples as the minimum is taken over state spaces 𝒮_1 and 𝒮_2.

Instead, we use the Wasserstein distance between the transition distributions directly and propose a sampling method for its approximation. Generally, the 1-Wasserstein (W_1) distance between two probability distributions 𝒫_i(ℝ_d), 𝒫_j(ℝ_d) is defined as,

    W_1(𝒫_i, 𝒫_j) = inf_γ∈Γ(𝒫_i, 𝒫_j)∫_ℝ^d ×ℝ^d||𝐱 - 𝐲||_2 γ(δ𝐱,δ𝐲),

where γ∈Γ(𝒫_i, 𝒫_j) is the set of all joint probability distributions with marginal distributions 𝒫_i and 𝒫_j <cit.>. Informally, the W_1 distance is often called the `earth mover's distance', the minimal earth (probability mass) that must be moved from one pile, 𝒫_i, to produce another, 𝒫_j.

Focusing upon MDPs, we define the W_1-MDP distance as the W_1 distance between transition probability density functions, 

    W_1(ℳ_i, ℳ_j) = W_1(𝒫_i(s', r |s, a), 𝒫_j(s', r |s, a))
        = inf_γ∈Γ(𝒫_i, 𝒫_j)∫_ℝ^d ×ℝ^d||𝐬^∗_i - 𝐬^∗_j||_2 γ[δ𝐬^∗_i,δ𝐬^∗_j],

with 𝐬^∗_i = (𝐬', r | 𝐬, 𝐚)_i distributed by 𝒫_i(𝐬', r | 𝐬, 𝐚).

This metric is still incalculable in continuous state and action spaces, as it requires evaluating 𝒫_i and 𝒫_j for all possible (s, a) pairs. However, W_1(𝒫_i, 𝒫_j)'s infimum is evaluated over the distributions 𝒫_i and 𝒫_j rather than (<ref>)'s minimum over state pairs. By estimating 𝒫_i and 𝒫_j with empirical distributions 𝒫̂_i, 𝒫̂_j from transition samples, we can estimate W_1(𝒫_i, 𝒫_j) with W_1(𝒫̂_i, 𝒫̂_j) so long as a suitable sampling scheme is identified.



 §.§ Approximating the W_1-MDP Distance with Sampling
 
The accuracy and precision of W_1(𝒫̂_i, 𝒫̂_j) depends upon how representative our transition samples are. A natural approach would be to randomly sample state-action pairs to construct the 𝒫̂_i and 𝒫̂_j distributions. We propose an alternate scheme based on the intuition that sampling transitions from a range of low and high reward states may better represent an optimal policy's trajectory than random samples.



We first sample the base MDP's reward bounds and use this as a target for Monte Carlo Cross-Entropy (MCCE) sampling of state-action pairs <cit.>. The state-action pair is then executed in both MDPs n_t times to capture any stochasticity of transitions. The result is two sets of transition samples of size n_s × n_t; these sets form our empirical distributions 𝒫̂_i, 𝒫̂_j. The W_1 distance between these samples, W_1(𝒫̂_i, 𝒫̂_j) is the estimate of W_1(ℳ_i, ℳ_j).

Algorithm <ref> documents this sampling scheme; results comparing this scheme to random sampling are presented for two environments below.



§ VALIDATING W_1-MDP AS A CHIRP


Before using the W_1-MDP distance as a CHIRP, we must establish whether it exhibits the desired characteristics listed above. It is unlikely that W_1-MDP is perfectly correlated with SOPR in every case or that the relationship is the same in every environment, so understanding a CHIRP's limits is vital for its correct use.



 §.§ Validation in SimpleGrid


A bespoke toy environment inspired by MiniGrid <cit.> is designed for initial validation. SimpleGrid is a 20 × 20 square grid implemented in the gymnasium framework  <cit.>, illustrated in Figure <ref>. MDP variants are constructed by varying the agent and goal positions. The agent's state space is 4-dimensional and is constructed from the (x, y) coordinates of the agent and goal: s = ((x, y)_agent,(x, y)_goal). At each time step the agent chooses an action from 𝒜 = {up, down, left, right} to minimize its Manhattan distance to the goal,

    r(s, a, s') = -C|(x, y)_agent - (x, y)_goal|_1,

with C as a scaling constant determined by the agent's initial position.



The results in Figure <ref> show SOPR and W_1-MDP distances from a base MDP to 10,500 MDPs with random agent and goal positions. The Pearson correlation coefficient indicates SOPR is strongly positively correlated with W_1-MDP (ρ=0.875, p<0.001) and is strongly monotonic (r_s=0.861, p<0.001) by Spearman's rank correlation.



Figure <ref> shows a strong linear relationship between our CHIRP and SOPR, but SimpleGrid is a trivial example where W_1-MDP can be calculated in full. For this to be useful in practice, the viability of estimating W_1-MDP has to be established, either through random sampling or through Algorithm <ref>'s reward-shaped sampling.

Our analysis of the two sampling methods is primarily focused on variance, rather than bias; in this case, a bias when estimating a proxy metric is relatively unimportant compared to the estimates' variance, as a systematic bias is easily removed during calibration.

Estimates of the true distances measured for Figure <ref> were calculated under the two sampling schemes with n_s=15 and n_t=1; in contrast, the full calculation uses 1,296 states. The biases and variances for both schemes are provided in Table <ref>.

Random sampling produces estimates of W_1-MDP with lower variance than reward-shaped sampling; reward-shaped sampling also has a statistically significant (p<0.001) bias under a two-sided t-test with a null hypothesis of zero-mean errors. The bias is small when compared to the scale, however: 0.079 is 0.76% of the median W_1-MDP of 10.3. 

Though reward-shaped sampling achieves a higher variance, these results validate it as a viable sampling scheme. A similar analysis is also performed below in Metaworld with contrasting results.





 §.§ Verification in MetaWorld


SimpleGrid has provided evidence that the CHIRP concept is valid and that W_1-MDP can be estimated through sampling. The SimpleGrid environment is unrepresentative of real-world problems, however; we therefore choose MetaWorld <cit.>, a suite of robotic arm tasks, as a more representative setting for further testing.



MetaWorld's continuous state and action spaces make calculating W_1-MDP impossible. The correlation between W_1-MDP and SOPR must be analyzed using estimates of W_1-MDP, requiring us to choose a sampling scheme: reward-shaped or random.

Though accuracy is indeterminable, estimates' variance under each sampling scheme can still be measured. Using 270 standard deviations of estimates for both methods over the 10 MetaWorld tasks (Figure <ref>) with n_s ∈{25, 50}, n_t ∈{1, 5, 10}, we find that estimates using reward-shaped sampling had a smaller average standard deviation of 0.331 in comparison to random sampling's 0.384, a 14% reduction (p<0.001).

We use reward-shaped sampling in MetaWorld due to its lower variance and the argument presented above of the ease of adjusting for systematic bias during calibration.

To measure SOPR's relationship with W_1-MDP, 200 soft actor-critic agents were trained on one of 10 MetaWorld tasks, with 20 agents per task. After training, their weights were frozen. Each agent was exposed to increasing state and action errors to simulate sensor and actuator degradation. The SOPRs between these MDPs were estimated using the highest and lowest returns observed across all agents to avoid the otherwise extreme cost of calculating SOPR.

Figure <ref> shows the aggregated Ŵ_1-SOPR relationship across the 200 agents and 10 tasks. A positive monotonic relationship exists with a moderate positive correlation (ρ=0.60, p<0.001) and monotonicity (r_s=0.70, p<0.001). The variance of SOPR values within each bin is wider than for SimpleGrid; this stems from the tendency of agents to either succeed or fail in MetaWorld, with fewer examples of mediocre returns than in SimpleGrid. 

The relationship displayed in Figure <ref> is not linear, and total performance loss (a SOPR of 1) occurs at much smaller CHIRP distances compared with SimpleGrid's results. Therefore, our CHIRP measurements are incomparable across environments in their raw form; we address this with calibration further below.




§ LIFELONG REINFORCEMENT LEARNING WITH CHIRPS IN METAWORLD


Analyzing a CHIRP's correlation, bias, and variance with SOPR is useful, but lacking context. What degree of correlation is required for use, and does the variance in Figure <ref> prohibit this?

To test our CHIRP's value to lifelong reinforcement learning we modify <cit.>'s Lifetime Policy Reuse (LPR) agent. In LPR, a multi-armed bandit learns a strategy to reuse k policies across n tasks with k <= n. The bandit and k policies begin untrained and are learned with standard algorithms such as PPO <cit.> and Q-learning <cit.>. 

Intuitively, LPR's bandit can be seen as finding groups of MDPs over which a single policy can achieve good returns; the learned groupings may be similar to clustering MDPs with low CHIRP values together. Therefore, the need to learn the mapping from experience (and suffer sub-optimal choices during this) could be avoided with mappings pre-computed from CHIRP metrics.



We formalize this idea as CHIRP Policy Reuse (CPR). To pre-compute CPR's reuse strategy, the CHIRP values between the n(n-1)/2 unique MDP pairs are estimated, producing a distance matrix as in Figure <ref>. k-medoids clustering <cit.> is applied to this matrix to identify clusters with low intra-cluster CHIRP values. Figure <ref> visualizes the resulting clusters.



Comparing Figures <ref> and <ref> with the tasks shown in Figure <ref> provides some insight into the task space; MDPs 5 through 9 all share low CHIRP values and visually appear to be similar tasks, and are all clustered together. 3 and 4 are similar, while 0, 1 and 2 are more distinct from other MDPs. With only 3 policies to share between 5 MDPs, the most distinct MDPs of 0 and 2 have dedicated policies assigned, while 1, 3, and 4 share the last policy.

We compare CHIRP Policy Reuse (CPR) against Lifetime Policy Reuse (LPR) and two other LRL methods; Lifelong Reinforcement Learning with Modulating Masks (Mask-LRL) <cit.> and Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting (LPG-FTW) <cit.>, two approaches to LRL that are distinct from policy reuse. These methods were each chosen based on their strong results in MetaWorld and Continual World <cit.>, a MetaWorld derivative. A variant of CPR is also included in these comparisons, `Bandit CPR'. This agent uses CPR's clusters to initialize a multi-armed bandit as in LPR. 

To test each method's sample efficiency and robustness, random sequences of the ten MetaWorld MDPs were generated with a p=0.1 probability of changing at the end of each episode. Each method will be judged based on the lifetime average returns they achieve (Equation (<ref>)). The task sequences were limited to T=1×10^6 timesteps; the shorter duration and frequent MDP changes require agents to demonstrate high sample efficiency and robustness to change. 


    R̅ = 1/T∑_T=0^TR_t.


Every method but the Mask-LRL approach has a policy library size parameter; the three policy reuse methods have a fixed library size of k, while LPG-FTW grows its library until a limit k is reached. To select k for each method, a search over k ∈{2, 4, 6, 8, 10} was used to identify the value which maximized Equation <ref> across 20 agents and T=3×10^5 timesteps.

With each method's best k determined, 20 new agents per method were trained for T=1×10^6 timesteps resulting in Table <ref>. CPR achieves the highest lifetime average return of 1516.1 ± 1398.7, 17% higher than the next best method with a mean increase of 220.6 (p<0.001) above LPR. Figure <ref> shows a rolling average of episodic performance; CPR also consistently achieves higher returns on average than other methods.

All experimentation was performed on Intel Xeon Gold 6138 and AMD Ryzen 5600X CPUs with 16Gb RAM.







  §.§.§ Why don't LPR agents learn to outperform CPR's fixed strategy?

CPR's results are significantly higher than both the LPR and Bandit CPR methods; this is surprising, as methods that update their reuse strategies should outperform strategies fixed on imperfect information like a proxy metric.

Analyzing one LPR agent's (k=6, R̅ = 1120) task-policy mapping against the equivalent medoid clustering result where k=6 shows that LPR is susceptible to becoming trapped in local optima. The LPR agent reused five policies across the ten MDPs, leaving one policy `spare'. This suboptimality is unlikely to be escaped in future learning; the spare policy's episodic returns are lower in every MDP than at least one other policy, and it is only used when selected by the bandit's epsilon-greedy exploration. Its lack of selection causes a lack of training, exacerbating this policy's low performance, and cementing its exclusion from the strategy.





  §.§.§ Why do Mask-LRL and LPG-FTW perform poorly?


The rolling averages in Figure <ref> show a significant difference in performance between Mask-LRL and LPG-FTW and the reuse methods.

One cause for this may be the interleaving of tasks in our benchmarking. In Mask-LRL and LPG-FTW's original benchmarking, agents are trained against each MDP in a single block of one million time steps, for ten million total. This approach allows the agents to converge to optimal policies in each task individually. 

The frequent MDP changes in our testing may provide specific difficulties to these methods. In LPG-FTW, each new task is solved in part by combining solutions to previous tasks.  Mask-LRL similarly combines prior tasks' maps to transfer learning into new tasks. If the agent has not had the opportunity to converge on previous tasks, then the new tasks will be learned from combining prior task solutions that are themselves changing. This likely provides an additional level of difficulty to the learning problem.



§ CALIBRATING A CHIRP TO COMPARE DIFFICULTY ACROSS ENVIRONMENTS


Comparing Figures <ref> and <ref>'s  distinctly different CHIRP-SOPR relationships highlights how they can only be assumed to hold in the `local area' of the MDP. This illustrates the need for calibration when comparing across distinctly different problems. 

As an example of calibration, two B-spline functions were fit to the data from figures <ref> and <ref>; the results are indicated by the dashed lines in these figures. These functions are effectively a predictive relationship between the `raw' CHIRP values and SOPR; we refer to these SOPR predictions as the `calibrated CHIRP' in Figures <ref> and <ref>.

Figures <ref>'s results for SimpleGrid exemplify a desirable calibration; a calibrated CHIRP value can be used directly as an estimate of SOPR; Combined with MetaWorld's calibrated results in <ref>, the calibrated CHIRP values can be compared across environments to determine a predicted SOPR. Calibrating CHIRPs in this way permits novel comparisons of difficulties that are agnostic to the benchmarking environments used.







 §.§ Understanding the limits of a CHIRP-SOPR relationship


The relationship between a proxy metric and its target cannot be assumed to be globally fixed; the need to calibrate to the local conditions is important, and conversely, understanding the limits of where an established relationship ends.

To highlight this, we discuss one inherent limitation of choosing W_1-MDP as our example CHIRP. The Wasserstein metric is symmetric by definition; it cannot capture non-symmetry in SOPR. In general, SOPR(ℳ_i, ℳ_j) ≠SOPR(ℳ_j, ℳ_i) as two MDPs' optimal policies may not achieve the same returns in the other MDP. Table <ref> shows the non-symmetry of SOPR for MDPs A and B shown in Figure <ref>.









§ CONCLUSIONS


Lifelong Reinforcement Learning studies the impacts of change on agents, but our understanding of this relationship is currently limited. Directly measuring agent performance across many changes is too costly, and model-based metrics are generally unsuitable for non-trivial problems.

To our knowledge, this paper is the first to investigate whether measuring a change directly can be predictive of its performance impact. We used one example to demonstrate the utility of a potential class of proxy metrics, CHIRPs, in two ways: lifelong reinforcement learning and benchmark-agnostic difficulty comparisons. 

For lifelong reinforcement learning, MDPs were clustered for policy reuse based on their CHIRP values. Despite this agent using a fixed policy, it achieved a 17% performance increase above the next best method tested. We also include a simple calibration of a CHIRP to two environments to discuss how change-induced difficulty can be compared across otherwise incomparable problems.



