Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?

The evidence to be evaluated is:We introduce a novel _Exogenous Block Markov Decision Process_ (EX-BMDP) setting to model systems with exogenous noise.

We start with describing the Block Markov Decision Process (BMDP) Du et al. (2019). This process consists of a finite set of observations \(\mathcal{X}\), a set of _latent_ states \(\mathcal{Z}\) with cardinality \(Z\), a finite set of actions \(\mathcal{A}\) with cardinality \(A\), a transition function \(T:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{Z})\), an emission function \(q:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{X})\), a reward function \(R:\mathcal{X}\times\mathcal{A}\rightarrow[0,1]\), a horizon \(H\in\mathbb{N}\), and a start state distribution \(\mu\in\Delta(\mathcal{Z})\). The agent interacts with the environment by repeatedly generating \(H\)-step trajectories \((z_{1},x_{1},a_{1},r_{1},\cdots,z_{H},x_{H},a_{H},r_{H})\) where \(z_{1}\sim\mu(\cdot)\) and for every \(h\in[H]\) we have \(x_{h}\sim q(\cdot\mid z_{h})\), \(r_{h}=R(x_{h},a_{h})\), and if \(h<H\), then \(z_{h+1}\sim T(\cdot\mid z_{h},a_{h})\). The agent does not observe the states \((z_{1},\cdots,z_{H})\), instead receiving only the observations \((x_{1},\cdots,x_{H})\) and rewards \((r_{1},\cdots,r_{H})\). We assume that the emission distributions of any two latent states are disjoint, usually referred as the _block assumption_: \(supp(q(\cdot\mid z_{1}))\cap supp(q(\cdot\mid z_{2}))=\emptyset\) when \(z_{1}\neq z_{2}\). The agent chooses actions using a policy \(\pi:\mathcal{X}\rightarrow\Delta(\mathcal{A})\). We also define the set of non-stationary policies \(\Pi_{\mathrm{NS}}=\Pi^{H}\) as a \(H\)-length tuple, with \((\pi_{1},\cdots,\pi_{H})\in\Pi_{\mathrm{NS}}\) denoting that the action at time step \(h\) is taken as \(a_{h}\sim\pi_{h}(\cdot\mid x_{h})\). The value \(V(\pi)\) of a policy \(\pi\) is the expected episodic sum of rewards \(V(\pi):=\mathbb{E}_{\pi}[\sum_{h=1}^{H}R(x_{h},a_{h})]\).

_An EX-BMDP is a BMDP such that the latent state can be decoupled into two parts \(z=(s,\xi)\) where \(s\in\mathcal{S}\) is the endogenous state and \(\xi\in\Xi\) is the exogenous state.

The endogenous state \(s\) captures the information that can be manipulated by the agent.

We assume that the set of all endogenous states \(\mathcal{S}\) is finite with cardinality \(S\). The exogenous state \(\xi\) captures all the other information that the agent cannot control and does not affect the information it can manipulate. Again, we make no assumptions on the exogenous dynamics nor on its cardinality \(|\Xi|\) which may be arbitrarily large.

The decoupled dynamics assumption made in the EX-BMDP setting is a natural way to characterize exogenous noise; the type of noise that is not affected by our actions nor affects the endogenous state but may have non-trivial dynamic. This decoupling captures the movement of ducks, captured in the visual field of the agent in Figure 1, and many additional exogenous processes (e.g., movement of clouds in a navigation task).

Thus, for a BMDP, the cardinality of the policy cover scales with \(|\mathcal{Z}|\). The structure of EX-BMDPs allows to reduce the size of the policy cover significantly to \(|\mathcal{S}|\ll|\mathcal{Z}|=|\mathcal{S}|\,|\Xi|\) when the size of the exogenous state space is large. Specifically, we show that the set of policies that reach each _endogenous_ state, and _do not depend on the exogenous_ part of the state is also a policy cover (see Appendix B, Proposition 4).

Unfortunately, in the presence of exogenous noise, Homer distinguishes between two transitions that represent transition between the same latent endogenous states but different exogenous states. In our walk in the park example, even if the agent moves between same points in two transitions, the model maybe able to tell these transitions apart by looking at the position of ducks which may have different behaviour in the two transitions. This results in the Homer creating \(\mathcal{O}(|\mathcal{Z}|)\) many abstract states. We call this the _under-abstraction_ problem.

Such a representation can ignore all information that is not relevant for action prediction, which includes all exogenous/uncontrollable information.

Zhang et al. (2020) proposed learning a bisimulation metric to learn a representation which is invariant to exogenous noise.

Bellman rank might depend on \(|\Xi|\). The Bellman rank was introduced in Jiang et al. (2017) as a complexity measure for the learnability of an RL problem with function approximations.

However, we show in Appendix A that Bellman rank for EX-BMDP can scale as \(\mathcal{O}(|\Xi|)\). This shows that EX-BMDP is a highly non-trivial setting as we don't even have sample-efficient algorithms regardless of computationally-efficient.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.

This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action.

Recovering an endogenous state decoder. We can recover a endogenous state decoder \(\hat{\phi}_{h}\) for each time step \(h\in\{2,3,\cdots,H\}\) directly from \(\hat{f}_{h}\) as shown below: \[\hat{\phi}_{h}(x)=\min\left\{i\mid\hat{f}_{h}(i\mid x)\geq\max_{j}\hat{f}_{h}(j \mid x)-\mathcal{O}(\nicefrac{{1}}{{\Gamma_{h}}}),i\in[\left|\Upsilon_{h} \right|]\right\}.\] Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting \(x\), and therefore, \(\phi^{\star}(x)\). We are implicitly using the decoder for exploring, since we rely on using \(\hat{f}_{h}\) for making planning decisions.

PPE can also be used to recover a latent endogenous transition dynamics.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

If the reward function only depends on the endogenous state, we can use a computationally cheaper value-iteration VI that uses the recovered transition dynamics. VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details). In each case, the sample complexity of learning a near-optimal policy, given the output of PPE, scales with the size of endogenous and not the exogenous state space.

**Assumption 1** (Near deterministic endogenous dynamics).: _We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\).

However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space. Further, since our analysis only uses standard uniform convergence arguments, it extends straightforwardly to infinitely large function classes by replacing \(\ln|\mathcal{F}|\) with other suitable complexity measures such as Rademacher complexity.

Let \(\mathbb{P}_{h}(\xi)\) denote the distribution over exogenous states at time step \(h\) which is independent of agent's policy. The Bayes optimal classifier (\(f^{*}_{h}:=f_{\Upsilon_{h},h}\)) of the prediction problem can be derived as: \[f^{*}_{h}(\texttt{idx}(v)\mid x):=\mathbb{P}_{h}(v\mid x)=\frac{ \mathbb{P}_{h}(x\mid v)\mathbb{P}(v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{ P}_{h}(x\mid v^{\prime})\mathbb{P}(v^{\prime})}\stackrel{{(a)}}{{=}} \frac{\mathbb{P}_{h}(x\mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}( x\mid v^{\prime})}\stackrel{{(b)}}{{=}}\frac{\mathbb{P}_{h}(\phi^{*}(x)) \mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}(\phi^{*}(x))\mid v^{ \prime})},\]

\[\mathbb{P}_{h}(x\mid v)=q\left(x\mid\phi^{*}(x),\phi^{*}_{\xi}(x)\right) \mathbb{P}_{h}(\phi^{*}(x)\mid v)\mathbb{P}_{h}(\phi^{*}_{\xi}(x)).\] Let \(v_{1},v_{2}\in\Upsilon_{h}\) be two paths with indices \(i\) and \(j\) respectively. We define their exact path prediction gap as \(\Delta(i,j):=\mathbb{E}_{x_{h}}\left[\left|f^{*}_{h}(i\mid x_{h})-f^{*}_{h}(j \mid x_{h})\right|\right]\). Assume that \(v_{1}\) visits an endogenous state \(s\) at time step \(h\) and denote \(\omega(s)\) as the number of paths in \(\Upsilon_{h}\) that reaches \(s\). Then \(f^{*}_{h}(i\mid x_{h})=\nicefrac{{1}}{{\omega(s)}}\) if \(\phi^{*}(x_{h})=s\), and \(0\) otherwise. If \(v_{2}\) also visits \(s\) at time step \(h\), then \(f^{*}_{h}(i\mid x_{h})=f^{*}_{h}(j\mid x_{h})\) all \(x_{h}\). This implies \(\Delta(i,j)=0\) and \(\mathtt{PPE}\) will filter out the path with higher index since it detected both paths reach to the same endogenous state. Conversely, let \(v_{2}\) visit a different state at time step \(h\). If \(x\) is an observation that maps to \(s\), then \(f^{*}_{h}(i\mid x)=\nicefrac{{1}}{{\omega(s)}}\) and \(f^{*}_{h}(j\mid x)=0\). This gives \(\left|f^{*}_{h}(i\mid x)-f^{*}_{h}(j\mid x)\right|=\nicefrac{{1}}{{\omega(s)} }\geq\nicefrac{{1}}{{\left|\Upsilon_{h}\right|}}\) and, consequently, \(\Delta(i,j)>0\). In fact, we can show \(\Delta(i,j)\geq\mathcal{O}(\nicefrac{{1}}{{1}}\nicefrac{{1}}{{\left|\Upsilon_ {h}\right|}})\). Thus, \(\mathtt{PPE}\) will not eliminate these paths upon comparison.

The combination lock problem is defined for a given horizon \(H\) by an endogenous state space \(\mathcal{S}=\{s_{1,a}\}\cup\{s_{h,a},s_{h,b},s_{h,c}\}_{h=2}^{H}\), an exogenous state space \(\Xi=\{0,1\}^{H}\), an action space \(\mathcal{A}\) with 10 actions, and a deterministic endogenous start state of \(s_{1,a}\). For any state \(s_{h,g}\) we call \(g\) as its _type_ which can be \(a,b\) or \(c\). States with type \(a\) and \(b\) are considered _good_ states and those with type \(c\) are considered _bad_ states. Each instance of this problem is defined by two good action sequences \((a_{h})^{H}_{h=2},(a^{\prime}_{h})^{H}_{h=2}\) with \(a_{h}\neq a^{\prime}_{h}\), which are chosen uniformly randomly and kept fixed throughout. At \(h=1\), the agent is in \(s_{1,a}\) and action \(a_{1}\) leads to \(s_{2,a}\), \(a^{\prime}_{h}\) leads to \(s_{2,b}\), and all other actions lead to \(s_{2,c}\). For \(h>2\), taking action \(a_{h}\) in \(s_{h,a}\) leads to \(s_{h+1,a}\) and taking action \(a^{\prime}_{h}\) in \(s_{h,b}\) leads to \(s_{h+1,b}\). In all other cases involving taking an action in a state \(s_{h,g}\), we transition to the next bad state \(s_{h+1,c}\).

The exogenous state evolves as follows. We set \(\xi_{1}\in\{0,1\}^{H}\) where \(\xi_{1}(i)\sim\mathtt{Unf}\left(\{0,1\}\right)\) for each \(i\in[H]\). At time step \(h\), \(\xi_{h}\) is generated from \(\xi_{h-1}\) by uniformly flipping each bit in \(\xi_{h-1}\) independently with probability 0.1.

An observation \(x\) is generated stochastically from a latent state \(z=(s,\xi)\). We map \(s\) to a vector \(w\) encoding the identity of the state. We concatenate \((w,\xi)\), add Gaussian noise to each dimension, and multiply the result by a Hadamard matrix to generate \(x\). See Appendix F for full details. Our construction is inspired by prior work (Du et al., 2019; Misra et al., 2020).

Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.

When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.

On adding exogenous noise, the accuracy of Homer plummets significantly. The accuracy of ID also drops but this drop is mild since unlike Homer, the ID objective is able to filter exogenous noise.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2

We add exogenous noise as follows: at the beginning of each episode, we independently sample position, size and color of 5 ellipses.

The exogenous state is given by the position, size and color of ellipses and is much larger than \(|\mathcal{S}|\leq 4N^{2}\).

Since the problem has deterministic dynamics, we evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm's mapping from observations to endogenous latent states.

We also evaluate the accuracy of state decoding on this problem. We compare the state decoding accuracy of PPE and Homer at \(H=2\) using an identical evaluation setup to the one we used for combination lock. Figure 2(d) shows the results. As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder. In this work, we introduce the EX-BMDP setting, an RL setting that models exogenous noise, ubiquitous in many real-world systems.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments. To our knowledge, this is the first such algorithm with guarantees for settings with exogenous noise.

Input: Figure 1: Left: An agent is walking next to a pond in a park and observes the world as an image. The world consists of a latent endogenous state, containing variable such as agent’s position, and a much larger latent exogenous state containing variables such as motion of ducks, ripples in the water, etc. Center: Graphical model of the EX-BMDP. Right: PPE learns a generalized form of inverse dynamics that recovers the endogenous state.

Input: Figure 2: Results on combination lock. Left: We show the latent transition dynamics of combination lock. Observations are not shown for brevity. Center: Shows minimal number of episodes needed to achieve a mean regret of at most V (π?)/2. Right: State decoding accuracy (in percent) of decoders learned by different methods. Solid lines implies no exogenous dimension while dashed lines imply an exogenous dimension of 100.

Input: Figure 3: Results on visual grid world. Left two: Shows sampled observations for the first two steps from the visual gridworld domain. The agent is depicted as a red-triangle, lava in orange, walls in grey, and the goal in green. Center Right: Shows errors of type 1 and type 2 made by the PPE in recovering the latent endogenous dynamics. Right: State decoding accuracy of PPE, Homer and a random uniform decoder. (see Section 6)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll assess the claim: "How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?"

The evidence strongly supports this claim through both theoretical analysis and experimental results. The paper:
- Demonstrates mathematically that PPE's sample complexity scales with endogenous state size (S) rather than full state space (S|Ξ|)
- Shows empirically in combination lock and grid world experiments that PPE successfully learns with exogenous noise while maintaining good performance
- Provides theoretical guarantees for near-deterministic settings
- Includes experimental results showing PPE achieves high accuracy in state decoding and model learning despite exogenous noise

The claim is well-stated and accurately reflects the capabilities demonstrated in the evidence without overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?

The review comment to be evaluated is:

The paper is very well motivated. It has convinced me that being invariant to exogenous noise is important and that the EX-BMDP is useful abstraction for making this problem concrete. It is also very clearly presented. The argumentation is clear, and the flow of the paper is quite natural. Even the lengthy appendix is fairly easy to parse, with straightforward proofs and relevant details (e.g. specific claims about the unsuitability of alternative approaches).

One shortcoming is potential impact / applicability. I'm not convinced that environments with single starting states and near-deterministic dynamics is a very rich problem class. Or more specifically, I'm not convinced that restricting myself to this class would be preferable to working in the full space of MDPs with e.g. the possibility of aliasing a few states with a single-step inverse-dynamics approach. I am aware that impact is very hard to predict in advance, so I won't let this aspect unduly affect my scoring. And I'm convinced that even if PPE is never used in practice, this paper should still be accepted for its useful problem formulation and theoretical results. But a quick read would leave one with the impression that the alternatives (e.g. contrastive learning, inverse dynamics, bisimulation metrics) flaws outweigh their considerable advantages (e.g. applicability with stochastic dynamics) --  explicitly noting where alternative approaches are preferable would be appreciated.

I also noticed what I believe to be an omission in your related works: mutual information / empowerment based methods. "Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning" seems particularly related in its search for predicable action sequences. And the follow-up work "Variational Intrinsic Control" appears to be a multi-step inverse dynamics analog that doesn't require open-loop policies. If you can show that these approaches are flawed in a way that PPE is not, then that would considerably raise my assessment of this work. And even if not, I believe this field (if not these specific works) should be acknowledged, as they similarly attempt to learn what in the environment is controllable.

While I agree that PPE is applicable in no-reward situations whereas bi-simulation metrics are not, your experiments all involve rewards, so I'm surprised a bi-simulation metric method was not included as a baseline.

While this isn't necessary for acceptance, it is worth noting that prior work has established much more challenging benchmarks for evaluating the representation of exogenous noise, and utilizing a pre-existing benchmark would make your empirical results considerably more impressive.

This is a small point, but it is not initially obvious why inverse dynamics fails on the combination lock problem. Does alliasing occur just when e.g. s_2a --> s_3a and s_2b-->s_3b have the same action?

The evidence to be evaluated is:We introduce a novel _Exogenous Block Markov Decision Process_ (EX-BMDP) setting to model systems with exogenous noise.

We start with describing the Block Markov Decision Process (BMDP) Du et al. (2019). This process consists of a finite set of observations \(\mathcal{X}\), a set of _latent_ states \(\mathcal{Z}\) with cardinality \(Z\), a finite set of actions \(\mathcal{A}\) with cardinality \(A\), a transition function \(T:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{Z})\), an emission function \(q:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{X})\), a reward function \(R:\mathcal{X}\times\mathcal{A}\rightarrow[0,1]\), a horizon \(H\in\mathbb{N}\), and a start state distribution \(\mu\in\Delta(\mathcal{Z})\). The agent interacts with the environment by repeatedly generating \(H\)-step trajectories \((z_{1},x_{1},a_{1},r_{1},\cdots,z_{H},x_{H},a_{H},r_{H})\) where \(z_{1}\sim\mu(\cdot)\) and for every \(h\in[H]\) we have \(x_{h}\sim q(\cdot\mid z_{h})\), \(r_{h}=R(x_{h},a_{h})\), and if \(h<H\), then \(z_{h+1}\sim T(\cdot\mid z_{h},a_{h})\). The agent does not observe the states \((z_{1},\cdots,z_{H})\), instead receiving only the observations \((x_{1},\cdots,x_{H})\) and rewards \((r_{1},\cdots,r_{H})\). We assume that the emission distributions of any two latent states are disjoint, usually referred as the _block assumption_: \(supp(q(\cdot\mid z_{1}))\cap supp(q(\cdot\mid z_{2}))=\emptyset\) when \(z_{1}\neq z_{2}\). The agent chooses actions using a policy \(\pi:\mathcal{X}\rightarrow\Delta(\mathcal{A})\). We also define the set of non-stationary policies \(\Pi_{\mathrm{NS}}=\Pi^{H}\) as a \(H\)-length tuple, with \((\pi_{1},\cdots,\pi_{H})\in\Pi_{\mathrm{NS}}\) denoting that the action at time step \(h\) is taken as \(a_{h}\sim\pi_{h}(\cdot\mid x_{h})\). The value \(V(\pi)\) of a policy \(\pi\) is the expected episodic sum of rewards \(V(\pi):=\mathbb{E}_{\pi}[\sum_{h=1}^{H}R(x_{h},a_{h})]\).

_An EX-BMDP is a BMDP such that the latent state can be decoupled into two parts \(z=(s,\xi)\) where \(s\in\mathcal{S}\) is the endogenous state and \(\xi\in\Xi\) is the exogenous state.

The endogenous state \(s\) captures the information that can be manipulated by the agent.

We assume that the set of all endogenous states \(\mathcal{S}\) is finite with cardinality \(S\). The exogenous state \(\xi\) captures all the other information that the agent cannot control and does not affect the information it can manipulate. Again, we make no assumptions on the exogenous dynamics nor on its cardinality \(|\Xi|\) which may be arbitrarily large.

The decoupled dynamics assumption made in the EX-BMDP setting is a natural way to characterize exogenous noise; the type of noise that is not affected by our actions nor affects the endogenous state but may have non-trivial dynamic. This decoupling captures the movement of ducks, captured in the visual field of the agent in Figure 1, and many additional exogenous processes (e.g., movement of clouds in a navigation task).

Thus, for a BMDP, the cardinality of the policy cover scales with \(|\mathcal{Z}|\). The structure of EX-BMDPs allows to reduce the size of the policy cover significantly to \(|\mathcal{S}|\ll|\mathcal{Z}|=|\mathcal{S}|\,|\Xi|\) when the size of the exogenous state space is large. Specifically, we show that the set of policies that reach each _endogenous_ state, and _do not depend on the exogenous_ part of the state is also a policy cover (see Appendix B, Proposition 4).

Unfortunately, in the presence of exogenous noise, Homer distinguishes between two transitions that represent transition between the same latent endogenous states but different exogenous states. In our walk in the park example, even if the agent moves between same points in two transitions, the model maybe able to tell these transitions apart by looking at the position of ducks which may have different behaviour in the two transitions. This results in the Homer creating \(\mathcal{O}(|\mathcal{Z}|)\) many abstract states. We call this the _under-abstraction_ problem.

Such a representation can ignore all information that is not relevant for action prediction, which includes all exogenous/uncontrollable information.

Zhang et al. (2020) proposed learning a bisimulation metric to learn a representation which is invariant to exogenous noise.

Bellman rank might depend on \(|\Xi|\). The Bellman rank was introduced in Jiang et al. (2017) as a complexity measure for the learnability of an RL problem with function approximations.

However, we show in Appendix A that Bellman rank for EX-BMDP can scale as \(\mathcal{O}(|\Xi|)\). This shows that EX-BMDP is a highly non-trivial setting as we don't even have sample-efficient algorithms regardless of computationally-efficient.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.

This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action.

Recovering an endogenous state decoder. We can recover a endogenous state decoder \(\hat{\phi}_{h}\) for each time step \(h\in\{2,3,\cdots,H\}\) directly from \(\hat{f}_{h}\) as shown below: \[\hat{\phi}_{h}(x)=\min\left\{i\mid\hat{f}_{h}(i\mid x)\geq\max_{j}\hat{f}_{h}(j \mid x)-\mathcal{O}(\nicefrac{{1}}{{\Gamma_{h}}}),i\in[\left|\Upsilon_{h} \right|]\right\}.\] Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting \(x\), and therefore, \(\phi^{\star}(x)\). We are implicitly using the decoder for exploring, since we rely on using \(\hat{f}_{h}\) for making planning decisions.

PPE can also be used to recover a latent endogenous transition dynamics.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

If the reward function only depends on the endogenous state, we can use a computationally cheaper value-iteration VI that uses the recovered transition dynamics. VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details). In each case, the sample complexity of learning a near-optimal policy, given the output of PPE, scales with the size of endogenous and not the exogenous state space.

**Assumption 1** (Near deterministic endogenous dynamics).: _We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\).

However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space. Further, since our analysis only uses standard uniform convergence arguments, it extends straightforwardly to infinitely large function classes by replacing \(\ln|\mathcal{F}|\) with other suitable complexity measures such as Rademacher complexity.

Let \(\mathbb{P}_{h}(\xi)\) denote the distribution over exogenous states at time step \(h\) which is independent of agent's policy. The Bayes optimal classifier (\(f^{*}_{h}:=f_{\Upsilon_{h},h}\)) of the prediction problem can be derived as: \[f^{*}_{h}(\texttt{idx}(v)\mid x):=\mathbb{P}_{h}(v\mid x)=\frac{ \mathbb{P}_{h}(x\mid v)\mathbb{P}(v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{ P}_{h}(x\mid v^{\prime})\mathbb{P}(v^{\prime})}\stackrel{{(a)}}{{=}} \frac{\mathbb{P}_{h}(x\mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}( x\mid v^{\prime})}\stackrel{{(b)}}{{=}}\frac{\mathbb{P}_{h}(\phi^{*}(x)) \mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}(\phi^{*}(x))\mid v^{ \prime})},\]

\[\mathbb{P}_{h}(x\mid v)=q\left(x\mid\phi^{*}(x),\phi^{*}_{\xi}(x)\right) \mathbb{P}_{h}(\phi^{*}(x)\mid v)\mathbb{P}_{h}(\phi^{*}_{\xi}(x)).\] Let \(v_{1},v_{2}\in\Upsilon_{h}\) be two paths with indices \(i\) and \(j\) respectively. We define their exact path prediction gap as \(\Delta(i,j):=\mathbb{E}_{x_{h}}\left[\left|f^{*}_{h}(i\mid x_{h})-f^{*}_{h}(j \mid x_{h})\right|\right]\). Assume that \(v_{1}\) visits an endogenous state \(s\) at time step \(h\) and denote \(\omega(s)\) as the number of paths in \(\Upsilon_{h}\) that reaches \(s\). Then \(f^{*}_{h}(i\mid x_{h})=\nicefrac{{1}}{{\omega(s)}}\) if \(\phi^{*}(x_{h})=s\), and \(0\) otherwise. If \(v_{2}\) also visits \(s\) at time step \(h\), then \(f^{*}_{h}(i\mid x_{h})=f^{*}_{h}(j\mid x_{h})\) all \(x_{h}\). This implies \(\Delta(i,j)=0\) and \(\mathtt{PPE}\) will filter out the path with higher index since it detected both paths reach to the same endogenous state. Conversely, let \(v_{2}\) visit a different state at time step \(h\). If \(x\) is an observation that maps to \(s\), then \(f^{*}_{h}(i\mid x)=\nicefrac{{1}}{{\omega(s)}}\) and \(f^{*}_{h}(j\mid x)=0\). This gives \(\left|f^{*}_{h}(i\mid x)-f^{*}_{h}(j\mid x)\right|=\nicefrac{{1}}{{\omega(s)} }\geq\nicefrac{{1}}{{\left|\Upsilon_{h}\right|}}\) and, consequently, \(\Delta(i,j)>0\). In fact, we can show \(\Delta(i,j)\geq\mathcal{O}(\nicefrac{{1}}{{1}}\nicefrac{{1}}{{\left|\Upsilon_ {h}\right|}})\). Thus, \(\mathtt{PPE}\) will not eliminate these paths upon comparison.

The combination lock problem is defined for a given horizon \(H\) by an endogenous state space \(\mathcal{S}=\{s_{1,a}\}\cup\{s_{h,a},s_{h,b},s_{h,c}\}_{h=2}^{H}\), an exogenous state space \(\Xi=\{0,1\}^{H}\), an action space \(\mathcal{A}\) with 10 actions, and a deterministic endogenous start state of \(s_{1,a}\). For any state \(s_{h,g}\) we call \(g\) as its _type_ which can be \(a,b\) or \(c\). States with type \(a\) and \(b\) are considered _good_ states and those with type \(c\) are considered _bad_ states. Each instance of this problem is defined by two good action sequences \((a_{h})^{H}_{h=2},(a^{\prime}_{h})^{H}_{h=2}\) with \(a_{h}\neq a^{\prime}_{h}\), which are chosen uniformly randomly and kept fixed throughout. At \(h=1\), the agent is in \(s_{1,a}\) and action \(a_{1}\) leads to \(s_{2,a}\), \(a^{\prime}_{h}\) leads to \(s_{2,b}\), and all other actions lead to \(s_{2,c}\). For \(h>2\), taking action \(a_{h}\) in \(s_{h,a}\) leads to \(s_{h+1,a}\) and taking action \(a^{\prime}_{h}\) in \(s_{h,b}\) leads to \(s_{h+1,b}\). In all other cases involving taking an action in a state \(s_{h,g}\), we transition to the next bad state \(s_{h+1,c}\).

The exogenous state evolves as follows. We set \(\xi_{1}\in\{0,1\}^{H}\) where \(\xi_{1}(i)\sim\mathtt{Unf}\left(\{0,1\}\right)\) for each \(i\in[H]\). At time step \(h\), \(\xi_{h}\) is generated from \(\xi_{h-1}\) by uniformly flipping each bit in \(\xi_{h-1}\) independently with probability 0.1.

An observation \(x\) is generated stochastically from a latent state \(z=(s,\xi)\). We map \(s\) to a vector \(w\) encoding the identity of the state. We concatenate \((w,\xi)\), add Gaussian noise to each dimension, and multiply the result by a Hadamard matrix to generate \(x\). See Appendix F for full details. Our construction is inspired by prior work (Du et al., 2019; Misra et al., 2020).

Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.

When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.

On adding exogenous noise, the accuracy of Homer plummets significantly. The accuracy of ID also drops but this drop is mild since unlike Homer, the ID objective is able to filter exogenous noise.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2

We add exogenous noise as follows: at the beginning of each episode, we independently sample position, size and color of 5 ellipses.

The exogenous state is given by the position, size and color of ellipses and is much larger than \(|\mathcal{S}|\leq 4N^{2}\).

Since the problem has deterministic dynamics, we evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm's mapping from observations to endogenous latent states.

We also evaluate the accuracy of state decoding on this problem. We compare the state decoding accuracy of PPE and Homer at \(H=2\) using an identical evaluation setup to the one we used for combination lock. Figure 2(d) shows the results. As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder. In this work, we introduce the EX-BMDP setting, an RL setting that models exogenous noise, ubiquitous in many real-world systems.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments. To our knowledge, this is the first such algorithm with guarantees for settings with exogenous noise.

Input: Figure 1: Left: An agent is walking next to a pond in a park and observes the world as an image. The world consists of a latent endogenous state, containing variable such as agent’s position, and a much larger latent exogenous state containing variables such as motion of ducks, ripples in the water, etc. Center: Graphical model of the EX-BMDP. Right: PPE learns a generalized form of inverse dynamics that recovers the endogenous state.

Input: Figure 2: Results on combination lock. Left: We show the latent transition dynamics of combination lock. Observations are not shown for brevity. Center: Shows minimal number of episodes needed to achieve a mean regret of at most V (π?)/2. Right: State decoding accuracy (in percent) of decoders learned by different methods. Solid lines implies no exogenous dimension while dashed lines imply an exogenous dimension of 100.

Input: Figure 3: Results on visual grid world. Left two: Shows sampled observations for the first two steps from the visual gridworld domain. The agent is depicted as a red-triangle, lava in orange, walls in grey, and the goal in green. Center Right: Shows errors of type 1 and type 2 made by the PPE in recovering the latent endogenous dynamics. Right: State decoding accuracy of PPE, Homer and a random uniform decoder. (see Section 6)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the comprehensive evidence provided, including theoretical framework, algorithm details, experimental results, and reviewer comments, I'll evaluate the claim: "How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?"

Summary: The evidence strongly supports this claim through:
1. Theoretical guarantees showing sample complexity depends on endogenous state size not exogenous
2. Empirical validation in combination lock and visual grid world experiments
3. Clear mathematical formulation of EX-BMDP setting
4. Demonstrated success of PPE algorithm in filtering exogenous noise
5. Reviewer acknowledges strong theoretical results

The claim is well-stated and accurately reflects the paper's contributions without exaggeration. The limitations (near-deterministic dynamics requirement) are acknowledged.

<score>1</score>

Justification:
1. The paper provides formal theoretical guarantees backing the claim: "Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space."
2. Experimental results demonstrate effective learning despite large exogenous state spaces (e.g. "The exogenous state is given by the position, size and color of ellipses and is much larger than |S|≤4N²")
3. The reviewer notes "the paper should still be accepted for its useful problem formulation and theoretical results"
4. Limitations are clearly acknowledged (e.g. Assumption 1 on near-deterministic dynamics)
5. No evidence of overstatement - claims align precisely with theoretical and empirical results presented


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?

The review comment to be evaluated is:

Strengths:

- The problem setting is important and seems to be relevant to many real-world problems.
- While the idea of using inverse dynamics and even multi-step inverse dynamics [1] to learn representations in these types of environments to learn controllable representations is not novel, this paper provides a theoretical basis for this choice, which I think is valuable.
- I found sections 1 through 5 to be very well written.

Weaknesses:

- In the experiment section, it seems as though you are testing the planning/policy learning abilities with PPE in both of the environments. However, as far as I can tell, there is no indication of how you perform this planning in PPE in the main text. The paper simply switches from talking about learning policy covers to showing regret for environments with only a small mention of the fact that you could plan using the data collected with PPE. The way the experiments are run and the choice to test the planning performance / how the state representations are obtained in experiments should be explained in more detail in my opinion.
- The paper discusses prior works with inverse dynamics seemingly only in the context of their applications in representation learning in two prior works on exploration. I think the application of multi-step inverse dynamics in goal-conditioned environments described in [1] is worth mentioning, since for a fixed goal this algorithm seems like it is doing something similar to PPL. Related is the lack of a "related works" section in the paper, which would be helpful for framing PPL in the context of prior works.
- There is no conclusion section, which I feel would improve the paper.

My major questions are:

1. How is the policy learning done in the experiment section? How are representations found in the decoding accuracy plot?
2. How does this paper relate to prior works on inverse dynamics?

[1] Paster, Keiran, et al. Planning from Pixels Using Inverse Dynamics Models. 2020. openreview.net, https://openreview.net/forum?id=V6BjBgku7Ro.

The evidence to be evaluated is:We introduce a novel _Exogenous Block Markov Decision Process_ (EX-BMDP) setting to model systems with exogenous noise.

We start with describing the Block Markov Decision Process (BMDP) Du et al. (2019). This process consists of a finite set of observations \(\mathcal{X}\), a set of _latent_ states \(\mathcal{Z}\) with cardinality \(Z\), a finite set of actions \(\mathcal{A}\) with cardinality \(A\), a transition function \(T:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{Z})\), an emission function \(q:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{X})\), a reward function \(R:\mathcal{X}\times\mathcal{A}\rightarrow[0,1]\), a horizon \(H\in\mathbb{N}\), and a start state distribution \(\mu\in\Delta(\mathcal{Z})\). The agent interacts with the environment by repeatedly generating \(H\)-step trajectories \((z_{1},x_{1},a_{1},r_{1},\cdots,z_{H},x_{H},a_{H},r_{H})\) where \(z_{1}\sim\mu(\cdot)\) and for every \(h\in[H]\) we have \(x_{h}\sim q(\cdot\mid z_{h})\), \(r_{h}=R(x_{h},a_{h})\), and if \(h<H\), then \(z_{h+1}\sim T(\cdot\mid z_{h},a_{h})\). The agent does not observe the states \((z_{1},\cdots,z_{H})\), instead receiving only the observations \((x_{1},\cdots,x_{H})\) and rewards \((r_{1},\cdots,r_{H})\). We assume that the emission distributions of any two latent states are disjoint, usually referred as the _block assumption_: \(supp(q(\cdot\mid z_{1}))\cap supp(q(\cdot\mid z_{2}))=\emptyset\) when \(z_{1}\neq z_{2}\). The agent chooses actions using a policy \(\pi:\mathcal{X}\rightarrow\Delta(\mathcal{A})\). We also define the set of non-stationary policies \(\Pi_{\mathrm{NS}}=\Pi^{H}\) as a \(H\)-length tuple, with \((\pi_{1},\cdots,\pi_{H})\in\Pi_{\mathrm{NS}}\) denoting that the action at time step \(h\) is taken as \(a_{h}\sim\pi_{h}(\cdot\mid x_{h})\). The value \(V(\pi)\) of a policy \(\pi\) is the expected episodic sum of rewards \(V(\pi):=\mathbb{E}_{\pi}[\sum_{h=1}^{H}R(x_{h},a_{h})]\).

_An EX-BMDP is a BMDP such that the latent state can be decoupled into two parts \(z=(s,\xi)\) where \(s\in\mathcal{S}\) is the endogenous state and \(\xi\in\Xi\) is the exogenous state.

The endogenous state \(s\) captures the information that can be manipulated by the agent.

We assume that the set of all endogenous states \(\mathcal{S}\) is finite with cardinality \(S\). The exogenous state \(\xi\) captures all the other information that the agent cannot control and does not affect the information it can manipulate. Again, we make no assumptions on the exogenous dynamics nor on its cardinality \(|\Xi|\) which may be arbitrarily large.

The decoupled dynamics assumption made in the EX-BMDP setting is a natural way to characterize exogenous noise; the type of noise that is not affected by our actions nor affects the endogenous state but may have non-trivial dynamic. This decoupling captures the movement of ducks, captured in the visual field of the agent in Figure 1, and many additional exogenous processes (e.g., movement of clouds in a navigation task).

Thus, for a BMDP, the cardinality of the policy cover scales with \(|\mathcal{Z}|\). The structure of EX-BMDPs allows to reduce the size of the policy cover significantly to \(|\mathcal{S}|\ll|\mathcal{Z}|=|\mathcal{S}|\,|\Xi|\) when the size of the exogenous state space is large. Specifically, we show that the set of policies that reach each _endogenous_ state, and _do not depend on the exogenous_ part of the state is also a policy cover (see Appendix B, Proposition 4).

Unfortunately, in the presence of exogenous noise, Homer distinguishes between two transitions that represent transition between the same latent endogenous states but different exogenous states. In our walk in the park example, even if the agent moves between same points in two transitions, the model maybe able to tell these transitions apart by looking at the position of ducks which may have different behaviour in the two transitions. This results in the Homer creating \(\mathcal{O}(|\mathcal{Z}|)\) many abstract states. We call this the _under-abstraction_ problem.

Such a representation can ignore all information that is not relevant for action prediction, which includes all exogenous/uncontrollable information.

Zhang et al. (2020) proposed learning a bisimulation metric to learn a representation which is invariant to exogenous noise.

Bellman rank might depend on \(|\Xi|\). The Bellman rank was introduced in Jiang et al. (2017) as a complexity measure for the learnability of an RL problem with function approximations.

However, we show in Appendix A that Bellman rank for EX-BMDP can scale as \(\mathcal{O}(|\Xi|)\). This shows that EX-BMDP is a highly non-trivial setting as we don't even have sample-efficient algorithms regardless of computationally-efficient.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.

This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action.

Recovering an endogenous state decoder. We can recover a endogenous state decoder \(\hat{\phi}_{h}\) for each time step \(h\in\{2,3,\cdots,H\}\) directly from \(\hat{f}_{h}\) as shown below: \[\hat{\phi}_{h}(x)=\min\left\{i\mid\hat{f}_{h}(i\mid x)\geq\max_{j}\hat{f}_{h}(j \mid x)-\mathcal{O}(\nicefrac{{1}}{{\Gamma_{h}}}),i\in[\left|\Upsilon_{h} \right|]\right\}.\] Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting \(x\), and therefore, \(\phi^{\star}(x)\). We are implicitly using the decoder for exploring, since we rely on using \(\hat{f}_{h}\) for making planning decisions.

PPE can also be used to recover a latent endogenous transition dynamics.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

If the reward function only depends on the endogenous state, we can use a computationally cheaper value-iteration VI that uses the recovered transition dynamics. VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details). In each case, the sample complexity of learning a near-optimal policy, given the output of PPE, scales with the size of endogenous and not the exogenous state space.

**Assumption 1** (Near deterministic endogenous dynamics).: _We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\).

However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space. Further, since our analysis only uses standard uniform convergence arguments, it extends straightforwardly to infinitely large function classes by replacing \(\ln|\mathcal{F}|\) with other suitable complexity measures such as Rademacher complexity.

Let \(\mathbb{P}_{h}(\xi)\) denote the distribution over exogenous states at time step \(h\) which is independent of agent's policy. The Bayes optimal classifier (\(f^{*}_{h}:=f_{\Upsilon_{h},h}\)) of the prediction problem can be derived as: \[f^{*}_{h}(\texttt{idx}(v)\mid x):=\mathbb{P}_{h}(v\mid x)=\frac{ \mathbb{P}_{h}(x\mid v)\mathbb{P}(v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{ P}_{h}(x\mid v^{\prime})\mathbb{P}(v^{\prime})}\stackrel{{(a)}}{{=}} \frac{\mathbb{P}_{h}(x\mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}( x\mid v^{\prime})}\stackrel{{(b)}}{{=}}\frac{\mathbb{P}_{h}(\phi^{*}(x)) \mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}(\phi^{*}(x))\mid v^{ \prime})},\]

\[\mathbb{P}_{h}(x\mid v)=q\left(x\mid\phi^{*}(x),\phi^{*}_{\xi}(x)\right) \mathbb{P}_{h}(\phi^{*}(x)\mid v)\mathbb{P}_{h}(\phi^{*}_{\xi}(x)).\] Let \(v_{1},v_{2}\in\Upsilon_{h}\) be two paths with indices \(i\) and \(j\) respectively. We define their exact path prediction gap as \(\Delta(i,j):=\mathbb{E}_{x_{h}}\left[\left|f^{*}_{h}(i\mid x_{h})-f^{*}_{h}(j \mid x_{h})\right|\right]\). Assume that \(v_{1}\) visits an endogenous state \(s\) at time step \(h\) and denote \(\omega(s)\) as the number of paths in \(\Upsilon_{h}\) that reaches \(s\). Then \(f^{*}_{h}(i\mid x_{h})=\nicefrac{{1}}{{\omega(s)}}\) if \(\phi^{*}(x_{h})=s\), and \(0\) otherwise. If \(v_{2}\) also visits \(s\) at time step \(h\), then \(f^{*}_{h}(i\mid x_{h})=f^{*}_{h}(j\mid x_{h})\) all \(x_{h}\). This implies \(\Delta(i,j)=0\) and \(\mathtt{PPE}\) will filter out the path with higher index since it detected both paths reach to the same endogenous state. Conversely, let \(v_{2}\) visit a different state at time step \(h\). If \(x\) is an observation that maps to \(s\), then \(f^{*}_{h}(i\mid x)=\nicefrac{{1}}{{\omega(s)}}\) and \(f^{*}_{h}(j\mid x)=0\). This gives \(\left|f^{*}_{h}(i\mid x)-f^{*}_{h}(j\mid x)\right|=\nicefrac{{1}}{{\omega(s)} }\geq\nicefrac{{1}}{{\left|\Upsilon_{h}\right|}}\) and, consequently, \(\Delta(i,j)>0\). In fact, we can show \(\Delta(i,j)\geq\mathcal{O}(\nicefrac{{1}}{{1}}\nicefrac{{1}}{{\left|\Upsilon_ {h}\right|}})\). Thus, \(\mathtt{PPE}\) will not eliminate these paths upon comparison.

The combination lock problem is defined for a given horizon \(H\) by an endogenous state space \(\mathcal{S}=\{s_{1,a}\}\cup\{s_{h,a},s_{h,b},s_{h,c}\}_{h=2}^{H}\), an exogenous state space \(\Xi=\{0,1\}^{H}\), an action space \(\mathcal{A}\) with 10 actions, and a deterministic endogenous start state of \(s_{1,a}\). For any state \(s_{h,g}\) we call \(g\) as its _type_ which can be \(a,b\) or \(c\). States with type \(a\) and \(b\) are considered _good_ states and those with type \(c\) are considered _bad_ states. Each instance of this problem is defined by two good action sequences \((a_{h})^{H}_{h=2},(a^{\prime}_{h})^{H}_{h=2}\) with \(a_{h}\neq a^{\prime}_{h}\), which are chosen uniformly randomly and kept fixed throughout. At \(h=1\), the agent is in \(s_{1,a}\) and action \(a_{1}\) leads to \(s_{2,a}\), \(a^{\prime}_{h}\) leads to \(s_{2,b}\), and all other actions lead to \(s_{2,c}\). For \(h>2\), taking action \(a_{h}\) in \(s_{h,a}\) leads to \(s_{h+1,a}\) and taking action \(a^{\prime}_{h}\) in \(s_{h,b}\) leads to \(s_{h+1,b}\). In all other cases involving taking an action in a state \(s_{h,g}\), we transition to the next bad state \(s_{h+1,c}\).

The exogenous state evolves as follows. We set \(\xi_{1}\in\{0,1\}^{H}\) where \(\xi_{1}(i)\sim\mathtt{Unf}\left(\{0,1\}\right)\) for each \(i\in[H]\). At time step \(h\), \(\xi_{h}\) is generated from \(\xi_{h-1}\) by uniformly flipping each bit in \(\xi_{h-1}\) independently with probability 0.1.

An observation \(x\) is generated stochastically from a latent state \(z=(s,\xi)\). We map \(s\) to a vector \(w\) encoding the identity of the state. We concatenate \((w,\xi)\), add Gaussian noise to each dimension, and multiply the result by a Hadamard matrix to generate \(x\). See Appendix F for full details. Our construction is inspired by prior work (Du et al., 2019; Misra et al., 2020).

Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.

When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.

On adding exogenous noise, the accuracy of Homer plummets significantly. The accuracy of ID also drops but this drop is mild since unlike Homer, the ID objective is able to filter exogenous noise.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2

We add exogenous noise as follows: at the beginning of each episode, we independently sample position, size and color of 5 ellipses.

The exogenous state is given by the position, size and color of ellipses and is much larger than \(|\mathcal{S}|\leq 4N^{2}\).

Since the problem has deterministic dynamics, we evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm's mapping from observations to endogenous latent states.

We also evaluate the accuracy of state decoding on this problem. We compare the state decoding accuracy of PPE and Homer at \(H=2\) using an identical evaluation setup to the one we used for combination lock. Figure 2(d) shows the results. As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder. In this work, we introduce the EX-BMDP setting, an RL setting that models exogenous noise, ubiquitous in many real-world systems.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments. To our knowledge, this is the first such algorithm with guarantees for settings with exogenous noise.

Input: Figure 1: Left: An agent is walking next to a pond in a park and observes the world as an image. The world consists of a latent endogenous state, containing variable such as agent’s position, and a much larger latent exogenous state containing variables such as motion of ducks, ripples in the water, etc. Center: Graphical model of the EX-BMDP. Right: PPE learns a generalized form of inverse dynamics that recovers the endogenous state.

Input: Figure 2: Results on combination lock. Left: We show the latent transition dynamics of combination lock. Observations are not shown for brevity. Center: Shows minimal number of episodes needed to achieve a mean regret of at most V (π?)/2. Right: State decoding accuracy (in percent) of decoders learned by different methods. Solid lines implies no exogenous dimension while dashed lines imply an exogenous dimension of 100.

Input: Figure 3: Results on visual grid world. Left two: Shows sampled observations for the first two steps from the visual gridworld domain. The agent is depicted as a red-triangle, lava in orange, walls in grey, and the goal in green. Center Right: Shows errors of type 1 and type 2 made by the PPE in recovering the latent endogenous dynamics. Right: State decoding accuracy of PPE, Homer and a random uniform decoder. (see Section 6)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The claim asking "How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?" is well-stated based on the evidence presented. The paper:

1. Clearly defines the EX-BMDP setting and demonstrates how it separates controllable endogenous states from uncontrollable exogenous noise
2. Provides theoretical analysis showing their PPE algorithm's sample complexity scales with endogenous state space size rather than full state space
3. Validates claims through experiments on combination lock and visual grid world environments, showing PPE effectively learns state representations while filtering exogenous noise
4. Acknowledges limitations like near-deterministic dynamics assumption

The claim accurately reflects what the evidence demonstrates without overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

_How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?

The review comment to be evaluated is:

STRENGTHS

(S1) The problem of efficiently learning in high-dimensional observation spaces is a good one, and the authors provided an excellent discussion of the technical components of this problem in the early parts of the paper. I thought the definition and presentation of the EX-BMDP was particularly good.

(S2) I appreciate the authors providing experimental results to show that PPE performs well in some regards.

WEAKNESSES

(W1) I'm afraid I found the discussion of PPE to be a bit difficult to decode. Even looking at Algorithm 1, I find myself with a number of important questions unanswered, chief among them being "what about $\phi^*_e$"? It strikes me as odd that nowhere in this algorithm are the inverse mappings from observation to exogenous/endogenous state represented or used. Why is this?

(W2) I'm a little concerned with the computational tractability of the algorithm. My understanding from the paper is that one seeks some sort of "covering" of the state space--in what sense is it practical to obtain and/or store this covering? If the problem setting consists of high-dimensional observations this seems especially challenging.

(W3) From my naive perspective, it seems that the ultimate goal here is to use PPE and then actually perform reinforcement learning. However, I didn't understand from the brief discussion at the end of Section 4 (or, frankly, from reading the Appendix) how exactly that would be accomplished. I feel this is important enough that it should appear in the main paper.

POST-DISCUSSION COMMENTS

During the discussion, I feel as though the authors adequately addressed each of the issues I raised above, and so I'm happy to raise my score to accept.

The evidence to be evaluated is:We introduce a novel _Exogenous Block Markov Decision Process_ (EX-BMDP) setting to model systems with exogenous noise.

We start with describing the Block Markov Decision Process (BMDP) Du et al. (2019). This process consists of a finite set of observations \(\mathcal{X}\), a set of _latent_ states \(\mathcal{Z}\) with cardinality \(Z\), a finite set of actions \(\mathcal{A}\) with cardinality \(A\), a transition function \(T:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{Z})\), an emission function \(q:\mathcal{Z}\times\mathcal{A}\rightarrow\Delta(\mathcal{X})\), a reward function \(R:\mathcal{X}\times\mathcal{A}\rightarrow[0,1]\), a horizon \(H\in\mathbb{N}\), and a start state distribution \(\mu\in\Delta(\mathcal{Z})\). The agent interacts with the environment by repeatedly generating \(H\)-step trajectories \((z_{1},x_{1},a_{1},r_{1},\cdots,z_{H},x_{H},a_{H},r_{H})\) where \(z_{1}\sim\mu(\cdot)\) and for every \(h\in[H]\) we have \(x_{h}\sim q(\cdot\mid z_{h})\), \(r_{h}=R(x_{h},a_{h})\), and if \(h<H\), then \(z_{h+1}\sim T(\cdot\mid z_{h},a_{h})\). The agent does not observe the states \((z_{1},\cdots,z_{H})\), instead receiving only the observations \((x_{1},\cdots,x_{H})\) and rewards \((r_{1},\cdots,r_{H})\). We assume that the emission distributions of any two latent states are disjoint, usually referred as the _block assumption_: \(supp(q(\cdot\mid z_{1}))\cap supp(q(\cdot\mid z_{2}))=\emptyset\) when \(z_{1}\neq z_{2}\). The agent chooses actions using a policy \(\pi:\mathcal{X}\rightarrow\Delta(\mathcal{A})\). We also define the set of non-stationary policies \(\Pi_{\mathrm{NS}}=\Pi^{H}\) as a \(H\)-length tuple, with \((\pi_{1},\cdots,\pi_{H})\in\Pi_{\mathrm{NS}}\) denoting that the action at time step \(h\) is taken as \(a_{h}\sim\pi_{h}(\cdot\mid x_{h})\). The value \(V(\pi)\) of a policy \(\pi\) is the expected episodic sum of rewards \(V(\pi):=\mathbb{E}_{\pi}[\sum_{h=1}^{H}R(x_{h},a_{h})]\).

_An EX-BMDP is a BMDP such that the latent state can be decoupled into two parts \(z=(s,\xi)\) where \(s\in\mathcal{S}\) is the endogenous state and \(\xi\in\Xi\) is the exogenous state.

The endogenous state \(s\) captures the information that can be manipulated by the agent.

We assume that the set of all endogenous states \(\mathcal{S}\) is finite with cardinality \(S\). The exogenous state \(\xi\) captures all the other information that the agent cannot control and does not affect the information it can manipulate. Again, we make no assumptions on the exogenous dynamics nor on its cardinality \(|\Xi|\) which may be arbitrarily large.

The decoupled dynamics assumption made in the EX-BMDP setting is a natural way to characterize exogenous noise; the type of noise that is not affected by our actions nor affects the endogenous state but may have non-trivial dynamic. This decoupling captures the movement of ducks, captured in the visual field of the agent in Figure 1, and many additional exogenous processes (e.g., movement of clouds in a navigation task).

Thus, for a BMDP, the cardinality of the policy cover scales with \(|\mathcal{Z}|\). The structure of EX-BMDPs allows to reduce the size of the policy cover significantly to \(|\mathcal{S}|\ll|\mathcal{Z}|=|\mathcal{S}|\,|\Xi|\) when the size of the exogenous state space is large. Specifically, we show that the set of policies that reach each _endogenous_ state, and _do not depend on the exogenous_ part of the state is also a policy cover (see Appendix B, Proposition 4).

Unfortunately, in the presence of exogenous noise, Homer distinguishes between two transitions that represent transition between the same latent endogenous states but different exogenous states. In our walk in the park example, even if the agent moves between same points in two transitions, the model maybe able to tell these transitions apart by looking at the position of ducks which may have different behaviour in the two transitions. This results in the Homer creating \(\mathcal{O}(|\mathcal{Z}|)\) many abstract states. We call this the _under-abstraction_ problem.

Such a representation can ignore all information that is not relevant for action prediction, which includes all exogenous/uncontrollable information.

Zhang et al. (2020) proposed learning a bisimulation metric to learn a representation which is invariant to exogenous noise.

Bellman rank might depend on \(|\Xi|\). The Bellman rank was introduced in Jiang et al. (2017) as a complexity measure for the learnability of an RL problem with function approximations.

However, we show in Appendix A that Bellman rank for EX-BMDP can scale as \(\mathcal{O}(|\Xi|)\). This shows that EX-BMDP is a highly non-trivial setting as we don't even have sample-efficient algorithms regardless of computationally-efficient.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.

This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action.

Recovering an endogenous state decoder. We can recover a endogenous state decoder \(\hat{\phi}_{h}\) for each time step \(h\in\{2,3,\cdots,H\}\) directly from \(\hat{f}_{h}\) as shown below: \[\hat{\phi}_{h}(x)=\min\left\{i\mid\hat{f}_{h}(i\mid x)\geq\max_{j}\hat{f}_{h}(j \mid x)-\mathcal{O}(\nicefrac{{1}}{{\Gamma_{h}}}),i\in[\left|\Upsilon_{h} \right|]\right\}.\] Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting \(x\), and therefore, \(\phi^{\star}(x)\). We are implicitly using the decoder for exploring, since we rely on using \(\hat{f}_{h}\) for making planning decisions.

PPE can also be used to recover a latent endogenous transition dynamics.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

If the reward function only depends on the endogenous state, we can use a computationally cheaper value-iteration VI that uses the recovered transition dynamics. VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details). In each case, the sample complexity of learning a near-optimal policy, given the output of PPE, scales with the size of endogenous and not the exogenous state space.

**Assumption 1** (Near deterministic endogenous dynamics).: _We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\).

However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space. Further, since our analysis only uses standard uniform convergence arguments, it extends straightforwardly to infinitely large function classes by replacing \(\ln|\mathcal{F}|\) with other suitable complexity measures such as Rademacher complexity.

Let \(\mathbb{P}_{h}(\xi)\) denote the distribution over exogenous states at time step \(h\) which is independent of agent's policy. The Bayes optimal classifier (\(f^{*}_{h}:=f_{\Upsilon_{h},h}\)) of the prediction problem can be derived as: \[f^{*}_{h}(\texttt{idx}(v)\mid x):=\mathbb{P}_{h}(v\mid x)=\frac{ \mathbb{P}_{h}(x\mid v)\mathbb{P}(v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{ P}_{h}(x\mid v^{\prime})\mathbb{P}(v^{\prime})}\stackrel{{(a)}}{{=}} \frac{\mathbb{P}_{h}(x\mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}( x\mid v^{\prime})}\stackrel{{(b)}}{{=}}\frac{\mathbb{P}_{h}(\phi^{*}(x)) \mid v)}{\sum_{v^{\prime}\in\Upsilon_{h}}\mathbb{P}_{h}(\phi^{*}(x))\mid v^{ \prime})},\]

\[\mathbb{P}_{h}(x\mid v)=q\left(x\mid\phi^{*}(x),\phi^{*}_{\xi}(x)\right) \mathbb{P}_{h}(\phi^{*}(x)\mid v)\mathbb{P}_{h}(\phi^{*}_{\xi}(x)).\] Let \(v_{1},v_{2}\in\Upsilon_{h}\) be two paths with indices \(i\) and \(j\) respectively. We define their exact path prediction gap as \(\Delta(i,j):=\mathbb{E}_{x_{h}}\left[\left|f^{*}_{h}(i\mid x_{h})-f^{*}_{h}(j \mid x_{h})\right|\right]\). Assume that \(v_{1}\) visits an endogenous state \(s\) at time step \(h\) and denote \(\omega(s)\) as the number of paths in \(\Upsilon_{h}\) that reaches \(s\). Then \(f^{*}_{h}(i\mid x_{h})=\nicefrac{{1}}{{\omega(s)}}\) if \(\phi^{*}(x_{h})=s\), and \(0\) otherwise. If \(v_{2}\) also visits \(s\) at time step \(h\), then \(f^{*}_{h}(i\mid x_{h})=f^{*}_{h}(j\mid x_{h})\) all \(x_{h}\). This implies \(\Delta(i,j)=0\) and \(\mathtt{PPE}\) will filter out the path with higher index since it detected both paths reach to the same endogenous state. Conversely, let \(v_{2}\) visit a different state at time step \(h\). If \(x\) is an observation that maps to \(s\), then \(f^{*}_{h}(i\mid x)=\nicefrac{{1}}{{\omega(s)}}\) and \(f^{*}_{h}(j\mid x)=0\). This gives \(\left|f^{*}_{h}(i\mid x)-f^{*}_{h}(j\mid x)\right|=\nicefrac{{1}}{{\omega(s)} }\geq\nicefrac{{1}}{{\left|\Upsilon_{h}\right|}}\) and, consequently, \(\Delta(i,j)>0\). In fact, we can show \(\Delta(i,j)\geq\mathcal{O}(\nicefrac{{1}}{{1}}\nicefrac{{1}}{{\left|\Upsilon_ {h}\right|}})\). Thus, \(\mathtt{PPE}\) will not eliminate these paths upon comparison.

The combination lock problem is defined for a given horizon \(H\) by an endogenous state space \(\mathcal{S}=\{s_{1,a}\}\cup\{s_{h,a},s_{h,b},s_{h,c}\}_{h=2}^{H}\), an exogenous state space \(\Xi=\{0,1\}^{H}\), an action space \(\mathcal{A}\) with 10 actions, and a deterministic endogenous start state of \(s_{1,a}\). For any state \(s_{h,g}\) we call \(g\) as its _type_ which can be \(a,b\) or \(c\). States with type \(a\) and \(b\) are considered _good_ states and those with type \(c\) are considered _bad_ states. Each instance of this problem is defined by two good action sequences \((a_{h})^{H}_{h=2},(a^{\prime}_{h})^{H}_{h=2}\) with \(a_{h}\neq a^{\prime}_{h}\), which are chosen uniformly randomly and kept fixed throughout. At \(h=1\), the agent is in \(s_{1,a}\) and action \(a_{1}\) leads to \(s_{2,a}\), \(a^{\prime}_{h}\) leads to \(s_{2,b}\), and all other actions lead to \(s_{2,c}\). For \(h>2\), taking action \(a_{h}\) in \(s_{h,a}\) leads to \(s_{h+1,a}\) and taking action \(a^{\prime}_{h}\) in \(s_{h,b}\) leads to \(s_{h+1,b}\). In all other cases involving taking an action in a state \(s_{h,g}\), we transition to the next bad state \(s_{h+1,c}\).

The exogenous state evolves as follows. We set \(\xi_{1}\in\{0,1\}^{H}\) where \(\xi_{1}(i)\sim\mathtt{Unf}\left(\{0,1\}\right)\) for each \(i\in[H]\). At time step \(h\), \(\xi_{h}\) is generated from \(\xi_{h-1}\) by uniformly flipping each bit in \(\xi_{h-1}\) independently with probability 0.1.

An observation \(x\) is generated stochastically from a latent state \(z=(s,\xi)\). We map \(s\) to a vector \(w\) encoding the identity of the state. We concatenate \((w,\xi)\), add Gaussian noise to each dimension, and multiply the result by a Hadamard matrix to generate \(x\). See Appendix F for full details. Our construction is inspired by prior work (Du et al., 2019; Misra et al., 2020).

Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.

When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.

On adding exogenous noise, the accuracy of Homer plummets significantly. The accuracy of ID also drops but this drop is mild since unlike Homer, the ID objective is able to filter exogenous noise.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2

We add exogenous noise as follows: at the beginning of each episode, we independently sample position, size and color of 5 ellipses.

The exogenous state is given by the position, size and color of ellipses and is much larger than \(|\mathcal{S}|\leq 4N^{2}\).

Since the problem has deterministic dynamics, we evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm's mapping from observations to endogenous latent states.

We also evaluate the accuracy of state decoding on this problem. We compare the state decoding accuracy of PPE and Homer at \(H=2\) using an identical evaluation setup to the one we used for combination lock. Figure 2(d) shows the results. As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder. In this work, we introduce the EX-BMDP setting, an RL setting that models exogenous noise, ubiquitous in many real-world systems.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments. To our knowledge, this is the first such algorithm with guarantees for settings with exogenous noise.

Input: Figure 1: Left: An agent is walking next to a pond in a park and observes the world as an image. The world consists of a latent endogenous state, containing variable such as agent’s position, and a much larger latent exogenous state containing variables such as motion of ducks, ripples in the water, etc. Center: Graphical model of the EX-BMDP. Right: PPE learns a generalized form of inverse dynamics that recovers the endogenous state.

Input: Figure 2: Results on combination lock. Left: We show the latent transition dynamics of combination lock. Observations are not shown for brevity. Center: Shows minimal number of episodes needed to achieve a mean regret of at most V (π?)/2. Right: State decoding accuracy (in percent) of decoders learned by different methods. Solid lines implies no exogenous dimension while dashed lines imply an exogenous dimension of 100.

Input: Figure 3: Results on visual grid world. Left two: Shows sampled observations for the first two steps from the visual gridworld domain. The agent is depicted as a red-triangle, lava in orange, walls in grey, and the goal in green. Center Right: Shows errors of type 1 and type 2 made by the PPE in recovering the latent endogenous dynamics. Right: State decoding accuracy of PPE, Homer and a random uniform decoder. (see Section 6)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim, evidence, and reviewer comments, I'll assess whether the question "How can we learn using an amount of data scaling with just the size of the endogenous latent state, while ignoring the temporally correlated exogenous observation noise?" is well-stated or overstated.

Summary: The evidence strongly supports this being a well-stated claim. The paper:
- Formally defines the EX-BMDP setting that separates endogenous and exogenous states
- Provides theoretical guarantees showing sample complexity depends on endogenous state size
- Demonstrates empirically through combination lock and visual grid world experiments that PPE can learn accurate state decoders while filtering exogenous noise
- Acknowledges limitations around near-deterministic dynamics assumption
- Reviewer concerns were adequately addressed per post-discussion comments

The claim accurately reflects what is demonstrated without exaggerating capabilities or glossing over constraints.

<score>1</score>