Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using \(O((SA)^{2}H\log(

The evidence to be evaluated is:In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1). Before describing PPE, we highlight that PPE can be thought of as a computationally-efficient and simpler alternative to Algorithm 4 of Du et al. (2019) who studied rich-observation setting without exogenous noise.1 Footnote 1: Alg. 4 has time complexity of \(\mathcal{O}(S^{4}A^{4}H)\) compared to \(\mathcal{O}(S^{3}A^{3}H)\) for PPE. Furthermore, Alg. 4 requires an upper bound on \(S\), whereas PPE is adaptive to it. Lastly, Du et al. (2019) assumed deterministic setting while we provide a generalization to near-determinism.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model. PPE works by eliminating paths in \(\Upsilon_{h}\) so that we are left with just a single path for each reachable state. This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated. PPE reduces RL to performing \(H\) standard classification problems.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

PPE runs in a reward-free setting. However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

We analyze the algorithm in near-deterministic MDPs defined as follows: Two transition functions \(T_{1}\) and \(T_{2}\) are \(\eta\)_-close_ if for all \(h\in[H],a\in\mathcal{A},s\in\mathcal{S}_{h}\) it holds that \(||T_{1}(\cdot\mid s,a)-T_{2}(\cdot\mid s,a)||_{1}\leq\eta\).

We emphasize that near-deterministic dynamics are common in real-world applications like robotics.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\).

In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

_Fix \(\delta\in(0,1)\). Then, with probability greater than \(1-\delta\), PPE returns a policy cover \(\{\Psi_{h}\}_{h=1}^{H}\) such that for any \(h\in[H]\), \(\Psi_{h}\) is a \(\eta H\)-policy cover for time step \(h\) and \(|\Psi_{h}|\leq S\), which gives the total number of episodes used by PPE as \(\mathcal{O}\left(S^{2}A^{2}H\ln\frac{|\mathcal{F}|SAH}{\delta}\right)\).

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines.

Lastly, we observe that PPE is always able to learn a good decoder and is more sample efficient than baselines.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step. This is important since even a single type 1 error can cause exploration failures. Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Therefore, at \(2\times 10^{5}\) samples, the algorithm makes 0 type 1 errors and just a handful type 2 errors. This is remarkable considering that PPE compares roughly \(2\times 10^{5}\) pairs of paths in the entire run. Hence, it makes only \(\leq 0.03\%\) type 2 errors. Further, the agent is able to plan using the learned transition model and receive the optimal return.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim: "For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using O((SA)²H log(...)"

Analysis:
The claim appears well-stated based on the following evidence:

1. The paper provides formal theoretical guarantees, stating "with probability greater than 1-δ, PPE returns a policy cover {...} which gives the total number of episodes used by PPE as O(S²A²H ln(|F|SAH/δ))"

2. The assumptions and limitations are clearly stated:
- Near-deterministic dynamics requirement (η ≤ 1/4SH)
- Realizability assumption for the regression problem
- Clear experimental validation showing PPE's effectiveness

3. The complexity bound is directly supported by the theoretical analysis and matches the claim.

The claim directly reflects the mathematical guarantees provided in the paper without exaggeration or overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using \(O((SA)^{2}H\log(

The review comment to be evaluated is:

The paper is very well motivated. It has convinced me that being invariant to exogenous noise is important and that the EX-BMDP is useful abstraction for making this problem concrete. It is also very clearly presented. The argumentation is clear, and the flow of the paper is quite natural. Even the lengthy appendix is fairly easy to parse, with straightforward proofs and relevant details (e.g. specific claims about the unsuitability of alternative approaches).

One shortcoming is potential impact / applicability. I'm not convinced that environments with single starting states and near-deterministic dynamics is a very rich problem class. Or more specifically, I'm not convinced that restricting myself to this class would be preferable to working in the full space of MDPs with e.g. the possibility of aliasing a few states with a single-step inverse-dynamics approach. I am aware that impact is very hard to predict in advance, so I won't let this aspect unduly affect my scoring. And I'm convinced that even if PPE is never used in practice, this paper should still be accepted for its useful problem formulation and theoretical results. But a quick read would leave one with the impression that the alternatives (e.g. contrastive learning, inverse dynamics, bisimulation metrics) flaws outweigh their considerable advantages (e.g. applicability with stochastic dynamics) --  explicitly noting where alternative approaches are preferable would be appreciated.

I also noticed what I believe to be an omission in your related works: mutual information / empowerment based methods. "Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning" seems particularly related in its search for predicable action sequences. And the follow-up work "Variational Intrinsic Control" appears to be a multi-step inverse dynamics analog that doesn't require open-loop policies. If you can show that these approaches are flawed in a way that PPE is not, then that would considerably raise my assessment of this work. And even if not, I believe this field (if not these specific works) should be acknowledged, as they similarly attempt to learn what in the environment is controllable.

While I agree that PPE is applicable in no-reward situations whereas bi-simulation metrics are not, your experiments all involve rewards, so I'm surprised a bi-simulation metric method was not included as a baseline.

While this isn't necessary for acceptance, it is worth noting that prior work has established much more challenging benchmarks for evaluating the representation of exogenous noise, and utilizing a pre-existing benchmark would make your empirical results considerably more impressive.

This is a small point, but it is not initially obvious why inverse dynamics fails on the combination lock problem. Does alliasing occur just when e.g. s_2a --> s_3a and s_2b-->s_3b have the same action?

The evidence to be evaluated is:In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1). Before describing PPE, we highlight that PPE can be thought of as a computationally-efficient and simpler alternative to Algorithm 4 of Du et al. (2019) who studied rich-observation setting without exogenous noise.1 Footnote 1: Alg. 4 has time complexity of \(\mathcal{O}(S^{4}A^{4}H)\) compared to \(\mathcal{O}(S^{3}A^{3}H)\) for PPE. Furthermore, Alg. 4 requires an upper bound on \(S\), whereas PPE is adaptive to it. Lastly, Du et al. (2019) assumed deterministic setting while we provide a generalization to near-determinism.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model. PPE works by eliminating paths in \(\Upsilon_{h}\) so that we are left with just a single path for each reachable state. This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated. PPE reduces RL to performing \(H\) standard classification problems.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

PPE runs in a reward-free setting. However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

We analyze the algorithm in near-deterministic MDPs defined as follows: Two transition functions \(T_{1}\) and \(T_{2}\) are \(\eta\)_-close_ if for all \(h\in[H],a\in\mathcal{A},s\in\mathcal{S}_{h}\) it holds that \(||T_{1}(\cdot\mid s,a)-T_{2}(\cdot\mid s,a)||_{1}\leq\eta\).

We emphasize that near-deterministic dynamics are common in real-world applications like robotics.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\).

In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

_Fix \(\delta\in(0,1)\). Then, with probability greater than \(1-\delta\), PPE returns a policy cover \(\{\Psi_{h}\}_{h=1}^{H}\) such that for any \(h\in[H]\), \(\Psi_{h}\) is a \(\eta H\)-policy cover for time step \(h\) and \(|\Psi_{h}|\leq S\), which gives the total number of episodes used by PPE as \(\mathcal{O}\left(S^{2}A^{2}H\ln\frac{|\mathcal{F}|SAH}{\delta}\right)\).

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines.

Lastly, we observe that PPE is always able to learn a good decoder and is more sample efficient than baselines.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step. This is important since even a single type 1 error can cause exploration failures. Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Therefore, at \(2\times 10^{5}\) samples, the algorithm makes 0 type 1 errors and just a handful type 2 errors. This is remarkable considering that PPE compares roughly \(2\times 10^{5}\) pairs of paths in the entire run. Hence, it makes only \(\leq 0.03\%\) type 2 errors. Further, the agent is able to plan using the learned transition model and receive the optimal return.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using O((SA)²H log(...)"

The evidence shows:
1. The paper proves a sample complexity bound of O(S²A²H ln(|F|SAH/δ)), which matches the claim's order of magnitude.
2. The assumptions are clearly stated (near-deterministic dynamics with η≤1/(4SH), realizability assumption).
3. The empirical results support the theoretical claims, showing successful exploration with reasonable sample complexity.
4. The review comment, while noting some limitations about applicability, doesn't dispute the theoretical results.

The claim appears well-stated as it:
- Accurately represents the proven complexity bound
- Clearly specifies the setting (near-deterministic EX-BMDPs)
- Makes no exaggerated assertions beyond what's proven
- Is supported by both theoretical proofs and experimental validation

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using \(O((SA)^{2}H\log(

The review comment to be evaluated is:

Strengths:

- The problem setting is important and seems to be relevant to many real-world problems.
- While the idea of using inverse dynamics and even multi-step inverse dynamics [1] to learn representations in these types of environments to learn controllable representations is not novel, this paper provides a theoretical basis for this choice, which I think is valuable.
- I found sections 1 through 5 to be very well written.

Weaknesses:

- In the experiment section, it seems as though you are testing the planning/policy learning abilities with PPE in both of the environments. However, as far as I can tell, there is no indication of how you perform this planning in PPE in the main text. The paper simply switches from talking about learning policy covers to showing regret for environments with only a small mention of the fact that you could plan using the data collected with PPE. The way the experiments are run and the choice to test the planning performance / how the state representations are obtained in experiments should be explained in more detail in my opinion.
- The paper discusses prior works with inverse dynamics seemingly only in the context of their applications in representation learning in two prior works on exploration. I think the application of multi-step inverse dynamics in goal-conditioned environments described in [1] is worth mentioning, since for a fixed goal this algorithm seems like it is doing something similar to PPL. Related is the lack of a "related works" section in the paper, which would be helpful for framing PPL in the context of prior works.
- There is no conclusion section, which I feel would improve the paper.

My major questions are:

1. How is the policy learning done in the experiment section? How are representations found in the decoding accuracy plot?
2. How does this paper relate to prior works on inverse dynamics?

[1] Paster, Keiran, et al. Planning from Pixels Using Inverse Dynamics Models. 2020. openreview.net, https://openreview.net/forum?id=V6BjBgku7Ro.

The evidence to be evaluated is:In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1). Before describing PPE, we highlight that PPE can be thought of as a computationally-efficient and simpler alternative to Algorithm 4 of Du et al. (2019) who studied rich-observation setting without exogenous noise.1 Footnote 1: Alg. 4 has time complexity of \(\mathcal{O}(S^{4}A^{4}H)\) compared to \(\mathcal{O}(S^{3}A^{3}H)\) for PPE. Furthermore, Alg. 4 requires an upper bound on \(S\), whereas PPE is adaptive to it. Lastly, Du et al. (2019) assumed deterministic setting while we provide a generalization to near-determinism.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model. PPE works by eliminating paths in \(\Upsilon_{h}\) so that we are left with just a single path for each reachable state. This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated. PPE reduces RL to performing \(H\) standard classification problems.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

PPE runs in a reward-free setting. However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

We analyze the algorithm in near-deterministic MDPs defined as follows: Two transition functions \(T_{1}\) and \(T_{2}\) are \(\eta\)_-close_ if for all \(h\in[H],a\in\mathcal{A},s\in\mathcal{S}_{h}\) it holds that \(||T_{1}(\cdot\mid s,a)-T_{2}(\cdot\mid s,a)||_{1}\leq\eta\).

We emphasize that near-deterministic dynamics are common in real-world applications like robotics.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\).

In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

_Fix \(\delta\in(0,1)\). Then, with probability greater than \(1-\delta\), PPE returns a policy cover \(\{\Psi_{h}\}_{h=1}^{H}\) such that for any \(h\in[H]\), \(\Psi_{h}\) is a \(\eta H\)-policy cover for time step \(h\) and \(|\Psi_{h}|\leq S\), which gives the total number of episodes used by PPE as \(\mathcal{O}\left(S^{2}A^{2}H\ln\frac{|\mathcal{F}|SAH}{\delta}\right)\).

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines.

Lastly, we observe that PPE is always able to learn a good decoder and is more sample efficient than baselines.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step. This is important since even a single type 1 error can cause exploration failures. Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Therefore, at \(2\times 10^{5}\) samples, the algorithm makes 0 type 1 errors and just a handful type 2 errors. This is remarkable considering that PPE compares roughly \(2\times 10^{5}\) pairs of paths in the entire run. Hence, it makes only \(\leq 0.03\%\) type 2 errors. Further, the agent is able to plan using the learned transition model and receive the optimal return.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using O((SA)²H log("

1. Summary of evaluation:
The claim appears to be well-stated based on the evidence provided. The paper presents detailed theoretical analysis showing that PPE returns a policy cover with probability >1-δ using O(S²A²H ln(|F|SAH/δ)) episodes. The algorithm's complexity is thoroughly justified through mathematical proofs, assumptions are clearly stated (like η-closeness to deterministic model), and limitations are acknowledged. The experimental results support the theoretical claims by demonstrating successful exploration in practice. The reviewer comments don't raise concerns about this specific complexity claim.

<score>1</score>

Justification:
1. The claim is directly supported by theoretical proof stating "total number of episodes used by PPE as O(S²A²H ln(|F|SAH/δ))"
2. Clear prerequisites are established (near-deterministic dynamics, realizability assumption)
3. Experimental validation shows PPE achieves intended exploration with stated complexity
4. No evidence of exaggeration - claim matches exactly what is proven
5. Reviewer comments focus on other aspects and don't dispute this complexity result


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using \(O((SA)^{2}H\log(

The review comment to be evaluated is:

STRENGTHS

(S1) The problem of efficiently learning in high-dimensional observation spaces is a good one, and the authors provided an excellent discussion of the technical components of this problem in the early parts of the paper. I thought the definition and presentation of the EX-BMDP was particularly good.

(S2) I appreciate the authors providing experimental results to show that PPE performs well in some regards.

WEAKNESSES

(W1) I'm afraid I found the discussion of PPE to be a bit difficult to decode. Even looking at Algorithm 1, I find myself with a number of important questions unanswered, chief among them being "what about $\phi^*_e$"? It strikes me as odd that nowhere in this algorithm are the inverse mappings from observation to exogenous/endogenous state represented or used. Why is this?

(W2) I'm a little concerned with the computational tractability of the algorithm. My understanding from the paper is that one seeks some sort of "covering" of the state space--in what sense is it practical to obtain and/or store this covering? If the problem setting consists of high-dimensional observations this seems especially challenging.

(W3) From my naive perspective, it seems that the ultimate goal here is to use PPE and then actually perform reinforcement learning. However, I didn't understand from the brief discussion at the end of Section 4 (or, frankly, from reading the Appendix) how exactly that would be accomplished. I feel this is important enough that it should appear in the main paper.

POST-DISCUSSION COMMENTS

During the discussion, I feel as though the authors adequately addressed each of the issues I raised above, and so I'm happy to raise my score to accept.

The evidence to be evaluated is:In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1). Before describing PPE, we highlight that PPE can be thought of as a computationally-efficient and simpler alternative to Algorithm 4 of Du et al. (2019) who studied rich-observation setting without exogenous noise.1 Footnote 1: Alg. 4 has time complexity of \(\mathcal{O}(S^{4}A^{4}H)\) compared to \(\mathcal{O}(S^{3}A^{3}H)\) for PPE. Furthermore, Alg. 4 requires an upper bound on \(S\), whereas PPE is adaptive to it. Lastly, Du et al. (2019) assumed deterministic setting while we provide a generalization to near-determinism.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model. PPE works by eliminating paths in \(\Upsilon_{h}\) so that we are left with just a single path for each reachable state. This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated. PPE reduces RL to performing \(H\) standard classification problems.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

PPE runs in a reward-free setting. However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

We analyze the algorithm in near-deterministic MDPs defined as follows: Two transition functions \(T_{1}\) and \(T_{2}\) are \(\eta\)_-close_ if for all \(h\in[H],a\in\mathcal{A},s\in\mathcal{S}_{h}\) it holds that \(||T_{1}(\cdot\mid s,a)-T_{2}(\cdot\mid s,a)||_{1}\leq\eta\).

We emphasize that near-deterministic dynamics are common in real-world applications like robotics.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\).

In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

_Fix \(\delta\in(0,1)\). Then, with probability greater than \(1-\delta\), PPE returns a policy cover \(\{\Psi_{h}\}_{h=1}^{H}\) such that for any \(h\in[H]\), \(\Psi_{h}\) is a \(\eta H\)-policy cover for time step \(h\) and \(|\Psi_{h}|\leq S\), which gives the total number of episodes used by PPE as \(\mathcal{O}\left(S^{2}A^{2}H\ln\frac{|\mathcal{F}|SAH}{\delta}\right)\).

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines.

Lastly, we observe that PPE is always able to learn a good decoder and is more sample efficient than baselines.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step. This is important since even a single type 1 error can cause exploration failures. Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Therefore, at \(2\times 10^{5}\) samples, the algorithm makes 0 type 1 errors and just a handful type 2 errors. This is remarkable considering that PPE compares roughly \(2\times 10^{5}\) pairs of paths in the entire run. Hence, it makes only \(\leq 0.03\%\) type 2 errors. Further, the agent is able to plan using the learned transition model and receive the optimal return.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using O((SA)²H log("

Based on the evidence:

1. The paper provides theoretical analysis showing PPE's sample complexity is O(S²A²H ln(|F|SAH/δ)), which aligns with the claim's O((SA)²H) part.
2. The empirical results demonstrate PPE's effectiveness in a grid-world environment with very low error rates (0 type 1 errors, ~0.03% type 2 errors).
3. The assumptions are clearly stated (near-deterministic dynamics, realizability).
4. The reviewer comments don't directly challenge the complexity claim.

The claim appears well-stated as it directly reflects the mathematical analysis provided and is supported by both theoretical proofs and experimental validation.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using \(O((SA)^{2}H\log(

The review comment to be evaluated is:

Strengths:

- EX-BMDPs seem to capture several popular simulated distractors that are currently being used in applied RL papers, such as random video backgrounds in MuJoCo tasks. I suspect the fact that the exogenous dynamics are relatively unconstrained will also make them a good model for practical tasks. The requirement for near-determinism in the endogenous dynamics is unfortunate, particularly given that it applies to the initial state distribution as well (often you want to perform well over a distribution of similar-but-not-identical tasks). However, I think this is not a major concern for exploration methods: so long as the possible initial states are reachable from some common starting state within $H$ steps, PPE will eventually explore them.
- The fact that it is possible to obtain a sample complexity bound completely independent of the complexity of $\Xi$ was surprising to me given the relatively complex form of exogenous noise in an EX-BMDP, and shows that the EX-BMDP class is at least tractable.
- The clarity of writing was appreciated, particularly when stating assumptions and comparing against past algorithms.

Weaknesses:

- I noticed that RND is missing from Section 3/Appendix A (shortcomings of existing methods), but does surprisingly well on the combination lock problem. Is it possible to give some intuition for where RND fails in general?
- The visual gridworld experiments are a little confusing. The takeaway seems to be that PPE drives accuracy to \~100% eventually on a less-toy problem. However, there are no baselines, so it is hard to determine how significant this achievement is. It would be useful to see how the baselines from Figure 2 fare in this setting, particularly in terms of sample efficiency.

The evidence to be evaluated is:In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1). Before describing PPE, we highlight that PPE can be thought of as a computationally-efficient and simpler alternative to Algorithm 4 of Du et al. (2019) who studied rich-observation setting without exogenous noise.1 Footnote 1: Alg. 4 has time complexity of \(\mathcal{O}(S^{4}A^{4}H)\) compared to \(\mathcal{O}(S^{3}A^{3}H)\) for PPE. Furthermore, Alg. 4 requires an upper bound on \(S\), whereas PPE is adaptive to it. Lastly, Du et al. (2019) assumed deterministic setting while we provide a generalization to near-determinism.

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model. PPE works by eliminating paths in \(\Upsilon_{h}\) so that we are left with just a single path for each reachable state. This is done by collecting a dataset \(\mathcal{D}\) of tuples \((x,v)\) where \(v\) is a uniformly sampled from \(\Upsilon_{h}\) and \(x\sim\mathbb{P}_{h}(x\mid v)\) (line 4). We train a classifier \(\hat{f}_{h}\) using \(\mathcal{D}\) by predicting the index \(\mathrm{idx}(v)\) of the path \(v\) from the observation \(x\) (line 5).

For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated. PPE reduces RL to performing \(H\) standard classification problems.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

PPE runs in a reward-free setting. However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

We analyze the algorithm in near-deterministic MDPs defined as follows: Two transition functions \(T_{1}\) and \(T_{2}\) are \(\eta\)_-close_ if for all \(h\in[H],a\in\mathcal{A},s\in\mathcal{S}_{h}\) it holds that \(||T_{1}(\cdot\mid s,a)-T_{2}(\cdot\mid s,a)||_{1}\leq\eta\).

We emphasize that near-deterministic dynamics are common in real-world applications like robotics.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\).

In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

_Fix \(\delta\in(0,1)\). Then, with probability greater than \(1-\delta\), PPE returns a policy cover \(\{\Psi_{h}\}_{h=1}^{H}\) such that for any \(h\in[H]\), \(\Psi_{h}\) is a \(\eta H\)-policy cover for time step \(h\) and \(|\Psi_{h}|\leq S\), which gives the total number of episodes used by PPE as \(\mathcal{O}\left(S^{2}A^{2}H\ln\frac{|\mathcal{F}|SAH}{\delta}\right)\).

Our sample complexity guarantees do not depend directly on the size of observation space or the exogenous space.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines.

Lastly, we observe that PPE is always able to learn a good decoder and is more sample efficient than baselines.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step. This is important since even a single type 1 error can cause exploration failures. Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Therefore, at \(2\times 10^{5}\) samples, the algorithm makes 0 type 1 errors and just a handful type 2 errors. This is remarkable considering that PPE compares roughly \(2\times 10^{5}\) pairs of paths in the entire run. Hence, it makes only \(\leq 0.03\%\) type 2 errors. Further, the agent is able to plan using the learned transition model and receive the optimal return.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "For near-deterministic EX-BMDPs, we prove that PPE successfully explores the environment using O((SA)²H log(...)"

1. Summary of evaluation:
The claim appears well-stated based on the theoretical analysis and evidence provided. The paper:
- Clearly states assumptions (near-deterministic dynamics, realizability)
- Provides formal proof showing sample complexity of O(S²A²H ln(|F|SAH/δ))
- Demonstrates empirical validation in visual gridworld experiments
- Acknowledges limitations (near-deterministic requirement)
- The reviewer also confirms the theoretical contribution's significance, particularly noting the independence from exogenous state complexity

The claim directly reflects the mathematical guarantees without overstating the results or ignoring limitations.

2. <score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We further, show that PPE is able to recover the latent endogenous model in a visually complex navigation problem, in accordance with the theory.

The evidence to be evaluated is:The structure of EX-BMDPs allows to reduce the size of the policy cover significantly to \(|\mathcal{S}|\ll|\mathcal{Z}|=|\mathcal{S}|\,|\Xi|\) when the size of the exogenous state space is large. Specifically, we show that the set of policies that reach each _endogenous_ state, and _do not depend on the exogenous_ part of the state is also a policy cover (see Appendix B, Proposition 4).

In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1).

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.

Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action. For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated.

Recovering an endogenous state decoder. We can recover a endogenous state decoder \(\hat{\phi}_{h}\) for each time step \(h\in\{2,3,\cdots,H\}\) directly from \(\hat{f}_{h}\) as shown below: \[\hat{\phi}_{h}(x)=\min\left\{i\mid\hat{f}_{h}(i\mid x)\geq\max_{j}\hat{f}_{h}(j \mid x)-\mathcal{O}(\nicefrac{{1}}{{\Gamma_{h}}}),i\in[\left|\Upsilon_{h} \right|]\right\}.\] Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting \(x\), and therefore, \(\phi^{\star}(x)\). We are implicitly using the decoder for exploring, since we rely on using \(\hat{f}_{h}\) for making planning decisions.

PPE can also be used to recover a latent endogenous transition dynamics.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

If the reward function only depends on the endogenous state, we can use a computationally cheaper value-iteration VI that uses the recovered transition dynamics. VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details). In each case, the sample complexity of learning a near-optimal policy, given the output of PPE, scales with the size of endogenous and not the exogenous state space. We provide the main sample complexity guarantee for PPE as well as additional intuition for why it works.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\). _Realizability assumptions are common in theoretical analysis (e.g., Misra et al. (2020), Agarwal et al. (2020)). In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

Let \(v_{1},v_{2}\in\Upsilon_{h}\) be two paths with indices \(i\) and \(j\) respectively. We define their exact path prediction gap as \(\Delta(i,j):=\mathbb{E}_{x_{h}}\left[\left|f^{*}_{h}(i\mid x_{h})-f^{*}_{h}(j \mid x_{h})\right|\right]\). Assume that \(v_{1}\) visits an endogenous state \(s\) at time step \(h\) and denote \(\omega(s)\) as the number of paths in \(\Upsilon_{h}\) that reaches \(s\). Then \(f^{*}_{h}(i\mid x_{h})=\nicefrac{{1}}{{\omega(s)}}\) if \(\phi^{*}(x_{h})=s\), and \(0\) otherwise. If \(v_{2}\) also visits \(s\) at time step \(h\), then \(f^{*}_{h}(i\mid x_{h})=f^{*}_{h}(j\mid x_{h})\) all \(x_{h}\). This implies \(\Delta(i,j)=0\) and \(\mathtt{PPE}\) will filter out the path with higher index since it detected both paths reach to the same endogenous state. Conversely, let \(v_{2}\) visit a different state at time step \(h\). If \(x\) is an observation that maps to \(s\), then \(f^{*}_{h}(i\mid x)=\nicefrac{{1}}{{\omega(s)}}\) and \(f^{*}_{h}(j\mid x)=0\). This gives \(\left|f^{*}_{h}(i\mid x)-f^{*}_{h}(j\mid x)\right|=\nicefrac{{1}}{{\omega(s)} }\geq\nicefrac{{1}}{{\left|\Upsilon_{h}\right|}}\) and, consequently, \(\Delta(i,j)>0\). In fact, we can show \(\Delta(i,j)\geq\mathcal{O}(\nicefrac{{1}}{{1}}\nicefrac{{1}}{{\left|\Upsilon_ {h}\right|}})\). Thus, \(\mathtt{PPE}\) will not eliminate these paths upon comparison. Our complete analysis in the Appendix generalizes the above reasoning to finite sample setting where we can only approximate \(f^{*}_{h}\) and \(\Delta\), as well as to EX-BMDPs with near-deterministic dynamics.

We evaluate \(\mathtt{PPE}\) on two domains: a challenging exploration problem called _combination lock_ to test whether \(\mathtt{PPE}\) can learn an optimal policy and an accurate state decoder, and a visual-grid world with complex visual representations to test whether \(\mathtt{PPE}\) is able to recover the latent dynamics.

We visualize the latent endogenous dynamics in Figure 1(a).

An observation \(x\) is generated stochastically from a latent state \(z=(s,\xi)\). We map \(s\) to a vector \(w\) encoding the identity of the state. We concatenate \((w,\xi)\), add Gaussian noise to each dimension, and multiply the result by a Hadamard matrix to generate \(x\). See Appendix F for full details.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines. Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.

When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

The exogenous state is given by the position, size and color of ellipses and is much larger than \(|\mathcal{S}|\leq 4N^{2}\).

Since the problem has deterministic dynamics, we evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm's mapping from observations to endogenous latent states.

Type 1 error computes whether PPE merged these paths, i.e., predicted them as mapping to the same abstract state, when they go to _different endogenous_ states.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step.

Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Further, the agent is able to plan using the learned transition model and receive the optimal return.

As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Our work also raises interesting future questions such as how to address the general setting with stochastic transitions, or handle more complex endogenous state representations.

Input: Figure 1: Left: An agent is walking next to a pond in a park and observes the world as an image. The world consists of a latent endogenous state, containing variable such as agent’s position, and a much larger latent exogenous state containing variables such as motion of ducks, ripples in the water, etc. Center: Graphical model of the EX-BMDP. Right: PPE learns a generalized form of inverse dynamics that recovers the endogenous state.

Input: Figure 2: Results on combination lock. Left: We show the latent transition dynamics of combination lock. Observations are not shown for brevity. Center: Shows minimal number of episodes needed to achieve a mean regret of at most V (π?)/2. Right: State decoding accuracy (in percent) of decoders learned by different methods. Solid lines implies no exogenous dimension while dashed lines imply an exogenous dimension of 100.

Input: Figure 3: Results on visual grid world. Left two: Shows sampled observations for the first two steps from the visual gridworld domain. The agent is depicted as a red-triangle, lava in orange, walls in grey, and the goal in green. Center Right: Shows errors of type 1 and type 2 made by the PPE in recovering the latent endogenous dynamics. Right: State decoding accuracy of PPE, Homer and a random uniform decoder. (see Section 6)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me assess the claim: "We further, show that PPE is able to recover the latent endogenous model in a visually complex navigation problem, in accordance with the theory."

Based on the evidence provided, including the experimental results in the visual grid world domain, this claim appears well-stated. The evidence shows:

1. PPE successfully reduced Type 1 errors to 0 and Type 2 errors to a low number with sufficient episodes
2. The visual grid world experiments demonstrate recovery of latent endogenous dynamics in a complex visual environment
3. The results align with theoretical predictions
4. Quantitative results are provided showing the model's performance in state decoding accuracy
5. The limitations and assumptions are clearly stated

The claim matches the evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We further, show that PPE is able to recover the latent endogenous model in a visually complex navigation problem, in accordance with the theory.

The review comment to be evaluated is:

The paper is very well motivated. It has convinced me that being invariant to exogenous noise is important and that the EX-BMDP is useful abstraction for making this problem concrete. It is also very clearly presented. The argumentation is clear, and the flow of the paper is quite natural. Even the lengthy appendix is fairly easy to parse, with straightforward proofs and relevant details (e.g. specific claims about the unsuitability of alternative approaches).

One shortcoming is potential impact / applicability. I'm not convinced that environments with single starting states and near-deterministic dynamics is a very rich problem class. Or more specifically, I'm not convinced that restricting myself to this class would be preferable to working in the full space of MDPs with e.g. the possibility of aliasing a few states with a single-step inverse-dynamics approach. I am aware that impact is very hard to predict in advance, so I won't let this aspect unduly affect my scoring. And I'm convinced that even if PPE is never used in practice, this paper should still be accepted for its useful problem formulation and theoretical results. But a quick read would leave one with the impression that the alternatives (e.g. contrastive learning, inverse dynamics, bisimulation metrics) flaws outweigh their considerable advantages (e.g. applicability with stochastic dynamics) --  explicitly noting where alternative approaches are preferable would be appreciated.

I also noticed what I believe to be an omission in your related works: mutual information / empowerment based methods. "Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning" seems particularly related in its search for predicable action sequences. And the follow-up work "Variational Intrinsic Control" appears to be a multi-step inverse dynamics analog that doesn't require open-loop policies. If you can show that these approaches are flawed in a way that PPE is not, then that would considerably raise my assessment of this work. And even if not, I believe this field (if not these specific works) should be acknowledged, as they similarly attempt to learn what in the environment is controllable.

While I agree that PPE is applicable in no-reward situations whereas bi-simulation metrics are not, your experiments all involve rewards, so I'm surprised a bi-simulation metric method was not included as a baseline.

While this isn't necessary for acceptance, it is worth noting that prior work has established much more challenging benchmarks for evaluating the representation of exogenous noise, and utilizing a pre-existing benchmark would make your empirical results considerably more impressive.

This is a small point, but it is not initially obvious why inverse dynamics fails on the combination lock problem. Does alliasing occur just when e.g. s_2a --> s_3a and s_2b-->s_3b have the same action?

The evidence to be evaluated is:The structure of EX-BMDPs allows to reduce the size of the policy cover significantly to \(|\mathcal{S}|\ll|\mathcal{Z}|=|\mathcal{S}|\,|\Xi|\) when the size of the exogenous state space is large. Specifically, we show that the set of policies that reach each _endogenous_ state, and _do not depend on the exogenous_ part of the state is also a policy cover (see Appendix B, Proposition 4).

In this section, we present an algorithm _Predictive Path Elimination_ (PPE) that we later show can provably solve any EX-BMDP with nearly deterministic dynamics and start state distribution of the endogenous state, while making no assumptions on the dynamics or start state distribution of the exogenous state (Algorithm 1).

```**Algorithm 1** PPE\((\delta,\eta)\): Predictive Path EliminationPPE performs iterations over the time steps \(h\in\{2,\cdots,H\}\). In the \(h^{th}\) iteration, it learns a policy cover \(\Psi_{h}\) for time step \(h\) containing open-loop policies. This is done by first augmenting the policy cover for previous time step by one step. Formally, we define \(\Upsilon_{h}=\Psi_{h-1}\circ\mathcal{A}=\{\pi\circ a\mid\pi\in\Psi_{h-1},a\in \mathcal{A}\}\) where \(\pi\circ a\) is an open-loop policy that follows \(\pi\) till time step \(h-1\) and then takes action \(a\). Since we assume the transition dynamics to be near-deterministic, therefore, we know that there exists a policy cover for time step \(h\) that is a subset of \(\Upsilon_{h}\) and whose size is equal to the number of reachable states at time step \(h\). Further, as the transitions are near-deterministic, we refer to an open-loop policy as a path, as we can view the policy as tracing a path in the latent transition model.

Further, we can view this prediction problem as learning a multistep inverse dynamics model since the open-loop policy contains information about all previous actions and not just the last action. For every pair of paths in \(\Upsilon_{h}\), we first compute a path prediction gap \(\widehat{\Delta}(\text{line 7})\). If the gap is too small, we show it implies that these paths reach the same endogenous state, hence we can eliminate a single redundant path from this pair (line 8). Finally, \(\Psi_{h}\) is defined as the set of all paths in \(\Upsilon_{h}\) which were not eliminated.

Recovering an endogenous state decoder. We can recover a endogenous state decoder \(\hat{\phi}_{h}\) for each time step \(h\in\{2,3,\cdots,H\}\) directly from \(\hat{f}_{h}\) as shown below: \[\hat{\phi}_{h}(x)=\min\left\{i\mid\hat{f}_{h}(i\mid x)\geq\max_{j}\hat{f}_{h}(j \mid x)-\mathcal{O}(\nicefrac{{1}}{{\Gamma_{h}}}),i\in[\left|\Upsilon_{h} \right|]\right\}.\] Intuitively, this assigns the observation to the path with smallest index that has the highest chance of visiting \(x\), and therefore, \(\phi^{\star}(x)\). We are implicitly using the decoder for exploring, since we rely on using \(\hat{f}_{h}\) for making planning decisions.

PPE can also be used to recover a latent endogenous transition dynamics.

We then define an abstract state space \(\widehat{\mathcal{S}}_{h}\) for time step \(h\) that contains an abstract state \(j\) for each path \(\upsilon_{j}\in\Psi_{h}\). Further, we recover a latent deterministic transition dynamics for time step \(h-1\) as \(\hat{T}_{h-1}:\widehat{\mathcal{S}}_{h-1}\times\mathcal{A}\to\widehat{\mathcal{ S}}_{h}\) where we set \(\hat{T}_{h-1}(i,a)=j\) if the path \(\upsilon_{j}\in\Psi_{h}\) gets merged with path \(\upsilon_{i}^{\prime}\circ a\in\Psi_{h}\) where \(\upsilon_{i}^{\prime}\in\Psi_{h-1}\).

However, the recovered policy cover and dynamics can be directly used to optimize any given reward function with existing methods.

If the reward function only depends on the endogenous state, we can use a computationally cheaper value-iteration VI that uses the recovered transition dynamics. VI is a model-based algorithm that estimates the reward for each state and action, and performs dynamic programming on the model (see Appendix D.2 for details). In each case, the sample complexity of learning a near-optimal policy, given the output of PPE, scales with the size of endogenous and not the exogenous state space. We provide the main sample complexity guarantee for PPE as well as additional intuition for why it works.

_We assume the endogenous dynamics is \(\eta\)-close to a deterministic model \((\mu_{D,\eta},T_{D,\eta})\) where \(\eta\leq 1/(4SH)\). _We make a realizability assumption for the regression problem solved by PPE (line 5). We assume that \(\mathcal{F}\) is expressive enough to represent the Bayes optimal classifier of the regression problems created by PPE. **Assumption 2** (Realizability).: _For any \(h\in[H]\), and any set of paths \(\Upsilon\subseteq\mathcal{A}^{h}\) with \(|\Upsilon|\leq SA\) and where \(\mathcal{A}^{h}\) denotes the set of all paths of length \(h\), there exists \(f^{\star}_{\Upsilon,h}\in\mathcal{F}\) such that: \(f^{\star}_{\Upsilon,h}(\mathtt{idx}(\upsilon)\mid x)=\frac{\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon)}{\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}( \phi^{\star}(x))|\upsilon^{\prime})}\), for all \(\upsilon\in\Upsilon\) and \(x\in\mathcal{X}\) with \(\sum_{\upsilon^{\prime}\in\Upsilon}\mathbb{P}_{h}(\phi^{\star}(x))\mid \upsilon^{\prime})>0\). _Realizability assumptions are common in theoretical analysis (e.g., Misra et al. (2020), Agarwal et al. (2020)). In practice, we use expressive neural networks to solve the regression problem, so we expect the realizability assumption to hold. Note that there are at most \(A^{S(H+1)}\) Bayes classifiers for different prediction problems. However, this is acceptable since our guarantees will scale as \(\ln|\mathcal{F}|\) and, therefore, the function class \(\mathcal{F}\) can be exponentially large to accommodate all of them.

Let \(v_{1},v_{2}\in\Upsilon_{h}\) be two paths with indices \(i\) and \(j\) respectively. We define their exact path prediction gap as \(\Delta(i,j):=\mathbb{E}_{x_{h}}\left[\left|f^{*}_{h}(i\mid x_{h})-f^{*}_{h}(j \mid x_{h})\right|\right]\). Assume that \(v_{1}\) visits an endogenous state \(s\) at time step \(h\) and denote \(\omega(s)\) as the number of paths in \(\Upsilon_{h}\) that reaches \(s\). Then \(f^{*}_{h}(i\mid x_{h})=\nicefrac{{1}}{{\omega(s)}}\) if \(\phi^{*}(x_{h})=s\), and \(0\) otherwise. If \(v_{2}\) also visits \(s\) at time step \(h\), then \(f^{*}_{h}(i\mid x_{h})=f^{*}_{h}(j\mid x_{h})\) all \(x_{h}\). This implies \(\Delta(i,j)=0\) and \(\mathtt{PPE}\) will filter out the path with higher index since it detected both paths reach to the same endogenous state. Conversely, let \(v_{2}\) visit a different state at time step \(h\). If \(x\) is an observation that maps to \(s\), then \(f^{*}_{h}(i\mid x)=\nicefrac{{1}}{{\omega(s)}}\) and \(f^{*}_{h}(j\mid x)=0\). This gives \(\left|f^{*}_{h}(i\mid x)-f^{*}_{h}(j\mid x)\right|=\nicefrac{{1}}{{\omega(s)} }\geq\nicefrac{{1}}{{\left|\Upsilon_{h}\right|}}\) and, consequently, \(\Delta(i,j)>0\). In fact, we can show \(\Delta(i,j)\geq\mathcal{O}(\nicefrac{{1}}{{1}}\nicefrac{{1}}{{\left|\Upsilon_ {h}\right|}})\). Thus, \(\mathtt{PPE}\) will not eliminate these paths upon comparison. Our complete analysis in the Appendix generalizes the above reasoning to finite sample setting where we can only approximate \(f^{*}_{h}\) and \(\Delta\), as well as to EX-BMDPs with near-deterministic dynamics.

We evaluate \(\mathtt{PPE}\) on two domains: a challenging exploration problem called _combination lock_ to test whether \(\mathtt{PPE}\) can learn an optimal policy and an accurate state decoder, and a visual-grid world with complex visual representations to test whether \(\mathtt{PPE}\) is able to recover the latent dynamics.

We visualize the latent endogenous dynamics in Figure 1(a).

An observation \(x\) is generated stochastically from a latent state \(z=(s,\xi)\). We map \(s\) to a vector \(w\) encoding the identity of the state. We concatenate \((w,\xi)\), add Gaussian noise to each dimension, and multiply the result by a Hadamard matrix to generate \(x\). See Appendix F for full details.

Lastly, PPE is able to solve the problem for all values of \(H\) and is significantly more sample efficient than baselines. Since the reward function of the combination-lock problem depends only on the endogenous state, we run PPE and then a value-iteration like algorithm (see Appendix D.2) to learn a near optimal policy.

When there is no exogenous noise, Homer is able to learn a good state decoder with enough samples while ID fails to learn, in accordance with the theory.

We test the ability of PPE to recover the latent endogenous transition dynamics in visual grid-world problem.2 The agent navigates in a \(N\times N\) grid world where each grid can contain a stationary object, the goal, or the agent. The agent's endogenous state is given by its position in the grid and its direction amongst four possible canonical directions. The agent can take five different actions for navigation. The world is visible to the agent as a \(8N\times 8N\) sized RGB image.

The exogenous state is given by the position, size and color of ellipses and is much larger than \(|\mathcal{S}|\leq 4N^{2}\).

Since the problem has deterministic dynamics, we evaluate the accuracy of the learned transition model by measuring it in terms of accuracy of the elimination step (Algorithm 1, line 8), since this step induces our algorithm's mapping from observations to endogenous latent states.

Type 1 error computes whether PPE merged these paths, i.e., predicted them as mapping to the same abstract state, when they go to _different endogenous_ states.

We see that PPE is able to reduce the number of type 1 errors down to 0 using \(2\times 10^{5}\) episodes per time step.

Similarly, PPE is able to reduce type 2 errors and is able to get them down to 56 with \(5\times 10^{5}\) episodes.

Further, the agent is able to plan using the learned transition model and receive the optimal return.

As expected, PPE rapidly learns a highly accurate decoder while Homer performs only as well as a random uniform decoder.

We present PPE that learns a multi-step inverse dynamics to filter exogenous noise and successfully explores. We derive theoretical guarantees for PPE in near-deterministic setting and provide encouraging experimental evidence in support of our arguments.

Our work also raises interesting future questions such as how to address the general setting with stochastic transitions, or handle more complex endogenous state representations.

Input: Figure 1: Left: An agent is walking next to a pond in a park and observes the world as an image. The world consists of a latent endogenous state, containing variable such as agent’s position, and a much larger latent exogenous state containing variables such as motion of ducks, ripples in the water, etc. Center: Graphical model of the EX-BMDP. Right: PPE learns a generalized form of inverse dynamics that recovers the endogenous state.

Input: Figure 2: Results on combination lock. Left: We show the latent transition dynamics of combination lock. Observations are not shown for brevity. Center: Shows minimal number of episodes needed to achieve a mean regret of at most V (π?)/2. Right: State decoding accuracy (in percent) of decoders learned by different methods. Solid lines implies no exogenous dimension while dashed lines imply an exogenous dimension of 100.

Input: Figure 3: Results on visual grid world. Left two: Shows sampled observations for the first two steps from the visual gridworld domain. The agent is depicted as a red-triangle, lava in orange, walls in grey, and the goal in green. Center Right: Shows errors of type 1 and type 2 made by the PPE in recovering the latent endogenous dynamics. Right: State decoding accuracy of PPE, Homer and a random uniform decoder. (see Section 6)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "PPE is able to recover the latent endogenous model in a visually complex navigation problem, in accordance with the theory" appears to be well-supported. The evidence shows:

1. In the visual grid-world experiments, PPE successfully:
- Reduced Type 1 errors to 0 with sufficient samples
- Achieved high state decoding accuracy (~95%)
- Learned a transition model that enabled optimal returns
- Outperformed baseline methods like Homer

2. The results align with the theoretical framework presented for near-deterministic dynamics.

The claim is measured in its scope, acknowledging it's for a specific type of navigation problem, and the experimental results directly validate the theoretical predictions. The claim avoids overgeneralization or exaggeration.

<score>1</score>