The choice of inverse dynamics as a regularizer is not well-motivated. This choice is not grounded in the same way that the rest of the paper is. For example, why choose an inverse dynamics model and not a different self-supervised loss? Why not a different combination of approaches? The motivation given for inverse dynamics could also be applied to reconstructions or contrastive coding. Additionally, there are no experimental results showing that an inverse dynamics model is best; as such this choice is the weakest part of the paper.
As there is no experimental comparison to other additions to DBC (e.g. different self-supervised learning approaches), it is unclear why inverse dynamics is the preferred method to solving the issues with DBC.
The paper focuses heavily on one method; DBC, and as such is somewhat limited in scope.
The paper, while well-written, is not fully self-contained - certain terms are used without being defined. A few examples:
What is a ‘compact subset of S ’?
What is a diameter of a state space?
What is sup? The supremum?
What is supp? Also the supremum?
What is H
in line 259?
What is a low dispersion reward?
According to the paper, DBC suffers from common metric learning problems, but there are no references included to other metric learning papers that exhibit similar issues.
If the intrinsic reward is based on the forward model error, it should not be useful under state collapse, as the error would be 0.
The non stationarity introduced by IR might be an issue in more complex environments.
Line 294: Claim: “Hence, we expect less disruption to the BSM encoding than other auxiliary tasks (e.g. reconstruction)”. Reconstruction is not the only possible auxiliary task. What happens when using a better auxiliary loss, such as action-dependent contrastive coding?
Ablation study is incomplete -- DBC-ID and DBC-normed-ID are not included.
“(Left) Our method significantly improves upon DBC [36] and is on par with SAC” this looks like it isn’t true, as SAC achieves a higher performance in the end.
The method isn’t evaluated on the same benchmarks as DBC, with comparable levels of distraction (e.g. natural video). As a result, it is unclear what the performance improvement would be in those settings. Questions:
“two states that only differ in rewards far in the future should still not be bisimilar” - Standard bisimilarity is defined over immediate, not future rewards. Could you elaborate?
The assumption that the reward is normalized between 0 and 1 -- is this taken into account in the experiments?
The sentences 193-195 are unclear, could you clarify?
Eq. 11: x \in R^d is this d overloaded?
Eq. 11: what is B_c? The ball with radius c? It looks like c is not defined.
Eq. 12: Does M(B_c) mean the space of probability distributions over the ball with radius c?
Lemma 3 assumes a stationary distribution over states, but this distribution changes as the policy changes. How does this influence the theory/results?
Figure 1: what is magnitude? The magnitude of the embedding?
Minor comments:
Below are some suggested references on state representation learning that could be relevant to include in related work.
Plots are hard to read in black and white
While the motivation for requiring more informative rewards is clear (as mentioned in strengths), it is not motivated why this takes the form of intrinsic rewards, as compared to e.g. reward propagation.
SAC does not fail in sparse-cartpole with N_m=1, it just takes longer to converge Overall:
The score is mostly based on the following:
The choice of inverse dynamics (vs action-dependent contrastive coding or another type of auxiliary loss) should be properly motivated, either theoretically or experimentally.
Missing references should be included
Evaluation should be performed on the same benchmarks as DBC, e.g. with natural video distraction.
I would raise the score if these are addressed.
Missing references:
Biza et al, "Online Abstraction with MDP Homomorphisms for Deep Learning"
Grimm et al, "The value equivalence principle for model-based reinforcement learning"
Le Lan et al, "Metrics and continuity in reinforcement learning"
Ravindran & Barto, "Approximate Homomorphisms: A framework for non-exact minimization in Markov Decision Processes"
Taylor et al, "Bounding Performance Loss in Approximate MDP Homomorphisms"
Van der Pol et al, "Plannable Approximations to MDP homomorphisms: equivariance under actions"
Suggested references
Agarwal et al, "Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning"
Corneil et al, “Efficient model-based deep reinforcement learning with variational state tabulation”
Dadashi et al, Offline Reinforcement Learning with Pseudometric Learning
Francois-Lavet et al, “Combined reinforcement learning via abstract representations”
Watter et al, “Embed to control: A locally linear latent dynamics model for control from raw images” Update
Thank you for the rebuttal, and for the update on the experiments. I think adding the new experiments to the paper will make the paper more convincing. When reviewing, I only had a few issues with this paper: the lack of motivation for the choice of inverse dynamics (as opposed to another auxiliary loss), the missing evaluations on benchmarks with natural video distractions (for a fair comparison to DBC), the somewhat limited scope (due to the focus on DBC), some missing references, and the paper not being fully self-contained. In the author response, the authors motivate their choice of inverse dynamics, clarify that their results can be useful for bisimulation metric learning in general, promise to add the references and amend the paper where it is unclear, and in a second reply give an update on benchmarks with natural video distractions, with positive results.
The authors thus address my original concerns. However, reviewer pEeB raised concerns about the theoretical analysis that are important to address. I have therefore decided keep my original score. Limitations:
Yes, the authors discuss limitations of their work. Societal:
Yes, the authors discuss potential societal impact. A minor suggestion would be to add a comment on how improving RL (as this paper aims to do!) could impact society, e.g. in terms of labor markets or autonomous weapons.
11: what is B_c? The ball with radius c? It looks like c is not defined. Eq.