I have certain concerns about the paper.
1/ I think the contribution of the paper is a bit limited. V-MAIL combines several existing ideas, e.g., latent imagination, GAIL, variational state space model, etc., and achieves a good performance. However, how these components affect each other and how they contribute to the final performance are not clear. An ablation study is also missing from the paper. In this case, it would be hard to get inspiration from reading it.
2/ Although the paper has provided theoretical analysis about the model-based adversarial imitation learning (sect. 4.1 and sect. 4.2), they are disconnected from the practical implementations (sect. 4.3). In particular, Theorem 1 shows that the divergence of the visitation distributions in a MDP can be upper bounded by a divergence of the visitation distributions in a POMDP. However, in the practical implementation, a variational state space model captures the belief, rather than the visitation distribution of the belief. In addition, it feels difficult to compute the visitation frequency of a belief, whose size is exponential to the size of the history and the state space. I believe the proposed algorithm indeed has its merit, but I don’t think Theorem 1 provides a correct justification of the optimization objective used in this paper.
3/ I feel the author should be careful when making certain claims. For example, from line 39 to line 48, the authors are analyzing the limitations of the existing IRL methods and adversarial imitation learning methods. “These approaches explicitly train a GAN-based classifier [17] to distinguish the visitation distribution of the agent from the expert, and use it as a reward signal for training the agent with RL….” However, not all IRL methods are adversarial imitation learning. In fact, most of them don’t train a GAN-based classifier and do RL afterwards. Instead, a lot of them recover the reward and do planning instead.
4/ The authors claimed that V-MAIL achieves zero-shot transfer to novel environments. However, the policy is fine-tuned with additional expert demonstrations, as shown in Alg. 2. Why is this zero-shot? In addition, I guess the transferability might be limited by the real difficulty of the source task / target task. Walker-run is clearly harder than walker-walk, so the policy transfer here is possible. Also, for the manipulation scenario, 3-prong task with both clockwise / counter clockwise rotations, together with one 4-prong task, actually provides sufficient information about the target task. I guess it might be difficult to transfer a policy from simpler tasks to more complex tasks. This has to be made clear in the paper. Otherwise, it is quite misleading.
2. Why is this zero-shot? In addition, I guess the transferability might be limited by the real difficulty of the source task / target task. Walker-run is clearly harder than walker-walk, so the policy transfer here is possible. Also, for the manipulation scenario, 3-prong task with both clockwise / counter clockwise rotations, together with one 4-prong task, actually provides sufficient information about the target task. I guess it might be difficult to transfer a policy from simpler tasks to more complex tasks. This has to be made clear in the paper. Otherwise, it is quite misleading.