] *assumption* - I am not sure if it is safe to assume any programmatic policy can be parameterized by a vector \theta and is differentiable in \theta. (for Theorem 4.2) *initial policy* - In all the experiments (TORCS, MountainCar, and Pendulum), the IPPG polices improve upon the PRIOR. It is not clear if IPPG can learn from scratch. Showing the performance of IPPG learning from scratch would be important to verify this. - Can IPPG be initialized with a neural policy? It seems that it is possible based on Algorithm 1. If so, it would be interesting to see how well IPPG work using a neural policy learned with DDPG instead of PRIOR. Can IIPG improve upon DDPG? *experiment setup* - It is mentioned that "both NDPS and VIPER rely on imitating a fixed neural policy oracle" (L244). What is this policy oracle? Is this the policy learned using DDPG shown in the tables? If not, what's the performance of using NDPS and VIPER to distill the DDPG policies? - It would be interesting to see if the proposed framework works with different policy gradient approaches. *experiment results* - How many random seeds are used for learning the policies (DDPO and IPPG)? - What are the standard deviation or confidence intervals for all performance values? Are all the tracks deterministic? Are the DDPG policies deterministic during testing? - It would be better if the authors provided some videos showing different policies controlling cars on different tracks so that we can have a better idea of how different methods work. *reproducibility* - Some implementation details are lacking from the main paper, which makes reproducing the results difficult. It is not clear to me what policy gradient approach is used. - The provided dropbox link leads to an empty folder (I checked it on July 5th). *related work* - I believe it would be better if some prior works [1-5] exploring learning-based program synthesis frameworks were mentioned in the paper. *reference* [1] "Neuro-symbolic program synthesis" in ICLR 2017 [2] "Robustfill: Neural program learning under noisy I/O" in ICML 2017 [3] "Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis" in ICLR 2018 [4] "Neural program synthesis from diverse demonstration videos" in ICML 2018 [5] "Execution-Guided Neural Program Synthesis" in ICLR 2019 ----- final review ----- After reading the other reviews and the author response, I have mixed feelings about this paper. On one hand, I do recognize the importance of this problem and appreciate the proposed framework (IPPG). On the other hand, many of my concerns (e.g. the choices of initial policy, experiment setup, and experiment results) are not addressed, which makes me worried about the empirical performance of the proposed framework. To be more specific, I believe the following questions are important for understanding the performance of IPPG, which remain unanswered: (1) Can IPPG learn from scratch (i.e. where no neural policy could solve the task that we are interested in)? The authors stated that "IPPG can be initialized with a neural policy, learned for example via DDPG, and thus can be made to learn" in the rebuttal, which does not answer my question, but it is probably because my original question was confusing. (2) Can IPPG be initialized with a neural policy? If so, can IPPG be initialized with a policy learned using DDPG and improve it? As DDPG achieves great performance on different tracks, I am just interested in if IPPG can even improve it. (3) How many random seeds are used for learning the policies (DDPO and IPPG)? What are the standard deviation or confidence intervals for all performance values? I believe this is important for understanding the performance of RL algorithms. (4) What is the oracle policy that NDPS and VIPER learn from? If they do not learn from the DDPG policy, what is the performance if they distill the DDPG policy. (5) Can IPPG learn from a TPRO/PPO policy? While the authors mentioned that TRPO and PPO can't solve TORCS tasks, I believe this can be verified using the CartPole or other simpler environment. In sum, I decided to keep my score as 5. I am ok if this paper gets accepted (which is likely to happen given positive reviews from other reviewers) but I do hope this paper gets improved from the above points. Also, it would be good to discuss learning-based program synthesis frameworks as they are highly-related.
- It would be interesting to see if the proposed framework works with different policy gradient approaches. *experiment results* - How many random seeds are used for learning the policies (DDPO and IPPG)?