that limit the contribution. In particular: 1) the method does not seem to improve on the robustness and sensitivity as claimed in the motivation of using evolutionary methods in the first place. In fig 3 the new method is noisier in 2 domains and equally noisy as the competitors in the rest. 2) The paper claims SOTA in these domains compared to literature results. The baseline results reported in the paper under review in HalfCheetah, Swimmer and Hopper are worse the state of the art reported in the literature [1,2]. Either because the methods used achieved better results in [1,2] or because the SOTA in the domain was TRPO which was not reported in the paper under review. SOTA is a big claim; support it carefully. 3) There is significant over-claiming throughout the paper. E.g line 275 "best of both approaches", line 87 "maximal information extraction" 4) It is not clear why async methods like A3C were not discussed or compared against. This is critical given the SOTA claim 5) The paper did not really attempt to tease apart what was going on in the system. When evaluating how often was DDPG agent chosen for evaluation trials or did you prohibit this? What does the evaluation curve look like for the DDPG agent? That is do everything you have done, but evaluate the DDPG agent as the candidate from the population. Or allowing the population to produce the data for the DDPG and training it totally off-policy to see how well the DDPG learnings (breaking the bottom right link in fig 1). Does adding more RL agents help training? The paper is relatively clear and the is certainly original. However my concerns above highlight potential issues with quality and significance of the work. [1] https://arxiv.org/abs/1709.06560 [2] https://arxiv.org/pdf/1708.04133.pdf ++++++++++++ Ways to improve the paper that did not impact the scoring above: - you have listed some of the limitations of evolutionary methods, but I think there are much deeper things to say regarding leveraging state, reactiveness, and learning during an episode. Being honest and direct would work well for this work - the title is way to generic and vague - be precise when being critical. What does "brittle convergence properties mean" - I would say DeepRL methods are widely adopted. Consider the landscape 10 years ago. - claim V-trace is too expensive. I have no idea why - its important to note that evolutionary methods can be competitive but not better than RL methods - discussion starting on line 70 is unclear and seems not well supported by data. Say something more plain and provide data to back it up - definition of policy suggests deterministic actions - not sure what state space s = 11 means? typo - section at line 195 seems repetitive. omit
- you have listed some of the limitations of evolutionary methods, but I think there are much deeper things to say regarding leveraging state, reactiveness, and learning during an episode. Being honest and direct would work well for this work - the title is way to generic and vague - be precise when being critical. What does "brittle convergence properties mean" - I would say DeepRL methods are widely adopted. Consider the landscape 10 years ago.