A primary concern is how the proposed method deviates a lot from existing HRL methods which also consider action-repeats? Perhaps the key added value is the extension to continuous control for hybrid action space. It might be useful to elaborate on this in the discussion/related work. I would encourage you to expand the related work section to cover more HRL literature and discuss where the paper is positioned in the context of broader HRL methods.
Arguably most HRL methods allow for the switching between the flat (one-step policies) and abstract actions (multi-step policies), especially the intra-option learning methods, which is the primary claim of this work. So what have we learned here which allows us to scale better to the list of tasks studied here?
What happens when we don’t specifically want action-reproducible policies? Currently a lot of the improvements seem to stem from the nature of tasks where action-repeat might be beneficial.
While the n-score and n-AUC is better as an average over tasks, TAAC is only marginally better or same as top baselines in most tasks as reported in the Figure 8: The unnormalized reward curves of the 14 tasks. It might be useful to expand on this in the main paper as to why you choose the evaluation metric to be n-score and n-AUC and consider averaging over all tasks as the primary metric.
Empirical Analysis:
The experiments evaluate TAAC in 5 categories of 14 continuous control tasks, covering simple control, locomotion, terrain walking (Brockman et al.,572016), manipulation (Plappert et al., 2018), and self-driving.
The baselines are chosen fairly and compare a wide range of action-repeats methods. How do these methods compare to non-action repeat baselines? Any insights on this would be great. In particular, I am curious if the action that is being repeated is not encouraging exploration or performance, how does the method overcome such a choice?
How many random seeds have been considered? Please mention it while reporting.
Writing and Presentation:
The paper can be presented better in terms of explaining how this approach differs from existing approaches which consider action repeats. Consider a summary table which can highlight the novelty of the approach as compared to many other HRL off policy approaches as mentioned by the authors in Sec 1. Sec 2, and Sec. 5.2.
There are several insights in the more observations paragraph which is buried in 5.4. The paper might benefit from re-writing earlier parts to give some intuitions on key differences which this method brings to the table. For example 1) Persistent exploration and the compare-through operator are crucial to the success of this approach. 2) The importance of the formulation of the closed-loop action repetition.
Yes the authors have discussed societal impact. The limitations of the work are also covered in different sections of the paper - it would be valuable to consolidate them and add them in the conclusion section perhaps.
1) Persistent exploration and the compare-through operator are crucial to the success of this approach.