Weakness:
The presentation of background section can be improved as it is too dense with different notations. It may be a good idea to summarize these notations into a table.
Only consider 2 environments: 1 with non-local reward and 1 with long horizon and resonant frequencies.
I have the following questions and comments: Q1. What is the motivation of changing frequency of pendulum to 1.7? instead of 0.5Hz? Have you tried different settings of the pendulum environment (use the original frequency of 0.5Hz)? If yes I am curious how other methods perform compare to TDPO.
C1. It would be nice to have a discussion to compare the computation cost of each iteration of TDPO and other methods like TRPO and TD3.
C2. As shown in the experiments, TDPO does not work well in the common gym environments, is there any changes to algorithm design possible to improve the current algorithm for these setting?
C3. From the way the method is presented, it seems that it is not simple to implement the method? It is better to have a discussion on the implementation aspect of the method.
Small comments:
Secion 1, first paragraph, line 6, there are two \delta_s
The definition of KL divergence in terms of Hilbert space inner product missing a subscript?
Introduce the definition of the term h.o.t. before use although it can be inferred from the context
Overall, I suggest a weak-accept decision based on the following reason:
Novelty in algorithm development with theoretical justification for the monotonic improvement.
Although experiments indeed show the clear advantage of the proposed method over existing ones, more environments or more setting in the same environment should be presented to better evaluate its performance.
1 with non-local reward and 1 with long horizon and resonant frequencies. I have the following questions and comments: