As the authors note, it seems the method still requires a few tweaks to work well empirically. For example, we need to omit the log of the true rewards and scale the KL term in the policy objective to 0.1. While the authors provide a brief intuition on why those modifications are needed, I think the authors should provide more concrete analysis (e.g., empirical results) on what problems the original formula have and how the modifications fix them. Also, it would be better if the authors provide ablation results on those modifications. For example, does the performance drop drastically if the scale of the KL term changes (to 0.05, 0.2, 0.5, ...) ?
The compute comparison vs. REDQ on Figure 3 seems to be misleading. First, less runtime does not necessarily mean less computational cost. Second, if the authors had used the official implementation of REDQ [1], it should be emphasized that this implementation is very inefficient in terms of runtime. In detail, the implementation feeds forward to each Q-network one at a time while this feed-forward process is embarrassingly parallelizable. The runtime of REDQ will drop significantly if we parallelize this process.
The ablation experiments seem to show that the value term for the encoder is not necessary. It would be better to provide an explanation on this result.
Some of the equations seem to have typos. 1) On equation (4), the first product and the second product have exactly the same form. 2) On algorithm 1 Line 8, shouldn't we use s_n instead of s_t? Questions
I am curious of the asymptotic performance of the proposed method. If possible, can the authors provide average return results with more env steps?
[1] https://github.com/watchernyu/REDQ
2) On algorithm 1 Line 8, shouldn't we use s_n instead of s_t? Questions I am curious of the asymptotic performance of the proposed method. If possible, can the authors provide average return results with more env steps? [1] https://github.com/watchernyu/REDQ