I feel the motivation of the work is confusing. I can understand the authors want to improve CQL somehow further. But it is never made clear:
what existing problems are and why they matter; Is it the lower bound on the exiting CQL is too loose? Why is improving the bound important?
what is the effect you want to achieve? Is it an offline algorithm that can learn a better policy from data generated by a poor behavior policy?
The contribution is incremental, and I doubt the significance. As the paper cited in section 2.2, Kumar et al. (2020) propose to penalize the actions not described by the dataset, which enables a general definition of \mu. Note that the additional weighting scheme can be essentially thought of as a new type of \mu. I don’t see a clear difference between (2) and (3). Both can be considered as the specially designed action sampling distribution.
Why is theorem 1 useful? If I understand correctly, the key thing you want to say is the additional weighting can provide a tighter bound for OOD state-action pairs or those not close to the dataset? But the first step should be figuring out the effects of having a tight/loose bound. Does it hurt optimality/convergence rate/generalization…? Even partially answering this question can better motivate the reweighting approach. I believe the proof of the theorem is a simple modification from the existing CQL work.
The choice of the weighting scheme lacks justification. An intuitive choice is the RBF function. Furthermore, according to theorem 1, the proposed weighting is useful only when the action is OOD, i.e., the weight is 1; when the action is ID, weight should be 0, but your weighting scheme does not give zero?
Section 4.1, the proposed method comes out suddenly. Is there any reason to choose normalizing flows? Of course, normalizing flows is a good method enabling both efficient sampling and density evaluation. In your algorithm, you only need to evaluate the density but not to sample. There should be plenty of other choices. When testing ideas, it is more natural to start with some simple methods.
What is the purpose of section 4.3?
I expect to see how various concrete choices of the weighting scheme can affect the distance (e.g., KL divergence) between the learned policy and the behavior policy.
The experiments. 1. more random seeds should be tested (figure 1) — it is hard to distinguish algorithms from the current learning curves. Readers cannot see a clear message from them. 2. I expect more baselines to be compared and more domains to be tested. As I mentioned, the choices of the weighting and the way of learning density functions are not strongly motivated. In this case, I have to ask for stronger empirical results: baselines with other design choices and more domains. 3. The experiments in Fig 2 are incomplete. Why are there no experiments for half cheetah and walker with expert data? 4. Please provide reproducing details.
The abstract says, “… with a strong theoretical guarantee.” I don’t think there is any strong theory in the paper.
Page 3. Last paragraph. The criticism towards using empirical dataset distribution for \hat{pi}_\beta does not make sense to me. When the state/action is continuous, the empirical estimation should be kernel density estimation, which is a consistent estimator for estimating empirical distributions. The kernel can be chosen as smooth, so the KDE should have support everywhere. Minor:
there is a nontrivial number of grammar issues/typos. Please double-check.
Many sentences are confusing or logically disconnected.
e.g., in the abstract, “A compromise between enhancing …. To alleviate this issue, … ” what issue?
“Improving the learning process.” In what sense? Higher sample efficiency?
“Indeed, the lack of information … ” what information? Why does it provoke overestimation?
I believe saying “based on the amount of information collected” is inaccurate because the paper does not really introduce any information measurement.
2. I expect more baselines to be compared and more domains to be tested. As I mentioned, the choices of the weighting and the way of learning density functions are not strongly motivated. In this case, I have to ask for stronger empirical results: baselines with other design choices and more domains.