- The approach proposed is quite simple and straightforward without much technical innovation. For example, CODAC is a direct combination of CQL and QR-DQN to learn conservative quantiles of the return distribution. - Some parts of the paper need clearer writing (more below)
Comments and questions: - I think in a paragraph from lines 22-30 when discussing distributional RL, the paper lacks relevant literature on using moment matching (instead of quantile regression as most DRL methods) for DRL (Nguyen-Tang et al AAAI’21, “Distributional Reinforcement Learning via Moment Matching”). I think this should be properly discussed when talking about various approaches to DRL that have been developed so far, even though the present paper still uses quantile regression instead of moment matching. - More explanation is needed for Eq (5). For example, what is the meaning of the cost c 0 ( s , a )
? (e.g., to quantify out-of-distribution actions) - The use of s ′ and a ′
when defining $\hat{\pi}{\beta} a t l i n e 107 m i g h t c a u s e c o n f u s i o n a s \mathcal{D} c o n t a i n s
(s,a,r,s’)$. - This paper is about deriving a conservative estimate of the quantiles of the return from offline data where the conservativeness is for penalizing out-of-distribution actions. In the paper, they define OOD actions as those are not drawn from \hat{\pi}{\beta}(.|s) (line 109) but in Assumption 3.1. they assume that \hat{\pi}_{\beta}(a|s) > 0, i.e., there is no OOD actions. Thus, what is the merit of the theoretical result presented in the paper?
The authors have adequately addressed the limitations and social impact of their work.
- I think in a paragraph from lines 22-30 when discussing distributional RL, the paper lacks relevant literature on using moment matching (instead of quantile regression as most DRL methods) for DRL (Nguyen-Tang et al AAAI’21, “Distributional Reinforcement Learning via Moment Matching”). I think this should be properly discussed when talking about various approaches to DRL that have been developed so far, even though the present paper still uses quantile regression instead of moment matching.