- The objective function (1): I have two concerns about the definition of this objective: 1. If the intuitive goal consists of finding a set of policies that contains an optimal policy for every test MDP in S_{test}, I would rather evaluate the quality of \overline{\Pi} with the performance in the worst MDP. In other words, I would have employed the \min over S_{test} rather than the summation. With the summation we might select a subset of policies that are very good for the majority of the MDPs in S_{test} but very bad of the remaining ones and this phenomenon would be hidden by the summation but highlighted by the \min. 2. If no conditions on the complexity of \overline{\Pi} are enforced the optimum of (1) would be exactly \Pi, or, at least, the largest subset allowed. - Latent-Conditioned Policies: How is this different from considering a hyperpolicy that is used to sample the parameters of the policy, Like in Parameter-based Exploration Policy Gradients (PGPE)? Sehnke, Frank, et al. "Policy gradients with parameter-based exploration for control." International Conference on Artificial Neural Networks. Springer, Berlin, Heidelberg, 2008. - Choice of the latent variable distribution: at line 139 the authors say that p(Z) is chosen to be the uniform distribution, while at line 150 p(Z) is a categorical distribution. Which one is actually used in the algorithm? Is there a justification for choosing one distribution rather than another? Can the authors motivate? **Minor*** - line 64: remove comma after a_i - line 66: missing expectation around the summation - line 138: what is H(Z)? - line 322: don’t -> do not - Equation (2): at this point the policy becomes a parametric function of \theta. Moreover, the dependence on s when using the policy as an argument for R_{\mathcal{M}} should be removed - Figure 3: the labels on the axis are way too small - Font size of the captions should be the same as the text
- The objective function (1): I have two concerns about the definition of this objective: