(A) To my read, the discussion of the algorithm and implementation are not quite complete in the sense that I still have several important questions that I did not feel were answered: (1) I am confused as to why/how the \beta evolves during the learning process and how this effects the goals one has when using a BMDP. Do we not care about how much risk is induced (or negative effects experienced) during training? Moreover, if the policy selects a new \beta at each new time step, how do we enforce a *specific* \beta when we wish to *use* the policy? (2) The convex hull procedure seems critical to being able to actually use the algorithm, but the explanation is lacking any intuitive interpretation. For example, it's not clear to me from 4.1 and Algorithm 3 exactly how "Budget \beta [is] always respected" (comment on Algorithm 3, line 9). Can the authors provide more explanation/intuition about what is going on here? (3) The authors state that "The pseudo-code of our exploration procedure is shown in Algorithm 4 in Appendix B." Since this component of your algorithm is one of the main hypotheses that is being validated, it should appear in the main paper. Suggest swapping for Algorithms 1/2 since these are basic extensions of existing techniques. (B) There is a mismatch between the introduction of the paper, which speaks generally to using BMDPs, and the actual experiments run, which show the benefit of using BMDPs *compared to* computing a set of solutions using existing techniques like FTQ(\lambda). The introduction needs to mention that approaches like the latter *are* available solutions and frame the contribution of the paper rather as one of providing a "better" solution in whichever way the authors feel this is best described (more-efficient, etc.). MINOR COMMENTS: * It seems that `else` at the beginning of Algorithm 3, line 9 doesn't belong there. * Several times in the paper, it is mentioned that experiments are done in "two environments," but aren't there three? * The definition in (2) is odd given that you say the "budget evolves as part of the dynamics." That is, if \beta'=\beta_a (as suggested by the Dirac function, then it is merely whatever the action says it was, correct? Why is that part of "the dynamics?"
* Several times in the paper, it is mentioned that experiments are done in "two environments," but aren't there three?