This method trades one intractible problem for another: it requires the learning of cross-values v e ′ ( x t ; e )
for all pairs of possible environments e , e ′
. It is not clear that this will be an improvement when scaling up.
At a few points the paper introduces approximations, but the gap to the true value and the implications of these approximations are not made completely clear to me. The authors should be more precise about the tradeoffs and costs of the methods they propose, both in terms of accuracy and computational cost.
On page 6, it claims that estimating v c
according to samples will lead to Thompson sampling-like behavior, which might lead to better exploration. This seems a bit facetious given that this paper attempts to find a Bayes-optimal policy and explicitly points out the weaknesses of Thompson sampling in an earlier section.
Not scaled to larger domains, but this is understandable.
Questions and minor comments
Is the belief state conditioning the policy also supposed to change with time τ
? As written it looks like the optimal Bayes-adaptive policy conditions on one sampled belief about the environment and then plays without updating that belief.
It is not intuitive to me how it is possible to estimate v f
, despite the Bellman equation written in Eq. 12. It would seem that this update would have to integrate over all possible environments in order to be meaningful, assuming that the true environment is not known at update time. Is that correct?
I guess this was probably for space reasons, but the bolded sections in page 6 should really be broken out into \paragraphs — it's currently a huge wall of text.
12. It would seem that this update would have to integrate over all possible environments in order to be meaningful, assuming that the true environment is not known at update time. Is that correct? I guess this was probably for space reasons, but the bolded sections in page 6 should really be broken out into \paragraphs — it's currently a huge wall of text.