weakness of this paper is its isolation to the plain GP regression setting. Although this is expected given the methodology used to enable tractability, I would have appreciated at least some discussion into whether any of the material presented here can be extended to the classification setting. Of course, one could argue that using Laplace or EP already implicitly takes away from the âexactnessâ of a GP, but I think there is still scope for having an interesting discussion here (possibly akin to that provided in Cutajar et al, 2015). Likewise, any intuition of how/whether this can be extended to more advanced GP set-ups, such as multi-task, convolutional, and recurrent variations (among many others) would also be useful. -- Technical Quality/Evaluation -- The technical contributions and implementation details are easy to follow, and I did not find any faults in that respect. The experimental evaluation is also varied and convincingly shows that exact GP inference widely outperforms standard approximations. Nonetheless I have a few concerns listed below; - It appears that in nearly all experiments, the results are reported for a single held-out test set. Standard practice in most papers on GPs involves using a number of train/test splits or folds which give a more accurate illustration of the methodâs performance. While I imagine that the size of the datasets considered in this work entail that this can take quite a long time to complete, I highly encourage the authors to carry out this exercise; - If I understood correctly, a kernel with a shared length scale is used in all experiments, which does not conform to the ARD kernels one would typically use in a practical setting. While several papers presenting approximate GPs have also made this assumption in the past (e.g. Hensman et al (2013)), more recent work such as the AutoGP by Krauth et al. (2017) emphasise why ARD should more consistently be used, and demonstrate how automatic differentiation frameworks minimise the performance penalty introduced by using such schemes. I believe this has to be addressed more clearly in the paper, and would also give more meaning to the commentary provided in L243-248, which otherwise feels spurious. I consider this to be crucial for painting a more complete picture in the Results section. - For a GP paper, I also find it strange that results for negative log likelihood are not reported here. While these are expected to follow a similar trend to RMSE, I would definitely include such results in a future version of the manuscript since this also has implications on uncertainty calibration. On a related note, I was surprised this paper did not have any supplementary material attached, because further experiments and more results would definitely be welcome. -- Overall recommendation -- This paper does not introduce any major theoretical elements, but the technical contributions featured here, along with the associated practical considerations, are timely in showing how modern GPU architectures can be exploited for carrying out exact GP training and inference to an extent which had not previously been considered. The paper is well written and some of the results should indeed stimulate interesting discussions on whether standard GP approximations are still worthwhile for plain regression tasks. Unfortunately, given how this paperâs worth relies heavily on the quality of the experimental results, there are a few technical issues in the paper which I believe should be addressed in a published version. I also think that several of the discussions featured in the paper can be expanded further - the authors should not refrain from including their own intuition on the broader implications of this work, which I feel is currently missing. -- Post-rebuttal update -- I thank the authors for their rebuttal. I had a positive opinion of the paper in my initial review, and most of my principal concerns were sufficiently addressed in the rebuttal. After reading the other reviews, I do believe that there is a common interest in having more experiments included. Coupled with my other suggested changes to the current experimental set-up, I think there is still some work to be done in this respect. This also coincides with my wish to see more of your own insights included in the paper, which I think will steer and hopefully also encourage more discussion on this interesting dilemma on where to invest computational resources. Nevertheless, I do not expect such additional experiments to majorly alter the primary âstorylineâ of the paper, which is why Iâm not lowering my current evaluation of the paper. Due to the limited novelty of the technical aspects, I am likewise not inclined to raise my score either, but I think this is a good paper nonetheless. Irrespective of whether this paper is ultimately accepted or not, I definitely hope to see an updated version containing an extended experimental evaluation and discussion.
- It appears that in nearly all experiments, the results are reported for a single held-out test set. Standard practice in most papers on GPs involves using a number of train/test splits or folds which give a more accurate illustration of the methodâs performance. While I imagine that the size of the datasets considered in this work entail that this can take quite a long time to complete, I highly encourage the authors to carry out this exercise;