Neural Processes (NPs) (Garnelo et al., 2018) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data efficiently with linear complexity in the number of context input-output pairs, and can learn a wide family of conditional distributions; they learn predictive distributions conditioned on context sets of arbitrary size. Nonetheless, we show that NPs suffer a fundamental drawback of underfitting, giving inaccurate predictions at the inputs of the observed data they condition on. We address this issue by incorporating attention into NPs, allowing each input location to attend to the relevant context points for the prediction. We show that this greatly improves the accuracy of predictions, results in noticeably faster training, and expands the range of functions that can be modelled.
A model for regression that learns conditional distributions of a stochastic process, by incorporating attention into Neural Processes. Proposes to resolve the issue of underfitting in the neural process method by adding an attention mechanism to the deterministic path. An extension to the framework of Neural Processes that adds an attention-based conditioning mechanism, allowing the model to better capture dependencies in the conditioning set. The authors extend neural processes by incorporating self-attention for enriching the features of the context points and cross-attention for producing a query-specific representation. They resolve the underfitting problem of NPs and show ANPs to converge better and faster than NPs.