instruction:
In this task, you are given the abstract of a research paper. Your task is to generate a summary of this abstract. Your summary should not be very short, but it's better if it's not more than 30 words.
question:
Deep networks often perform well on the data distribution on which they are trained, yet give incorrect (and often very confident) answers when evaluated on points from off of the training distribution. This is exemplified by the adversarial examples phenomenon but can also be seen in terms of model generalization and domain shift.   Ideally, a model would assign lower confidence to points unlike those from the training distribution.   We propose a regularizer which addresses this issue by training with interpolated hidden states and encouraging the classifier to be less confident at these points.   Because the hidden states are learned, this has an important effect of encouraging the hidden states for a class to be concentrated in such a way so that interpolations within the same class or between two different classes do not intersect with the real data points from other classes.   This has a major advantage in that it avoids the underfitting which can result from interpolating in the input space.   We prove that the exact condition for this problem of underfitting to be avoided by Manifold Mixup is that the dimensionality of the hidden states exceeds the number of classes, which is often the case in practice.   Additionally, this concentration can be seen as making the features in earlier layers more discriminative.   We show that despite requiring no significant additional computation, Manifold Mixup achieves large improvements over strong baselines in supervised learning, robustness to single-step adversarial attacks, semi-supervised learning, and Negative Log-Likelihood on held out samples.
answer:
A method for learning better representations, that acts as a regularizer and despite its no significant additional computation cost , achieves improvements over strong baselines on Supervised and Semi-supervised Learning tasks.


question:
In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.
answer:
Use rate-distortion theory to bound how much a latent variable model can be improved


question:
The question why deep learning algorithms generalize so well has attracted increasing
 research interest. However, most of the well-established approaches,
 such as hypothesis capacity, stability or sparseness, have not provided complete
 explanations (Zhang et al., 2016; Kawaguchi et al., 2017). In this work, we focus
 on the robustness approach (Xu & Mannor, 2012), i.e., if the error of a hypothesis
 will not change much due to perturbations of its training examples, then it
 will also generalize well. As most deep learning algorithms are stochastic (e.g.,
 Stochastic Gradient Descent, Dropout, and Bayes-by-backprop), we revisit the robustness
 arguments of Xu & Mannor, and introduce a new approach – ensemble
 robustness – that concerns the robustness of a population of hypotheses. Through
 the lens of ensemble robustness, we reveal that a stochastic learning algorithm can
 generalize well as long as its sensitiveness to adversarial perturbations is bounded
 in average over training examples. Moreover, an algorithm may be sensitive to
 some adversarial examples (Goodfellow et al., 2015) but still generalize well. To
 support our claims, we provide extensive simulations for different deep learning
 algorithms and different network architectures exhibiting a strong correlation between
 ensemble robustness and the ability to generalize.
answer:

Explaining the generalization of stochastic deep learning algorithms, theoretically and empirically, via ensemble robustness


