Systems and methods for training a neural network or an ensemble of neural networks are described. A hyper-parameter that controls the variance of the ensemble predictors is used to address overfitting. For larger values of the hyper-parameter, the predictions from the ensemble have more variance, so there is less overfitting. This technique can be applied to ensemble learning with various cost functions, structures and parameter sharing. A cost function is provided and a set of techniques for learning are described.
Please help me write a proper abstract based on the patent claims. CLAIM: 1. A computer-implemented method for use in training a plurality of neural networks, the method comprising: a. obtaining at least one training data item; b. computing outputs from one or more members within the plurality of neural networks for the at least one training data item; c. combining one or more of the outputs to form an aggregate output; d. selecting at least one neural network from the plurality of neural networks; e. computing a variance-adjusted output for each selected neural network by summing the aggregate output with a fixed multiple (a) of a difference between the output of the selected neural network and the aggregate output for the at least one training data item; f. computing a difference between the variance-adjusted outputs and a desired output for the training data item; and g. adjusting at least one parameter of the plurality of neural networks to reduce the difference between the variance-adjusted outputs and the desired output. 2. The method of claim 1, wherein the plurality of neural networks comprise one of linear regression models, logistic regression models, neural network models with at least one layer of hidden units, or a combination thereof. 3. The method of claim 1, wherein combining the one or more outputs comprises combining the outputs from a selected subset of neural networks within the plurality of neural networks, the selected subset being selected randomly, pseudorandomly or using a fixed pattern. 4. The method of claim 3, wherein combining the one or more outputs comprises averaging the outputs of the selected subset of neural networks. 5. The method of claim 1, wherein the difference between the variance-adjusted output and the desired output is computed using one of: squared error, absolute error, log-likelihood or cross-entropy. 6. The method of claim 1, wherein the variance-adjusted output is further processed by an additional neural network for generating an additional output for comparison with the desired output. 7. The method of claim 1, wherein the at least one parameter of the plurality of neural is adjusted using the gradient or the Hessian of a log-likelihood function or a squared error function, evaluated using the desired output and the variance-adjusted output. 8. The method of claim 7, wherein the gradient for each parameter for each neural network of the plurality of neural networks is computed from an average of the gradients across the plurality of neural networks. 9. The method of claim 7, wherein the at least one parameter is adjusted using one of: gradient descent, stochastic gradient descent, momentum, Nesterov's accelerated momentum, AdaGrad, RMSProp, conjugate gradient, or a combination thereof. 10. The method of claim 1, further comprising: a. obtaining at least one test data item; b. computing test outputs from one or more members within the plurality of neural networks for the at least one test data item; and c. combining one or more of the test outputs to form an aggregate output, by averaging the test outputs or by using majority voting. 11. The method of claim 1, wherein a is selected by: a. for a plurality of values of the parameter a, computing further outputs by the plurality of neural networks for a held out validation set of data items; b. computing a validation error or a validation log-likelihood for each of the further outputs; and c. selecting the value of a from the plurality of values for which the lowest error or the highest log-likelihood is computed. 12. The method of claim 11, wherein a and at least one other parameter are selected by applying a search technique to determine a jointly optimal hyper-parameter setting, the search technique comprising one of a grid search, random search, Bayesian hyper-parameter optimization, or a combination thereof. 13. A computer-implemented method for use in training a neural network, the method comprising: a. obtaining at least one training data item; b. computing a plurality of training outputs by repeatedly applying the neural network to the at least one training data item while disabling at least one of the hidden units or input units randomly with the predetermined probability, or pseudo-randomly with the predetermined probability, or using a predetermined set of binary masks that use the predetermined probability and where each mask is used only once, or according to a fixed pattern that uses the predetermined probability; c. computing aggregate training outputs for the at least one training data item, by averaging of the plurality of training outputs or by majority voting of the plurality of training outputs; d. for each training output, computing a variance-adjusted training output by summing the aggregate training output with a fixed number (a) times a difference between the training output and the aggregate training output; and e. adjusting at least one parameter of the neural network to reduce the difference between the variance-adjusted training outputs and a desired training output for the at least one training data item. 14. The method of claim 13, further comprising that for at least one of the inputs or hidden units that is not disabled, the value of the input or hidden unit is scaled by one divided by one minus the predetermined dropout probability. 15. The method of claim 13, further comprising: a. scaling the at least one parameter by one minus the predetermined probability; b. obtaining at least one test data item; and c. computing test outputs for the at least one test data item by applying the neural network to the at least one test data item. 16. The method of claim 13, further comprising: a. obtaining at least one test data item; b. computing a plurality of test outputs by repeatedly applying the neural network to the at least one test data item while disabling at least one of the hidden units or input units randomly with the predetermined probability, or pseudo-randomly with the predetermined probability, or using a predetermined set of binary masks that use the predetermined probability and where each mask is used only once, or according to a fixed pattern that uses the probability; and c. computing aggregate test outputs for the at least one test data item, by averaging of the plurality of test outputs or by majority voting of the plurality of test outputs. 17. The method of claim 13, wherein combining the plurality of training outputs to form an aggregate training output comprises averaging or majority voting of the plurality of outputs. 18. The method of claim 13, further comprising training an additional neural network, the method comprising: a. computing additional training outputs by applying an additional neural network that has as inputs the plurality of training outputs; and b. adjusting the at least one parameter of the additional neural network to reduce the error between the additional training outputs and the desired output for the at least one training data item. 19. The method of claim 13 wherein the parameter a is selected by: a. for a plurality of values of the parameter a, computing further outputs by the neural network for a held out validation set of data items; b. computing the sum of the squared errors for the further outputs for the validation data items; and c. selecting the value of a from the plurality of values for which the lowest error is computed. 20. The method of claim 19, wherein a and at least one other parameter are selected by applying a search technique to determine a jointly optimal hyper-parameter setting, the search technique comprising one of a grid search, random search, Bayesian hyper-parameter optimization, or a combination thereof.