Q:

How to determine the confidence of a neural network prediction?

To illustrate my question, suppose that I have a training set where the input has a degree of noise but the output does not, for example;
# Training data
[1.02, 1.95, 2.01, 3.06] : [1.0]
[2.03, 4.11, 5.92, 8.00] : [2.0]
[10.01, 11.02, 11.96, 12.04] : [1.0]
[2.99, 6.06, 9.01, 12.10] : [3.0]

here the output is the gradient of the input array if it were noiseless (not the actual gradient).
After training the network, the output should look something like this for a given input.
# Expected Output
[1.01, 1.96, 2.00, 3.06] : 95% confidence interval of [0.97, 1.03]
[2.03, 4.11, 3.89, 3.51] : 95% confidence interval of [2.30, 4.12]

My question is how can a neural network be created such that it will return a predicted value and a measure of confidence, such as a variance or confidence interval?

A:

It sounds like you are looking for a prediction-interval, i.e., an interval that contains a prespecified percentage of future realizations. (Look at the tag wikis for prediction-interval and confidence-interval for the difference.)
Your best bet is likely to work directly with NN architectures that do not output single point predictions, but entire predictive distributions. You can then directly extract desired prediction intervals (or mean, or median point predictions) from these distributions. I and others have been arguing that predictive distributions are much more useful than point predictions, but to be honest, I have not yet seen a lot of work on predictive distributions with neural nets, although I have been keeping my eyes open. This paper sounds like it might be useful. You might want to search a bit, perhaps also using other keywords like "forecast distributions" or "predictive densities" and such.
That said, you might want to look into Michael Feindt's NeuroBayes algorithm, which uses a Bayesian approach to forecast predictive densities. 

A:

I'm not sure you can compute a confidence interval for a single prediction, but you can indeed compute a confidence interval for error rate of the whole dataset (you can generalize for accuracy and whatever other measure you are assessing).
If $e$ is your error rate while classifying some data $S$ of size $n$, a 95% confidence interval for your error rate is given by:
$$ e \pm 1.96\sqrt{\frac{e\,(1-e)}{n}}$$.
(see "Machine Learning" book from Tom Mitchell, chapter 5.)
EDIT
Guess I should state a more general case, which is:
$$ e \pm z_N\sqrt{\frac{e\,(1-e)}{n}},$$
where common choices for $z_N$ are listed in the following table:
confidence level    80%    90%    95%    98%    99%
values of zN       1.28   1.64   1.96   2.33   2.58

A:

Prediction intervals (PI) in non parametric regression & classification problems, such as neural nets, SVMs, random forests, etc. are difficult to construct. I'd love to hear other opinions on this. 
However, as far as I know, Conformal Prediction (CP) is the only principled method for building calibrated PI for prediction in nonparametric regression and classification problems. For a tutorial on CP, see Shfer & Vovk (2008), J. Machine Learning Research 9, 371-421 [pdf]

