<p>This is a very broad question. In general, neural networks with one hidden layer, a nonlinear activation function and a sufficient number of hidden neurons are able to approximate any function with arbitrary precision. However, the error function is not convex and thus the result of the training depends on the initialization.</p>

<p>SVMs are able to approximate any function, too. They are very popular because the optimization problem has a unique solution and there might be some other reasons. But recent research has shown that neural networks like multilayer perceptrons, convolutional neural networks, deep belief neural networks, multi-column deep neural networks etc. are more efficient and achieve better results for complex applications with a huge amount of data. So it is always a trade-off as LiKao stated (no free lunch theorem) and no classifier is "perfect".</p>

<p>Here is a paper that describes the advantages of deep networks in comparison to "shallow networks" which includes Support Vector Machines: <a href="http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf">http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf</a></p>

<p>Here is a standard benchmark and a comparison of different learning algorithms: <a href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/</a></p>

<p>Here is a paper that describes a new kind of neural networks that is especially good at solving some vision problems (traffic sign recognition, ocr): <a href="http://arxiv.org/abs/1202.2745">http://arxiv.org/abs/1202.2745</a></p>
Advantages and disadvantages of neural networks
<p>Would I be right in saying a neural network are good at finding 'good enough' solutions for a problem?</p>

<p>I'm thinking this because they don't provide a binary output for an given input but a probability, for example 0.67 could be an output.</p>

<p>I'm also guessing because they're often used for generalisation they're good at find solutions that often solve the problem but in some rare cases won't.</p>

<p>Thank you!</p>