<p>Yes, they can be faster. The numbers you got are for networks trained for ImageNet classification, 1 Million images, 1000 classes. Unless your classification problem is similar, then using a ImageNet network is overkill.</p>

<p>You should also remember that these networks have weights in the order of 10-100 million, so they are quite expensive to evaluate. But you probably don't need a really big network, and you can design your own network, with less layers and parameters that is much cheaper to evaluate.</p>

<p>In my experience, I designed a network to classify 96x96 sonar image patches, and with around 4000 weights in total, it can get over 95% classification accuracy and run at 40 ms per frame on a RPi2.</p>

<p>A bigger network with 900K weights, same input size, takes 7 ms to evaluate on a Core i7. So this is surely possible, you just need to play with smaller network architectures. A good start is <a href="https://arxiv.org/abs/1602.07360" rel="nofollow noreferrer">SqueezeNet</a>, which is a network that can achieve good performance in Imagenet, but has 50 times less weights, and it is of course much faster than other networks.</p>
Can CNNs be faster than classic descriptors?
<p><strong>Disclamer:</strong> I don't know almost nothing on CNNs and I have no idea where I could ask this.</p>

<p>My research is focused on high performance on computer vision applications. We generate codes representing an image in less than 20 ms on images with the largest size of 500pxs. </p>

<p>This is done by combining SURF descriptors and <a href="http://www.vlfeat.org/overview/encodings.html" rel="nofollow noreferrer">VLAD</a> codes, obtaining a vector representing an image that will be used in our object recognition application. </p>

<p>Can CNNs be faster? According to <a href="https://github.com/soumith/convnet-benchmarks/blob/master/README.md" rel="nofollow noreferrer">this</a> benchmark (which is based on much smaller images) the times needed is longer, almost double considering that the size of the image is half of ours. </p>