<p>There are a lot of research going on in this area. There are roughly two lines of areas that deal with this:</p>

<ul>
<li>Efficient network architectures</li>
<li>Some post-processing for an already trained model </li>
</ul>

<p>You can have a look at <a href="https://github.com/hszhao/ICNet" rel="nofollow noreferrer">ICNet</a> Figure 1, where some architectures for fast inference for semantic segmentation are shown. Many of these models can be tweaked to do classification or other image processing tasks in real time. These models all have a low number of parameters compared to other networks and can be evaluated on embedded platforms.</p>

<p>For "post-hoc" optimizations you can look at TensorFlows Graph Transform Tool that do many of such for you: <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/graph_transforms/README.md" rel="nofollow noreferrer">Graph Transform Tool</a>
 or maybe look into the paper by Song Han <a href="https://arxiv.org/abs/1510.00149" rel="nofollow noreferrer">Deep Compression</a> where many of these ideas are described. Song Han has also given many great lectures in this area and you can e.g. find one in the <a href="http://cs231n.github.io/" rel="nofollow noreferrer">CS231n</a> class at Stanford.</p>

<p>The speed of the inference phase depend on a lot of other things than the number of parameters or neurons. So I don't think there is a rule of thumb for saying how many neurons are the maximum.</p>
How computational expensive is running data through a Neural Network? Can Smartphones or Raspberry PIs do it?
<p>I have only little background knowledge about Neural Networks (NN).
However, up to know I learnt, that training the network is the actual expensive part. Processing data by an already trained network is much cheaper/faster, ultimately.</p>

<p>Still, I'm not entirely sure what the expensive parts are within the processing chain. As far as I know, it's mostly Matrix-Multiplication for standard layers. Not the cheapest operation, but definitly doable. On top, there are are other layers, like max-pooling, or activation-functions at each node, which might have higher complexities. Are those the bottle-necks?</p>

<p>Now, I wonder if "simple" Hardware provided by Smartphones or even cheap stand-alone Hardware like Raspberry PIs are capable of utilizing a (convolutional-) Neuronal Networks to do, for example, Image Processing, like Object Detection. Of course, I mean doing the calculations on the device itself, not by transmitting the data to a second, powerful machine or even a cloud, which does the calculations, before sending back the results to the smartphone. </p>

<p>If so, what are the maximum Neurons such a Network should have (e.g. how many layers and how many neurons per layer), roughly estimated. And last, are there any good either projects, or librarys, using NNs for reduced simpler Hardware?</p>