<p>Have a look at <a href="https://arxiv.org/pdf/1707.09725.pdf#page=11" rel="nofollow noreferrer">my masters thesis, chapter 3</a>.</p>

<p>In general, there are no strict rules to follow when it comes to the nets architecture. There is a lot of experience in it. Exceptions are the input layer (nr. of features = nr. of neurons) and the output layer (in classification: nr of classes = nr of neurons).</p>

<p>However, there seem to be several trends / rules of thumb:</p>

<ul>
<li>For fully connected layers, use not "too few" neurons, but not more than about 3 times the last layer</li>
<li>If you have CNNs, dropout is REALLY important. Then you can have many layers / neurons and hope that dropout prevents overfitting</li>
<li><strong>Automatic topology creation</strong>: I haven't seen any of them in use often.

<ul>
<li><strong>Growing approaches</strong>: There are strategies like Cascade Correlation / Meiosis networks to start with a small network and make it bigger.</li>
<li><strong>Pruning approaches</strong>: There are strategies like Optimal Brain Damage / Optimal Brain Surgeon to start from a big network and make it smaller.</li>
<li><strong>Genetic approaches</strong>: NEAT (NeuroEvolution of Augmented Topologies)</li>
</ul></li>
<li>Bottleneck-layers are used when you want to use huge amounts of unlabeled data in an unsupervised fashion with (denoising) auto-encoders. I have seen that a couple of times.</li>
</ul>

<p>You might be interested in reading the <a href="http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf" rel="nofollow noreferrer">AlexNet</a> and the <a href="http://arxiv.org/abs/1409.4842" rel="nofollow noreferrer">GoogLeNet</a> papers</p>
About Dr. Hinton's architecture (784*500*500*2000*10) for MNIST
<p>I was doing a literature review of deep learning, recently.  Hinton in his papers <a href="http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf" rel="nofollow">http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf</a> <a href="http://www.cs.toronto.edu/~hinton/science.pdf" rel="nofollow">http://www.cs.toronto.edu/~hinton/science.pdf</a> uses a 784*500*500*2000*10 sized network for demonstrating RBM based pretraining  + finetuning using BP on MNIST dataset
Is there any specific reason we choose same number of hidden units(500) in subsequent hidden layers and increased number(2000) in the last layer? In general how to choose hidden layers/units for RBM depending on dataset (from practical experience other than Hinton's RBM manual).</p>

<p>This was a brain teasing question to me for a long time. I would be grateful for the answer.</p>