<blockquote>
  <p>If I've understood correctly, I can transform a 3-Layered NN into a DL NN by adding a RelU after the hidden layer, then repeating the hidden layer + RelU</p>
</blockquote>

<p>Deep learning is pretty much a buzz-word. It can be networks with anything, from 3 learning layers up at least 16. Depends on the author / when you wanted to know what "deep" means. For example, the <a href="https://arxiv.org/abs/1512.03385" rel="nofollow noreferrer">deep residual learning paper</a> has set the bar much higher to several hundrets of layers.</p>

<p>What is important, is that you have at least one hidden layer with a non-linearity. A network with an input layer, a hidden layer and an output layer (thus two learning layers). The non-linearities (logistic function, tanh, ReLU, ...) make neural networks so powerful.</p>

<p>For the dimensions: Those are matrix multiplications / vector additions. Your input is of dimension (1, M) and gets mutliplied with a matrix of dimension (M, N). The result is of dimension (1, N). The next matrix to be multiplied has to be of dimension (N, whatever). So you have to make sure the neighboring matrices have dimensions with fit. Just calculus 101.</p>
Going from a 3-Layered NN to Deep Learning with ReLU
<p>If I've understood correctly, I can transform a <code>3-Layered NN</code> into a <code>DL NN</code> by adding a <code>RelU</code> after <code>the hidden layer</code>, then repeating the <code>hidden layer</code> + <code>RelU</code></p>

<p>I'm having trouble visualizing how the dimensionality will work out.  I now have the following from a <a href="https://github.com/autojazari/xiaonet/blob/master/XiaoNet.ipynb" rel="nofollow noreferrer">small library</a> I am putting together so I can sink in the concepts</p>

<pre><code>M = 784 # 28 x 28 pixels
N = 512 # hidden neurons
P = 10 # number of possible classes

w1 = np.random.normal(0.0, pow(10, -0.5), (M, N))
w2 = np.random.normal(0.0, pow(10, -0.5), (N, P))

b1 = np.random.normal(0.0, pow(10, -0.5), (N))
b2 = np.random.normal(0.0, pow(10, -0.5), (P))

x = Input(w1, b1)
h = Hidden(x, w2, b2)
g = Softmax(h)
cost = CrossEntropy(g) # numpy.mean(CrossEntropy) over BATCH SIZE

train_data()
</code></pre>

<p>But I want to go to </p>

<pre><code>x = Input(w1, b1)
h = Hidden(x, w2, b2)
r = ReLU(h) 
h2 = Hidden(r, ??, ??) # 1
r2 = ReLU(h2)          # 2
.. &lt;repeat 1 and 2&gt;
g = Softmax(h)
cost = CrossEntropy(g) # numpy.mean(CrossEntropy) over BATCH SIZE

train_data()
</code></pre>

<p><a href="http://sdc.autojazari.com/gtsd-nn/" rel="nofollow noreferrer">Related article I a writing about this</a></p>