<p>There is nothing wrong. The problem is that increasing layers does not automatically means a higher accuracy (otherwise machine learning would be kind of solved, because if you need a better accuracy in an image classifier you would just add +1 layer to an inception and claim a victory).</p>

<p>To show you that this is not only your problem - take a look at this high-level paper: <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="noreferrer">Deep Residual Learning for Image Recognition</a> where they see that increasing the number of layers decreases the scoring function (which is not important) and their architecture to overcome this problem (which is important). Here is a small part from it:<a href="https://i.stack.imgur.com/msoTt.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/msoTt.jpg" alt="enter image description here"></a></p>

<blockquote>
  <p>The deeper network has higher training error and thus test error.</p>
</blockquote>
Two layer neural network performs worse than single layer
<p>I'm learning TensorFlow, and trying to create a simple two layer neural network.</p>

<p>The tutorial code <a href="https://www.tensorflow.org/get_started/mnist/pros" rel="nofollow noreferrer">https://www.tensorflow.org/get_started/mnist/pros</a> starts with this simple network, to get 92% accuracy:</p>

<pre><code>W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
</code></pre>

<p>I tried replacing it with this very simple network, adding a new layer, <strong>but accuracy now drops to 84%</strong>!!!</p>

<pre><code>layer1_len = 10
w1 = weight_var([784, layer1_len])
b1 = bias_var([layer1_len])
o1 = tf.nn.relu(tf.matmul(x, w1) + b1)
w2 = weight_var([layer1_len, 10])
b2 = bias_var([10])
y = tf.nn.softmax(tf.matmul(o1, w2) + b2)
</code></pre>

<p>I get that result with several different values for <code>layer1_len</code> as well as different numbers of training steps.  (Note that if I omit the <code>weight_var</code> and <code>bias_var</code> random initialization, and keep everything at zero, accuracy drops to close to 10%, essentially no better than guessing.)</p>

<p><strong>What am I doing wrong?</strong></p>