<h2>Solutions:</h2>

<h3>Make sure to preprocess your input.</h3>

<p>If your input pixels are in range [0, 255], it's better to rescale them into [0.0, 1.0]. This suffices in most situation.</p>

<p>A more advanced way would be using <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">batch normalization</a>.</p>

<h3>Make sure you initialize weight matrices in a normalized fashion.</h3>

<p>By normalized, I mean each of the 784-dimension column vector of the weight matrix should have a fixed L2-norm. For a simple setup, you could just normalize them to 1.</p>

<p>Weight matrix initialization is a research topic, for example using <a href="http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">glorot initialization</a> tend to show better results for deep networks.</p>
Training MNIST dataset with Sigmoid Activation function
<p>According to <a href="https://iamtrask.github.io/2015/07/12/basic-python-network/" rel="nofollow noreferrer">this tutorial</a>, with <code>Python</code> and <code>Numpy</code>, I want to train <code>MNIST</code> dataset to a neural network that can recognize handwritten digits. I understand the logic but i have a problem right now. </p>

<p>In this tutorial test case was <code>AND</code> logical gate and because of small amount of data, it works fine. But I'm using <code>MNIST</code> database and each image has <code>28*28</code> dimension,And when I convert each of them to a vector, I have <code>N*784</code> matrix. And if I have a <code>784*1</code> matrix as <code>weight matrix</code>, When I multiply it with the input matrix, The resulting numbers is going to be very small or very large numbers(Negative or Positive) and because i use <code>Sigmoid</code> activation function, all of my data divided in to two section, 1 and 0 at first learning cycle, But i need small numbers that converging slowly.</p>

<p>for example i get these numbers after multiplication: <code>-569.87541502</code> , <code>218.62477264</code> and the first one is 0 in <code>Sigmoid</code> activation function and the second is 1, And there is no room for training and converging. All of this because of large amount of data that summation of them resulting to this large or very small numbers.</p>

<p>I use this trick for generating very small weights than the original tutorial but i get the same results(I was thinking because these are small numbers, summation of them can't be very large but i get the same results):</p>

<pre><code>syn0 = (2*np.random.random((784,10))-1)*(0.00000005-0.00000001) + 0.00000001
</code></pre>

<p>I don't know how i can overcome to this.</p>