<p>In a couple of experiments in <a href="https://arxiv.org/pdf/1707.09725.pdf#page=73" rel="noreferrer">my masters thesis</a> (e.g. page 59), I found that the bias might be important for the first layer(s), but especially at the fully connected layers at the end it seems not to play a big role.</p>

<p>This might be highly dependent on the network architecture / dataset.</p>
What is the role of the bias in neural networks?
<p>I'm aware of the gradient descent and the back-propagation algorithm. What I don't get is: when is using a bias important and how do you use it?</p>
<p>For example, when mapping the <code>AND</code> function, when I use two inputs and one output, it does not give the correct weights. However, when I use three inputs (one of which is a bias), it gives the correct weights.</p>