title: Good implementation of bias units in neural network
body: <p>I am learning about neural network and in the process, I have implemented few fully connected nets. I usually add a <code>column bias units</code>(1s) in the input marix and an <code>extra row of weights</code> in weight matrix because that's how I learned to implement neural nets after taking an online course but in many implementations on github I have found that it can also be implemented without inserting bias units in the matrix but instead it can be added separately : <code>XW + b</code> , where b is bias unit . </p>
  
  <p>I don't understand how it works. It seems like a better  and more efficient implementation but I don't understand it. For instance , consider the following example:</p>
  
  <pre><code>        1 2 3 4       0.5 0.5
     X =  1 4 5 6    W= 2   3     X*W = [4x2 matrix] 
          1 8 9 5       5   8
                        2   3
  </code></pre>
  
  <p><code>The first column in X is bias unit and so is the first row in W</code> </p>
  
  <p>But if the same is written without directly inserting the bias column but by adding it separately it becomes:</p>
  
  <pre><code>       2 3 4       2 3
     X=  4 5 6    W= 5 8    b = 0.5 0.5    X*W = [3x2 matrix]
         8 9 5       2 3
  </code></pre>
  
  <p>It can be clearly seen that <code>X*W+b</code> from the second expression is not equal to first expression. And furthermore b, a <code>1x2 matrix</code> cannot be added to <code>X*W</code> which is <code>3x2 matrix</code>.</p>
  
  <p>So, how can i implement biases using the second method ?</p>