title: What if we do not apply activation to the hidden layers and only to the output layer of a feed forward neural network?
body: <p>Are there any hidden gotchas here? Even popular ReLU is max(0,x) where we will allow the maximum value to pass and clip the negative as zero. What is the problem if we allow both positive and negative values? Or why are we clipping the negative values to zero in ReLU.</p>