title: Neural Network - Input Normalization
body: <p>It is a common practice to normalize input values (to a neural network) to speed up the learning process, especially if features have very large scales.</p>
  
  <p>In its theory, normalization is easy to understand. But I wonder how this is done if the training data set is very large, say for 1 million training examples..? If # features per training example is large as well (say, 100 features per training example), 2 problems pop up all of a sudden:
  - It will take some time to normalize all training samples
  - Normalized training examples need to be saved somewhere, so that we need to double the necessary disk space (especially if we do not want to overwrite the original data).</p>
  
  <p>How is input normalization solved in practice, especially if the data set is very large?</p>
  
  <p>One option maybe is to normalize inputs dynamically in the memory per mini batch while training.. But normalization results will then be changing from one mini batch to another. Would it be tolerable then?</p>
  
  <p>There is maybe someone in this platform having hands on experience on this question. I would really appreciate if you could share your experiences.</p>
  
  <p>Thank you in advance.</p>