title: Adam optimizer goes haywire after 200k batches, training loss grows
body: <p>I've been seeing a very strange behavior when training a network, where after a couple of 100k iterations (8 to 10 hours) of learning fine, everything breaks and the training loss <em>grows</em>:</p>
  
  <p><a href="https://i.stack.imgur.com/47T4x.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/47T4x.jpg" alt="Loss explodes"></a></p>
  
  <p>The training data itself is randomized and spread across many <code>.tfrecord</code> files containing <code>1000</code> examples each, then shuffled again in the input stage and batched to <code>200</code> examples.</p>
  
  <h3>The background</h3>
  
  <p>I am designing a network that performs four different regression tasks at the same time, e.g. determining the likelihood of an object appearing in the image and simultanously determining its orientation. The network starts with a couple of convolutional layers, some with residual connections, and then branches into the four fully-connected segments.</p>
  
  <p>Since the first regression results in a probability, I'm using cross entropy for the loss, whereas the others use classical L2 distance. However, due to their nature, the probability loss is around the order of <code>0..1</code>, while the orientation losses can be much larger, say <code>0..10</code>. I already normalized both input and output values and use clipping</p>
  
  <pre><code>normalized = tf.clip_by_average_norm(inferred.sin_cos, clip_norm=2.)
  </code></pre>
  
  <p>in cases where things can get really bad.</p>
  
  <p>I've been (successfully) using the Adam optimizer to optimize on the tensor containing all distinct losses (rather than <code>reduce_sum</code>ing them), like so:</p>
  
  <pre><code>reg_loss = tf.reduce_sum(tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES))
  loss = tf.pack([loss_probability, sin_cos_mse, magnitude_mse, pos_mse, reg_loss])
  
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate,
                                     epsilon=self.params.adam_epsilon)
  op_minimize = optimizer.minimize(loss, global_step=global_step)
  </code></pre>
  
  <p>In order to display the results in TensorBoard, I then actually do</p>
  
  <pre><code>loss_sum = tf.reduce_sum(loss)
  </code></pre>
  
  <p>for a scalar summary.</p>
  
  <p>Adam is set to learning rate <code>1e-4</code> and epsilon <code>1e-4</code> (I see the same behavior with the default value for epislon and it breaks even faster when I keep the learning rate on <code>1e-3</code>). Regularization also has no influence on this one, it does this sort-of consistently at some point.</p>
  
  <p>I should also add that stopping the training and restarting from the last checkpoint - implying that the training input files are shuffled again as well - results in the same behavior. The training always seems to behave similarly at that point.</p>