title: Why does the loss of a CNN decrease for a long time and then suddenly increase?
body: <p>I made a simple network to find broken lines and I had a very strange training run. The loss, keras.losses.binary_crossentropy, was decreasing steadily for around 1500 epochs then suddenly, it shot up and plateaued.</p>
  
  <p><a href="https://i.stack.imgur.com/Vv5PI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Vv5PI.png" alt="graph of loss function vs epoch"></a></p>
  
  <p>What are some reasons this happens? Optimizers, loss function, network structure? </p>
  
  <p>I checked the weights, and none of the weights have a NaN value. The input data is 250,000+ 32x32 images with lines on them, and the same stack of images where the lines have a few pixels removed from so they're "broken".</p>
  
  <p><a href="https://i.stack.imgur.com/4e0Bq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4e0Bq.png" alt="sample solid lines and broken lines"></a></p>
  
  <p>Here is the model creation code:</p>
  
  <pre><code>input_shape = (1, 32, 32)
  kernel_shape = (16, 16)
  keras.backend.set_image_data_format("channels_first")
  n_filters = 64
  input_layer = engine.Input(input_shape)
  active_1 = layers.Activation("relu")(input_layer)
  conv_1 = layers.Conv2D(n_filters, kernel_shape)(active_1)
  conv_2 = layers.Conv2D(2*n_filters, kernel_shape)(conv_1)
  pool_1 = layers.MaxPooling2D()(conv_2)
  
  s = tupleFromShape(pool_1.shape)
  p = 1
  for d in s:
      p *= d
  
  shaped_1 = layers.Reshape((p,))(pool_1)
  dense_1 = layers.Dense(2)(shaped_1)
  out = layers.Activation("softmax")(dense_1)
  model = engine.Model(input_layer, out)
  model.save("broken-lines-start.h5")
  </code></pre>
  
  <p>And the training code:</p>
  
  <pre><code>full = #numpy array (c, slices, 32, 32)
  broken = #numpy array(c, slices, 32, 32)
  full = full[0]
  broken = broken[0]
  
  n = len(full) - 1024
  n2 = len(broken) - 1024
  
  random.shuffle(full)
  random.shuffle(broken)
  
  optimizer = keras.optimizers.Adam(0.00001)
  loss_function = keras.losses.binary_crossentropy
  model.compile(
          model, 
          optimizer, 
          loss_function=loss_function)
  batch_size = 256
  steps = n//batch_size + n2//batch_size
  model.fit_generator(generator=getDataGenerator(full[:n], broken[:n2], batch_size),
                  steps_per_epoch=steps,
                  epochs=4680,
                  validation_data=getDataGenerator(full[n:], broken[n2:], batch_size),
                  validation_steps=2048//batch_size,
                  callbacks=[saves_last_epoch_and_best_epoch]
                      )
  model.save("broken-lines-trained.h5")
  </code></pre>
  
  <p>The generator code:</p>
  
  <pre><code>def getDataGenerator(solid, broken, batch_size=128):
      zed = [([chunk], [1, 0]) for chunk in solid] + [([chunk], [0, 1]) for chunk in broken]
      random.shuffle(zed)
      xbatch = []
      ybatch = []
      while True:
          for i in range(len(zed)):
              x,y = zed[i]
              xbatch.append(x)
              ybatch.append(y)
              if len(xbatch)==batch_size:
                  yield numpy.array(xbatch),numpy.array(ybatch)
                  xbatch = []
                  ybatch = []
  </code></pre>
  
  <p>I have greatly improved this model, and it hasn't exhibited this behavior yet, but I would like to understand why this happened. </p>
  
  <p>Subsequent things I have tried:</p>
  
  <p>Change the loss function to logcosh -> works</p>
  
  <p>Change the epsilon value of the adam optimizer -> still blows up.</p>
  
  <p>Change the optimizer to SGD -> blows up faster, didn't have initial decrease.</p>