title: Early stopping based on AUC
body: <p>I am fairly new to ML and am currently implementing a simple 3D CNN in python using tensorflow and keras. I want to optimize based on the AUC and would also like to use early stopping/save the best network in terms of AUC score. I have been using tensorflow's AUC function for this as shown below, and it works well for the training. However, the hdf5 file is not saved (despite the checkpoint save_best_only=True) and hence I cannot get the best weights for the evaluation.</p>
  <p>Here are the relevant lines of code:</p>
  <pre><code>model.compile(loss='binary_crossentropy',
                optimizer=keras.optimizers.Adam(lr=lr),
                metrics=[tf.keras.metrics.AUC()]) 
  
  model.load_weights(path_weights)
  
  filepath = mypath
  
  check = tf.keras.callbacks.ModelCheckpoint(filepath, monitor=tf.keras.metrics.AUC(), save_best_only=True,
                                             mode='auto')
  
  earlyStopping = tf.keras.callbacks.EarlyStopping(monitor=tf.keras.metrics.AUC(), patience=hyperparams['pat'],mode='auto') 
  
  history = model.fit(X_trn, y_trn,
                          batch_size=bs,
                          epochs=n_epochs,
                          verbose=1,
                          callbacks=[check, earlyStopping],
                          validation_data=(X_val, y_val),
                          shuffle=True)
  </code></pre>
  <p>Interestingly, if I only change monitor='val_loss' in the early stopping and checkpoint (not the 'metrics' in model.compile), the hdf5 file is saved but obviously gives the best result in terms of validation loss. I have also tried using mode='max' but the problem is the same.
  I would very much appreciate your advise, or any other constructive ideas how to work around this problem.</p>