title: Why my model has a high performance (mAP = 0.88) and, however, the accuracy in training does not exceed a value of 0.55?
body: <p>The task is to classify 4 species of birds from the mel spectrograms of the vocalizations. The model learns samples of the presence and absence of each species. For that a custom loss function was implemented.</p>
  <pre><code>from tensorflow import keras
  from tensorflow.keras import backend as K
  from tensorflow.keras.models import model_from_json
  from keras.preprocessing.image import ImageDataGenerator
  from tensorflow.keras.applications import ResNet50
  import tensorflow.keras.models as models
  import tensorflow.keras.layers as layers
  import tensorflow as tf
  import _pickle as pickle
  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn import metrics
  import os
  import json
  import datetime
  import pandas as pd
  from livelossplot import PlotLossesKeras
  </code></pre>
  <p>Custom loss function:</p>
  <pre><code>def masked_loss(y_true, y_pred):
      return K.mean(K.mean(K.binary_crossentropy(tf.where(tf.math.is_nan(y_true), tf.zeros_like(y_true), y_true),
                                          tf.multiply(y_pred, tf.cast(tf.logical_not(tf.math.is_nan(y_true)), tf.float32))), axis=-1))
  </code></pre>
  <p>Load the ResNet50 model (Transfer Learning):</p>
  <pre><code>ResNet50_conv = ResNet50(input_shape=input_shape, 
                           weights='imagenet', 
                           include_top=False)
  
  for layer in ResNet50_conv.layers:
      layer.trainable = True
  
  # Create the model
  model = models.Sequential()
  # Add the convolutional base model
  model.add(ResNet50_conv)
  
  model.add(layers.AveragePooling2D((7, 7)))
  
  # Add new layers
  model.add(layers.Flatten())
  model.add(layers.Dense(512, activation='relu'))
  model.add(layers.Dropout(0.5))
  model.add(layers.Dense(num_classes, activation='sigmoid'))
  
   # Compile the model
  optimizer = keras.optimizers.Adam(learning_rate=0.0001, decay=1e-7)
  model.compile(loss=masked_loss, optimizer=optimizer, metrics=['accuracy'])
  
  model.summary()
  </code></pre>
  <p>See summary:
  <a href="https://i.stack.imgur.com/9ScP8.png" rel="nofollow noreferrer">https://i.stack.imgur.com/9ScP8.png</a></p>
  <p>Training:</p>
  <pre><code>model_json = model.to_json()
  with open(model_path+'/'+model_file_name+'.json', &quot;w&quot;) as json_file:
      json_file.write(model_json)
  with open(model_path+'/'+model_file_name+'_classes.json', 'w') as f:
      json.dump(class_dict, f)
  print('Saved model architecture')
  
  my_callbacks = [tf.keras.callbacks.EarlyStopping(patience=10), PlotLossesKeras()]
  
  model_history = model.fit(train_generator,
                            steps_per_epoch = len(train_generator),
                            epochs = epochs,
                            validation_data = validation_generator,
                            validation_steps = len(validation_generator),
                            callbacks = my_callbacks,
                            verbose = 1)
  print('Saving model...')
  model.save_weights(model_path+'/'+model_file_name+'.h5') 
  </code></pre>
  <p>See epochs image:
  <a href="https://i.stack.imgur.com/jKY45.png" rel="nofollow noreferrer">https://i.stack.imgur.com/jKY45.png</a></p>
  <p>The loss function was taken from Jack LeBien et al, 2020 (see item 2.3.3):
  <a href="https://www.sciencedirect.com/science/article/pii/S1574954120300637" rel="nofollow noreferrer">https://www.sciencedirect.com/science/article/pii/S1574954120300637</a></p>
  <blockquote>
  <p>A custom training loss function was defined to leverage both true and false detection training data. Typically, for multi-label prediction, the training target vectors consist of 0's and 1's indicating the presence or absence of each class in the input. This assumes fully labeled training data (i.e. all labels are known). In our case, each training sample is only labeled for presence or absence of a single species. Therefore, each target vector consisted of a 1 or 0 at the position of the species determined to be present or absent, and unknown values represented by NaN (“Not a Number”) for all other species.</p>
  <p>The total loss for a sample is simply the loss for the single labeled species. This allows for multi-label learning based on single labels at a time, but requires examples of both presence and absence for each class.</p>
  </blockquote>
  <p>Any information is welcome to better understand this experiment.</p>