title: How to write a conditonal regressional loss function based on binary cross entropy loss for Keras model
body: <p>I am building a key-point detection system of the human face. The goal is to have an image of the face be input into the model, and the model then detects anatomical landmarks in the image (eyes, nose) and outputs the pixel coordinates of the landmarks that are visible. There are three targets per landmark: x, y, visible. X and Y are the pixel coordinates, and visible is whether the landmark is in the image or not. The plan is to first have a binary cross entropy loss between predicted visibility and true visibility. Then, the second loss is a regression loss (I'm using MAPE) between the x,y coordinates and the targets. However, the regression loss would only be calculated for landmarks that are visible. The loss would look something like:</p>
  <pre><code>#Pseudo-code
  def loss(y_true,y_pred):
      if y_true[2] == 1
         #Probability that landmark is in image
         #Compute binary cross entropy loss
         #Compute MAPE regression loss
         Total_loss = Binary_loss + MAPE_loss
         return Total_loss
  
      else:
         Total_loss = Binary loss
         return Total_loss
  </code></pre>
  <p>Once the loss function is written, how would I go about implementing it in code? I know how to create models for each problem (checking the coordinates, and separately checking the visibility), but I'm not sure exactly how to go about combining the two heads with the conditional loss function. How would I combine the layers (Conv, Flatten, Dense for each head) to get the desired output? Thank you!</p>
  <p>EDIT:
  I'm not able to upload the data, but here is an image of it. The first 9 columns are the coordinates, and visibility of the landmarks. The last column is the corresponding image which has been flattened.  <a href="https://i.stack.imgur.com/iu6R8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iu6R8.png" alt="enter image description here" /></a> When I load in the data for training, these are the steps I do:</p>
  <pre><code>###Read in data file
  file = &quot;Directory/file.csv&quot;
  train_data = pd.read_csv(file)
  ###Convert each coordinate column to type float64
  train_data['xreye'] = train_data['xreye'].astype(np.float64)
  ...
  ###Convert image column to string type
  train_data['Image'] = train_data['Image'].astype(str)
  
  #Image is feature, other values are labels to predict later
  #Image column values are strings, also some missing values, have to split
  ##string by space and append it and handle missing values
  imag = []
  for i in range(len(train_data)):
      img = train_data['Image'][i].split(' ')
      img = ['0' if x == '' else x for x in img]      
      imag.append(img) 
  #Reshape and convert to float value
  image_list = np.array(imag,dtype = 'uint8')
  X_train = image_list.reshape(-1,256,256,1)
  
  ####Get pixel coordinates and visibility targets
  training = train_data[['xreye','yreye','reyev','xleye','yleye','leyev','xtsept','ytsept','tseptv']]
  y_train = []
  for i in range(len(train_data)):
      y = training.iloc[i,:]
      y_train.append(y)
  
  y_train = np.array(y_train, dtype='float')
  </code></pre>
  <p>EDIT: Model code, loss function, and fit method.</p>
  <pre><code>###Loss function
  visuals_mask = [False, False, True] * 3
  def loss_func(y_true, y_pred):
      visuals_true = tf.boolean_mask(y_true, visuals_mask, axis=1)
      visuals_pred = tf.boolean_mask(y_pred, visuals_mask, axis=1)
      visuals_loss = tf.keras.losses.BinaryCrossentropy(visuals_true, visuals_pred)
      visuals_loss = tf.reduce_mean(visuals_loss)
  
      coords_true = tf.boolean_mask(y_true, ~np.array(visuals_mask), axis=1)
      coords_pred = tf.boolean_mask(y_pred, ~np.array(visuals_mask), axis=1)
      coords_loss = tf.keras.losses.MeanAbsolutePercentageError(coords_true, coords_pred)
      coords_loss = tf.reduce_mean(coords_loss)
  
      return coords_loss + visuals_loss
  ####Model code
  model = Sequential()
  
  model.add(Conv2D(32, (3,3), activation='relu', padding='same', use_bias=False, input_shape=(256,256,1)))
  model.add(BatchNormalization())
  model.add(MaxPool2D(pool_size=(2,2)))
  
  model.add(Conv2D(64, (3,3), activation='relu', padding='same', use_bias=False))
  model.add(BatchNormalization())
  model.add(MaxPool2D(pool_size=(2,2)))
  
  model.add(Conv2D(128, (3,3), activation='relu', padding='same', use_bias=False))
  model.add(BatchNormalization())
  model.add(MaxPool2D(pool_size=(2,2)))
  
  model.add(Flatten())
  
  model.add(Dense(128, activation='relu'))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(32, activation='relu'))
  model.add(Dense(16, activation='relu'))
  model.add(Dense(8, activation='relu'))
  model.add(Dense(4, activation='relu'))
  model.add(Dense(9, activation='linear'))
  model.summary()
  model.compile(optimizer='adam', loss=loss_func)
  
  ###Model fit
  checkpointer = ModelCheckpoint('C:/Users/Cloud/.spyder-py3/x_y_shift/weights/vis_coords_TEST.hdf5', monitor='val_loss', verbose=1, mode = 'min', save_best_only=True)
  out = model.fit(X_train,y_train,epochs=5,batch_size=4,validation_split=0.1, verbose=1, callbacks=[checkpointer])
  
  </code></pre>