title: Loss function for class imbalanced multi-class classifier in Keras
body: <p>I am trying to apply deep learning to a multi-class classification problem with high class imbalance between target classes (10K, 500K, 90K, 30K). I want to write a custom loss function.
  This is my current model:</p>
  
  <pre><code>model = Sequential()
  
  model.add(LSTM(
                  units=10, # number of units returned by LSTM
                  return_sequences=True, 
                  input_shape=(timestamps,nb_features),
                  dropout=0.2, 
                  recurrent_dropout=0.2
                )
           )
  
  model.add(TimeDistributed(Dense(1)))
  
  model.add(Dropout(0.2))
  
  model.add(Flatten())
  
  model.add(Dense(units=nb_classes,
                 activation='softmax'))
  
  model.compile(loss="categorical_crossentropy",
                metrics = ['accuracy'],
                optimizer='adadelta')
  </code></pre>
  
  <p>Unfortunately, all predictions belong to class 1!!! The model always predicts 1 for any input...</p>
  
  <p>Appreciate any pointers on how I can solve this task.</p>
  
  <p><strong>Update:</strong></p>
  
  <p>Dimensions of input data:</p>
  
  <pre><code>94981 train sequences
  29494 test sequences
  X_train shape: (94981, 20, 18)
  X_test shape: (29494, 20, 18)
  y_train shape: (94981, 4)
  y_test shape: (29494, 4)
  </code></pre>
  
  <p>Basically in the train data I have 94981 samples. Each sample contains a sequence of 20 timestamps. There are 18 features. </p>
  
  <p>The imbalance between target classes (10K, 500K, 90K, 30K) is just an example. I have similar proportions in my real dataset.</p>
  
  <p><a href="https://i.stack.imgur.com/45uEJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/45uEJ.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.stack.imgur.com/x3VAB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/x3VAB.png" alt="enter image description here"></a></p>