title: Using sample_weight in Keras for sequence labelling
body: <p>I am working on a sequential labeling problem with unbalanced classes and I would like to use <code>sample_weight</code> to resolve the unbalance issue. Basically if I train the model for about 10 epochs, I get great results. If I train for more epochs, <code>val_loss</code> keeps dropping, but I get worse results. I'm guessing the model just detects more of the dominant class to the detriment of the smaller classes. </p>
  
  <p>The model has two inputs, for word embeddings and character embeddings, and the input is one of 7 possible classes from 0 to 6.</p>
  
  <p>With the padding, the shape of my input layer for word embeddings is <code>(3000, 150)</code> and the input layer for word embeddings is <code>(3000, 150, 15)</code>. I use a 0.3 split for testing and training data, which means <code>X_train</code> for word embeddings is <code>(2000, 150)</code> and <code>(2000, 150, 15)</code> for char embeddings. <code>y</code> contains the correct class for each word, encoded in a one-hot vector of dimension 7, so its shape is <code>(3000, 150, 7)</code>. <code>y</code> is likewise split into a training and testing set. Each input is then fed into a Bidirectional LSTM. </p>
  
  <p>The output is a matrix with one of the 7 categories assigned for each word of the 2000 training samples, so the size is <code>(2000, 150, 7)</code>.</p>
  
  <hr>
  
  <p>At first, I simply tried to define <code>sample_weight</code> as an <code>np.array</code> of length 7 containing the weights for each class:</p>
  
  <pre><code>count = [list(array).index(1) for arrays in y for array in arrays]
  count = dict(Counter(count))
  count[0] = 0
  total = sum([count[key] for key in count])
  count = {k: count[key] / total for key in count}
  category_weights = np.zeros(7)
  for f in count:
      category_weights[f] = count[f]
  </code></pre>
  
  <p>But I get the following error <code>ValueError: Found a sample_weight array with shape (7,) for an input with shape (2000, 150, 7). sample_weight cannot be broadcast.</code></p>
  
  <p>Looking at the docs, it looks like I should instead be passing <code>a 2D array with shape (samples, sequence_length)</code>. So I create a <code>(3000, 150)</code> array with a concatenation of the weights of every word of each sequence:</p>
  
  <pre><code>weights = []
  
  for sample in y:
      current_weight = []
      for line in sample:
          current_weight.append(frequency[list(line).index(1)])
      weights.append(current_weight)
  
  weights = np.array(weights)
  </code></pre>
  
  <p>and pass that to the fit function through the <code>sample_weight</code> parameter after having added the <code>sample_weight_mode="temporal"</code> option in <code>compile()</code>.</p>
  
  <p>I first got an error telling me the dimension was wrong, however after generating the weights for only the training sample, I end up with a <code>(2000, 150)</code> array that I can use to fit my model. </p>
  
  <hr>
  
  <ul>
  <li>Is this a proper way to define sample_weights or am I doing it all wrong ? I can't say I've noticed any improvements from adding the weights, so I must have missed something.</li>
  </ul>