title: Calculate F1-score in a Named Entity Recognition model with sklearn
body: <p>I'm trying to validate an ML model that performs Named Entity Recognition on some data.
  My question is related to the way F1-Score is calculated. Im using classification_report from sklearn</p>
  <pre><code>print(preds[:2],labels[:2])
  
  #output: [[5, 5, 5, 5, 5, 5, 5, 5, 1],[6, 2]]  [[1, 5, 5, 5, 5, 5, 5, 5, 5, 5],[6, 2]]
  </code></pre>
  <p>Here we have the true labels Ids and the predicted Ids. As we can see, the first label differs, i.e., in the prediction it has value &quot;1&quot;, however, in the labels it has value &quot;5&quot;. The way I see it, it means that the token was wrongly classified.
  Next, in order to use the sklearning metrics, we have to convert our arrays with fit_transform from MultiLabelBinarizer, since we have more than 2 labels.</p>
  <pre><code>transformed_labels = MultiLabelBinarizer().fit_transform(labels)
  transformed_preds = MultiLabelBinarizer().fit_transform(preds)
  print(transformed_preds [:2],transformed_labels [:2])
  
  #output: [[1 0 0 1 0],[0 1 0 0 1]]  [[1 0 0 1 0],[0 1 0 0 1]] 
  </code></pre>
  <p>This is the part I don't understand. This method only saves which entities were used in each sequence, but doesn't care about any order or quantity of labels.</p>
  <pre><code>labels = ['Date','Place','Org','Person','Event']
      print(classification_report(transformed_labels ,transformed_preds ,target_names=labels))
  
                precision    recall  f1-score   support
  
          Date       0.69      0.92      0.82       122
         Place       0.90      0.94      0.93       195
           Org       0.76      0.85      0.78        79
        Person       0.99      0.98      0.98       434
         Event       0.81      0.69      0.73        55
  </code></pre>
  <p>In the end, the metrics are quite high, but I don't think they are accurate since they were calculated only based on if the predicted sequence had at least one occurrence of each entity present in the true sequence.</p>
  <p>Im i reading it wrongly?</p>
  <p>Another approach i saw:</p>
  <pre><code>def calc_precision(pred, true):
      precision = len([x for x in pred if x in true]) / (len(pred) + 1e-20) # true positives / total pred
      return precision
  </code></pre>
  <p>Here, we are calculating the precision of the pred list against the true list.</p>
  <p>For that, the function only checks if the predicted labels are in the true labels list. Again, the number of occurrences and order is not taken into account.</p>
  <p>What would be the correct way to calculate the F1-score in NER?</p>