title: Multi Label Annotator agreement with Cohen Kappa
body: <p>Say that I want to have annotations for documents. Every document can be annotated with multiple labels. In this example, I have 2 annotators (a and b), and they each label two documents.</p>
  
  <pre><code>from sklearn.metrics import cohen_kappa_score
  annotator_a = [ 
      ["a","b","c"],
      ["d","e"]
  ]
  annotator_b = [
      ["b","c"],
      ["f"]
  ]
  </code></pre>
  
  <p>Annotator_a labels document 1 with labels a, b and c. Annotator_b labels documents 1 with labels b and c. </p>
  
  <p>I tried to calculate annotator agreement using:</p>
  
  <pre><code>cohen_kappa_score(annotator_a, annotator_b)
  </code></pre>
  
  <p>But this results in an error:</p>
  
  <pre><code>ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead.
  </code></pre>
  
  <p>Any ideas on how I can calculate annotator agreement on this set?</p>