title: NLTK inter-annotator agreement using Krippendorff Alpha Outputs Zero on only 1 disagreement
body: <p>I have a <a href="https://en.wikipedia.org/wiki/Sequence_labeling" rel="nofollow noreferrer">sequence-labeling</a> task in NLP, where the annotators are asked to provide 1 or more labels to each word in a sentence. For e.g., for a sentence [a, b, c, d]</p>
  <p>Ann 1. provided [[0, 1, 2], [0, 1], [0], [0]]</p>
  <p>Ann 2. provided [[0, 2], [0], [0], [0, 1]]</p>
  <p>I used the NLTK's <a href="https://www.nltk.org/_modules/nltk/metrics/agreement.html" rel="nofollow noreferrer">agreement module</a> to calculate the Krippendorff's Alpha (since this is a multi-label task). To calculate the alpha, I did</p>
  <pre><code>task_data = [
  ('coder0', 'word0', frozenset([0, 1, 2])), 
  ('coder0', 'word1', frozenset([0, 1])), 
  ('coder0', 'word2', frozenset([0])), 
  ('coder0', 'word2', frozenset([0])),
  
  ('coder1', 'word0', frozenset([0, 2])), 
  ('coder1', 'word1', frozenset([0])), 
  ('coder1', 'word2', frozenset([0])), 
  ('coder1', 'word2', frozenset([0, 1]))
  ]
  
  task = AnnotationTask(distance = masi_distance)
  task.load_array(task_data)
  print(task.alpha())
  </code></pre>
  <p>Using the above code, I received very low numbers for alpha. To investigate, I took a sample example:</p>
  <pre><code>task_data = [
  ('coder1','Item0',frozenset(['l1'])), 
  
  ('coder2','Item0',frozenset(['l1', 'l2']))
  ]
  
  task = AnnotationTask(distance = masi_distance)
  
  task.load_array(task_data)
  </code></pre>
  <p>and I get the  <code>task.alpha()</code>  as ZERO.</p>
  <p>My questions are:</p>
  <ol>
  <li>Is this the correct way of calculating agreement for a sequence labeling and multi-label task?</li>
  <li>Why for the toy example alpha is zero? I tried calculating Masi-distance and it's non-zero.</li>
  <li>Is there any other better/reliable metric for such a task?</li>
  </ol>