title: NLTK inter-annotator agreement using Krippendorff Alpha
body: <p>I am trying to compute inter-annotator agreement on a toy example using NLTK's <code>nltk.metrics.agreement</code> module. </p>
  
  <p>Specifically I am trying to compute agreement using the <code>alpha</code> metric (<a href="http://repository.upenn.edu/cgi/viewcontent.cgi?article=1043&amp;context=asc_papers" rel="nofollow noreferrer">Krippendorff</a>) using two different distance metrics(<code>binary_distance</code> and <code>interval_distance</code>). </p>
  
  <p>The expected result of toy example 1 below, which has near total agreement (only one pair disagrees), is a value close to <code>1</code>. However, in both cases res is <code>0.0</code>. Why? </p>
  
  <p>I understand that Krippendorff's alpha is designed for intervals rather than binary-like two-category labels. I would however not expect a zero agreement value back from the module. For the background, the toy example is simply a specific subset of a larger dataset containing annotation scores in the range [1,4]. The subset belongs to a particular population within that dataset. </p>
  
  <p>In toy example 2 things start to look better for the interval alpha. Binary alpha should probably raise an exception given that there are now three labels in the data.</p>
  
  <p><strong>Toy Example 1</strong></p>
  
  <pre><code>from nltk.metrics.agreement import AnnotationTask
  from nltk.metrics import interval_distance, binary_distance 
  
   annotation_triples = [('coder_1', '1', 4), 
                         ('coder_2', '1', 4), 
                         ('coder_1', '2', 4),
                         ('coder_2', '2', 4), 
                         ('coder_1', '3', 4), 
                         ('coder_2', '3', 4),
                         ('coder_1', '4', 4), 
                         ('coder_2', '4', 3)]
  
   t = AnnotationTask(annotation_triples, distance=binary_distance)
   result = t.alpha()
  
   t = AnnotationTask(annotation_triples, distance=interval_distance)
   result = t.alpha()
  
  result binary: 0.0
  result interval: 0.0
  </code></pre>
  
  <p><strong>Toy Example 2</strong> (replaced first pair using <code>1</code> instead of <code>4</code>)</p>
  
  <pre><code>annotation_triples = [('coder_1', '1', 1), 
                        ('coder_2', '1', 1),  
                        ('coder_1', '2', 4),
                        ('coder_2', '2', 4), 
                        ('coder_1', '3', 4), 
                        ('coder_2', '3', 4),
                        ('coder_1', '4', 4), 
                        ('coder_2', '4', 3)]
  
  result binary: 0.59
  result interval: 0.93
  </code></pre>