title: What's the difference between NLTK's BLEU score and SacreBLEU?
body: <p>I'm curious if anyone is familiar with the difference between using <a href="https://www.nltk.org/_modules/nltk/translate/bleu_score.html" rel="nofollow noreferrer">NLTK's BLEU score calculation</a> and the <a href="https://github.com/mjpost/sacrebleu" rel="nofollow noreferrer">SacreBLEU library</a>.</p>
  <p>In particular, I'm using both library's sentence BLEU scores, averaged over the entire dataset. The two give different results:</p>
  <pre><code>&gt;&gt;&gt; from nltk.translate import bleu_score
  &gt;&gt;&gt; from sacrebleu import sentence_bleu
  &gt;&gt;&gt; print(len(predictions))
  256
  &gt;&gt;&gt; print(len(targets))
  256
  &gt;&gt;&gt; prediction = &quot;this is the first: the world's the world's the world's the \
  ... world's the world's the world's the world's the world's the world's the world \
  ... of the world of the world'&quot;
  ...
  &gt;&gt;&gt; target = &quot;al gore: so the alliance for climate change has launched two campaigns.&quot;
  &gt;&gt;&gt; print(bleu_score.sentence_bleu([target], prediction))
  0.05422283394039736
  &gt;&gt;&gt; print(sentence_bleu(prediction, [target]).score)
  0.0
  &gt;&gt;&gt; print(sacrebleu.corpus_bleu(predictions, [targets]).score)
  0.678758518214081
  &gt;&gt;&gt; print(bleu_score.corpus_bleu([targets], [predictions]))
  0
  </code></pre>
  <p>As you can see, there's a lot of confusing inconsistencies going on. There's no way that my BLEU score is 67.8%, but it's also not supposed to be 0% (there are a lot of overlapping n-grams like &quot;the&quot;).</p>
  <p>I'd appreciate it if anyone could shed some light on this. Thanks.</p>