<p>You might be interested in my paper <a href="https://arxiv.org/pdf/1801.07779.pdf" rel="noreferrer">The WiLI benchmark dataset for written
language identification</a>. I also benchmarked a couple of tools.</p>

<p>TL;DR:</p>

<ul>
<li>CLD-2 is pretty good and extremely fast</li>
<li><a href="https://pypi.python.org/pypi/langdetect?" rel="noreferrer">lang-detect</a> is a tiny bit better, but much slower</li>
<li>langid is good, but CLD-2 and lang-detect are much better</li>
<li>NLTK's Textcat is neither efficient nor effective.</li>
</ul>

<p>You can install <a href="https://github.com/MartinThoma/lidtk" rel="noreferrer"><code>lidtk</code></a> and classify languages:</p>

<pre><code>$ lidtk cld2 predict --text "this is some text written in English"
eng
$ lidtk cld2 predict --text "this is some more text written in English"
eng
$ lidtk cld2 predict --text "Ce n'est pas en anglais"                  
fra
</code></pre>
Determine if text is in English?
<p>I am using both <a href="http://www.nltk.org/" rel="noreferrer">Nltk</a> and <a href="http://scikit-learn.org/stable/" rel="noreferrer">Scikit Learn</a> to do some text processing. However, within my list of documents I have some documents that are not in English. For example, the following could be true:</p>

<pre><code>[ "this is some text written in English", 
  "this is some more text written in English", 
  "Ce n'est pas en anglais" ] 
</code></pre>

<p>For the purposes of my analysis, I want all sentences that are not in English to be removed as part of pre-processing. However, is there a good way to do this? I have been Googling, but cannot find anything specific that will let me recognize if strings are in English or not. Is this something that is not offered as functionality in either <code>Nltk</code> or <code>Scikit learn</code>? <b>EDIT</b> I've seen questions both like <a href="https://stackoverflow.com/questions/29099621/how-to-find-out-wether-a-word-exists-in-english-using-nltk">this</a> and <a href="https://stackoverflow.com/questions/3788870/how-to-check-if-a-word-is-an-english-word-with-python">this</a> but both are for individual words... Not a "document". Would I have to loop through every word in a sentence to check if the whole sentence is in English?</p>

<p>I'm using Python, so libraries that are in Python would be preferable, but I can switch languages if needed, just thought that Python would be the best for this.</p>