<p>When the Facebook engineers have been asked similar questions in their Github repository issues, they've usually pointed to <a href="https://github.com/facebookresearch/fastText/issues/807#issuecomment-495552561" rel="nofollow noreferrer">one</a> or <a href="https://github.com/facebookresearch/fastText/issues/401#issuecomment-368935887" rel="nofollow noreferrer">the</a> other of two shell scripts in their public code (&amp; especially the 'normalize_text' functions within).</p>
<p><a href="https://github.com/facebookresearch/fastText/blob/master/tests/fetch_test_data.sh#L20" rel="nofollow noreferrer">https://github.com/facebookresearch/fastText/blob/master/tests/fetch_test_data.sh#L20</a></p>
<pre><code>normalize_text() {
  tr '[:upper:]' '[:lower:]' | sed -e 's/^/__label__/g' | \
    sed -e &quot;s/'/ ' /g&quot; -e 's/&quot;//g' -e 's/\./ \. /g' -e 's/&lt;br \/&gt;/ /g' \
        -e 's/,/ , /g' -e 's/(/ ( /g' -e 's/)/ ) /g' -e 's/\!/ \! /g' \
        -e 's/\?/ \? /g' -e 's/\;/ /g' -e 's/\:/ /g' | tr -s &quot; &quot; | myshuf
}
</code></pre>
<p><a href="https://github.com/facebookresearch/fastText/blob/master/get-wikimedia.sh#L12" rel="nofollow noreferrer">https://github.com/facebookresearch/fastText/blob/master/get-wikimedia.sh#L12</a></p>
<pre><code>normalize_text() {
    sed -e &quot;s/’/'/g&quot; -e &quot;s/′/'/g&quot; -e &quot;s/''/ /g&quot; -e &quot;s/'/ ' /g&quot; -e &quot;s/“/\&quot;/g&quot; -e &quot;s/”/\&quot;/g&quot; \
        -e 's/&quot;/ &quot; /g' -e 's/\./ \. /g' -e 's/&lt;br \/&gt;/ /g' -e 's/, / , /g' -e 's/(/ ( /g' -e 's/)/ ) /g' -e 's/\!/ \! /g' \
        -e 's/\?/ \? /g' -e 's/\;/ /g' -e 's/\:/ /g' -e 's/-/ - /g' -e 's/=/ /g' -e 's/=/ /g' -e 's/*/ /g' -e 's/|/ /g' \
        -e 's/«/ /g' | tr 0-9 &quot; &quot;
}
</code></pre>
<p>They've also referenced <a href="https://fasttext.cc/docs/en/crawl-vectors.html#tokenization" rel="nofollow noreferrer">this page's section on 'Tokenization'</a> (which names some libraries), and the <a href="https://arxiv.org/abs/1802.06893" rel="nofollow noreferrer">academic paper which describes the earlier work making individual language vectors</a>.</p>
<p>None of these are guaranteed to exactly match what was used to create their pretrained classification models, &amp; it's a bit frustrating that each release of such models doesn't contain the <em>exact</em> code to reproduce. But, these sources seem to be as much detail as is available, without getting direct answers/help from the team that created them.</p>
Text preprocessing for fasttext pretrained models
<p>I want to use pretreained fastext model for language detection: <a href="https://fasttext.cc/docs/en/language-identification.html" rel="nofollow noreferrer">https://fasttext.cc/docs/en/language-identification.html</a> . Where can I find <strong>the exact Python code</strong> for text preprocessing used for training this specific model? I am not interested in general answers about how should we prepare text for using models - I ma looking for identical transformations as those used for training.</p>