def tokenize(content):
    # TODO maybe ignore tokens with non-latin characters? (no chinese, arabic, russian etc.)
    return [token.encode('utf8') for token in utils.tokenize(content, lower=True, errors='ignore')
            if 2 <= len(token) <= 15 and not token.startswith('_')]
Tokenize a piece of text from wikipedia. The input string `content` is assumed
dataset/ETHPy150Open largelymfs/topical_word_embeddings/TWE-1/gensim/corpora/wikicorpus.py/tokenize