

def transform(self, texts):
    if self.lowercase:
        texts = [text.lower() for text in texts]
    if self.character:
        texts = [list(text) for text in texts]
    else:
        texts = [tokenize(token) for text in texts]
    tokens = [[self.encoder.get(token, 2) for token in text] for text in texts]
    return tokens

dataset/ETHPy150Open IndicoDataSolutions/Foxhound/foxhound/preprocessing.py Tokenizer.transform/VarMisuse@64/68 `text`->`token`