def read_tokens(file):
    lines = file.readlines()
    return [l.rstrip('\n') for l in lines if len(l.rstrip('\n')) > 0]
Reads tokens from a file handle
dataset/ETHPy150Open learntextvis/textkit/textkit/utils.py/read_tokens