def _get_textual_tokens(self, data):
        token_types = []
        token_types = defaultdict(lambda: BBCodeToken.TK_DATA, token_types)
        token_types['\n'] = BBCodeToken.TK_NEWLINE
        tokens = []

        data = re.split('(\n)', data)
        for value in data:
            if value:
                tokens.append(BBCodeToken(token_types[value], None, None, value))
        return tokens
Given a list of textual data, returns a list of TK_NEWLINE or TK_DATA tokens.
dataset/ETHPy150Open ellmetha/django-precise-bbcode/precise_bbcode/bbcode/parser.py/BBCodeParser._get_textual_tokens