<SYSTEM_TASK:>
Solve the following problem using Python, implementing the functions described below, one line at a time
<END_TASK>
<USER_TASK:>
Description:
 def tokenize(self, line: str, expand: bool = True) -> List[str]: """ Lex a string into a list of tokens. Shortcuts and aliases are expanded and comments are removed :param line: the command line being lexed :param expand: If True, then aliases and shortcuts will be expanded. Set this to False if no expansion should occur because the command name is already known. Otherwise the command could be expanded if it matched an alias name. This is for cases where a do_* method was called manually (e.g do_help('alias'). :return: A list of tokens :raises ValueError if there are unclosed quotation marks. """


        # expand shortcuts and aliases
        if expand:
            line = self._expand(line)

        # check if this line is a comment
        if line.lstrip().startswith(constants.COMMENT_CHAR):
            return []

        # split on whitespace
        tokens = shlex_split(line)

        # custom lexing
        tokens = self._split_on_punctuation(tokens)
        return tokens