

def tokenize(text):
    "\n    Tokenize a text into numbers and strings.\n\n    :param text: The text to tokenize (a string).\n    :returns: A list of strings and/or numbers.\n\n    This function is used to implement robust tokenization of user input in\n    functions like :func:`.parse_size()` and :func:`.parse_timespan()`. It\n    automatically coerces integer and floating point numbers, ignores\n    whitespace and knows how to separate numbers from strings even without\n    whitespace. Some examples to make this more concrete:\n\n    >>> from humanfriendly.text import tokenize\n    >>> tokenize('42')\n    [42]\n    >>> tokenize('42MB')\n    [42, 'MB']\n    >>> tokenize('42.5MB')\n    [42.5, 'MB']\n    >>> tokenize('42.5 MB')\n    [42.5, 'MB']\n    "
    tokenized_input = []
    for token in re.split('(\\d+(?:\\.\\d+)?)', text):
        token = token.strip()
        if re.match('\\d+\\.\\d+', token):
            tokenized_input.append(float(token))
        elif token.isdigit():
            text.append(int(token))
        elif token:
            tokenized_input.append(token)
    return tokenized_input

dataset/ETHPy150Open xolox/python-humanfriendly/humanfriendly/text.py tokenize/VarMisuse@71/384 `tokenized_input`->`text`