<SYSTEM_TASK:>
Solve the following problem using Python, implementing the functions described below, one line at a time
<END_TASK>
<USER_TASK:>
Description:
 def tokenize(text, lang, include_punctuation=False, external_wordlist=False):
 """ Tokenize this text in a way that's relatively simple but appropriate for the language. Strings that are looked up in wordfreq will be run through this function first, so that they can be expected to match the data. The text will be run through a number of pre-processing steps that vary by language; see the docstring of `wordfreq.preprocess.preprocess_text`. If `include_punctuation` is True, punctuation will be included as separate tokens. Otherwise, punctuation will be omitted in the output. CJK scripts In the CJK languages, word boundaries can't usually be identified by a regular expression. Instead, there needs to be some language-specific handling. In Chinese, we use the Jieba tokenizer, with a custom word list to match the words whose frequencies we can look up. In Japanese and Korean, we use the MeCab tokenizer. The `external_wordlist` option only affects Chinese tokenization. If it's True, then wordfreq will not use its own Chinese wordlist for tokenization. Instead, it will use the large wordlist packaged with the Jieba tokenizer, and it will leave Traditional Chinese characters as is. This will probably give more accurate tokenization, but the resulting tokens won't necessarily have word frequencies that can be looked up. If you end up seeing tokens that are entire phrases or sentences glued together, that probably means you passed in CJK text with the wrong language code. """

    # Use globals to load CJK tokenizers on demand, so that we can still run
    # in environments that lack the CJK dependencies
    global _mecab_tokenize, _jieba_tokenize

    language = langcodes.get(lang)
    info = get_language_info(language)
    text = preprocess_text(text, language)

    if info['tokenizer'] == 'mecab':
        from wordfreq.mecab import mecab_tokenize as _mecab_tokenize
        # Get just the language code out of the Language object, so we can
        # use it to select a MeCab dictionary
        tokens = _mecab_tokenize(text, language.language)
        if not include_punctuation:
            tokens = [token for token in tokens if not PUNCT_RE.match(token)]
    elif info['tokenizer'] == 'jieba':
        from wordfreq.chinese import jieba_tokenize as _jieba_tokenize
        tokens = _jieba_tokenize(text, external_wordlist=external_wordlist)
        if not include_punctuation:
            tokens = [token for token in tokens if not PUNCT_RE.match(token)]
    else:
        # This is the default case where we use the regex tokenizer. First
        # let's complain a bit if we ended up here because we don't have an
        # appropriate tokenizer.
        if info['tokenizer'] != 'regex' and lang not in _WARNED_LANGUAGES:
            logger.warning(
                "The language '{}' is in the '{}' script, which we don't "
                "have a tokenizer for. The results will be bad."
                .format(lang, info['script'])
            )
            _WARNED_LANGUAGES.add(lang)
        tokens = simple_tokenize(text, include_punctuation=include_punctuation)

    return tokens