original_docstring: 
          Tokenize a list of documents.
          :param data: a list of documents to process
          :param ngram_range: range for producing ngrams, ex. for unigrams + bigrams should be set to
          (1, 2), for bigrams only should be set to (2, 2)
          :param lowercase: whether to perform lowercasing or not
          :return: a single processed doc generator
          
docstring: Tokenize a list of documents.
docstring_tokens: Tokenize a list of documents .
code: def tokenize(self, data: List[str], ngram_range=(1, 1), lowercase=True) -> \
              Generator[List[str], Any, None]:
          """
          Tokenize a list of documents.
          :param data: a list of documents to process
          :param ngram_range: range for producing ngrams, ex. for unigrams + bigrams should be set to
          (1, 2), for bigrams only should be set to (2, 2)
          :param lowercase: whether to perform lowercasing or not
          :return: a single processed doc generator
          """
          size = len(data)
  
          _ngram_range = self.ngram_range or ngram_range
          _lowercase = self.lowercase or lowercase
  
          for i, doc in enumerate(data):
              logger.debug("Tokenize doc {} from {}".format(i, size))
              if _lowercase:
                  tokens = doc.lower().split()
              else:
                  tokens = doc.split()
              filtered = self._filter(tokens)
              processed_doc = ngramize(filtered, ngram_range=_ngram_range)
              yield from processed_doc
code_tokens: def tokenize ( self , data : List [ str ] , ngram_range = ( 1 , 1 ) , lowercase = True ) -> Generator [ List [ str ] , Any , None ] : size = len ( data ) _ngram_range = self . ngram_range or ngram_range _lowercase = self . lowercase or lowercase for i , doc in enumerate ( data ) : logger . debug ( "Tokenize doc {} from {}" . format ( i , size ) ) if _lowercase : tokens = doc . lower ( ) . split ( ) else : tokens = doc . split ( ) filtered = self . _filter ( tokens ) processed_doc = ngramize ( filtered , ngram_range = _ngram_range ) yield from processed_doc
short_docstring: Tokenize a list of documents.
short_docstring_tokens: Tokenize a list of documents .
comment: """
          Tokenize a list of documents.
          :param data: a list of documents to process
          :param ngram_range: range for producing ngrams, ex. for unigrams + bigrams should be set to
          (1, 2), for bigrams only should be set to (2, 2)
          :param lowercase: whether to perform lowercasing or not
          :return: a single processed doc generator
          """