original_docstring: Generate a sparse term-document co-occurrence matrix and create a TfIdfTransformer that can
      be used to derive the idf scores and a matrix of tf-idf scores for term-document pairs from the
      first matrix.
  
      Occurrences can additionally also be counted for n-grams using the n_lower and n_upper
      parameters, e.g. n_lower=1 and n_upper=2 will results in all unigrams and bigrams being counted.
  
      Args:
          documents: The documents to count term occurrences in.
          n_lower: The lower bound for n-grams.
          n_upper: The upper bound for n-grams.
  
      Returns:
          List of all words.
          Sparse co-occurrence matrix of term-document pairs.
          TfIdfTransformer that can extract a tf-idf matrix or the idf vector.
      
docstring: Generate a sparse term-document co-occurrence matrix and create a TfIdfTransformer that can
  be used to derive the idf scores and a matrix of tf-idf scores for term-document pairs from the
  first matrix.
  
  Occurrences can additionally also be counted for n-grams using the n_lower and n_upper
  parameters, e.g. n_lower=1 and n_upper=2 will results in all unigrams and bigrams being counted.
docstring_tokens: Generate a sparse term - document co - occurrence matrix and create a TfIdfTransformer that can be used to derive the idf scores and a matrix of tf - idf scores for term - document pairs from the first matrix . Occurrences can additionally also be counted for n - grams using the n_lower and n_upper parameters e . g . n_lower = 1 and n_upper = 2 will results in all unigrams and bigrams being counted .
code: def generate_token_matrices(
      documents: List[List[str]], n_lower: int, n_upper: int
  ) -> Tuple[List[str], csr_matrix, TfidfTransformer]:
      """Generate a sparse term-document co-occurrence matrix and create a TfIdfTransformer that can
      be used to derive the idf scores and a matrix of tf-idf scores for term-document pairs from the
      first matrix.
  
      Occurrences can additionally also be counted for n-grams using the n_lower and n_upper
      parameters, e.g. n_lower=1 and n_upper=2 will results in all unigrams and bigrams being counted.
  
      Args:
          documents: The documents to count term occurrences in.
          n_lower: The lower bound for n-grams.
          n_upper: The upper bound for n-grams.
  
      Returns:
          List of all words.
          Sparse co-occurrence matrix of term-document pairs.
          TfIdfTransformer that can extract a tf-idf matrix or the idf vector.
      """
      english_words = clean_.english_words()
      stopwords = clean_.stopwords_and_more()
      lemmatizer = clean_.get_lemmatizer()
      vectorizer = CountVectorizer(
          analyzer="word",
          tokenizer=lambda doc: clean_.remove_stopwords(
              clean_.tokenize_and_lemmatize(doc, lemmatizer), stopwords
          ),
          preprocessor=lambda doc: clean_.newline_hyphens(doc, english_words),
          lowercase=True,
          ngram_range=(n_lower, n_upper),
      )
      # the sklearn tokenizer splits "open-source", nltk does not
      counts_matrix = vectorizer.fit_transform(documents)  # is a sparse matrix
      tfidf_matrix = TfidfTransformer(smooth_idf=True, use_idf=True).fit(counts_matrix)
  
      return vectorizer.get_feature_names(), counts_matrix, tfidf_matrix
code_tokens: def generate_token_matrices ( documents : List [ List [ str ] ] , n_lower : int , n_upper : int ) -> Tuple [ List [ str ] , csr_matrix , TfidfTransformer ] : english_words = clean_ . english_words ( ) stopwords = clean_ . stopwords_and_more ( ) lemmatizer = clean_ . get_lemmatizer ( ) vectorizer = CountVectorizer ( analyzer = "word" , tokenizer = lambda doc : clean_ . remove_stopwords ( clean_ . tokenize_and_lemmatize ( doc , lemmatizer ) , stopwords ) , preprocessor = lambda doc : clean_ . newline_hyphens ( doc , english_words ) , lowercase = True , ngram_range = ( n_lower , n_upper ) , ) counts_matrix = vectorizer . fit_transform ( documents ) tfidf_matrix = TfidfTransformer ( smooth_idf = True , use_idf = True ) . fit ( counts_matrix ) return vectorizer . get_feature_names ( ) , counts_matrix , tfidf_matrix
short_docstring: Generate a sparse term-document co-occurrence matrix and create a TfIdfTransformer that can
  be used to derive the idf scores and a matrix of tf-idf scores for term-document pairs from the
  first matrix.
short_docstring_tokens: Generate a sparse term - document co - occurrence matrix and create a TfIdfTransformer that can be used to derive the idf scores and a matrix of tf - idf scores for term - document pairs from the first matrix .
comment: """Generate a sparse term-document co-occurrence matrix and create a TfIdfTransformer that can
      be used to derive the idf scores and a matrix of tf-idf scores for term-document pairs from the
      first matrix.
  
      Occurrences can additionally also be counted for n-grams using the n_lower and n_upper
      parameters, e.g. n_lower=1 and n_upper=2 will results in all unigrams and bigrams being counted.
  
      Args:
          documents: The documents to count term occurrences in.
          n_lower: The lower bound for n-grams.
          n_upper: The upper bound for n-grams.
  
      Returns:
          List of all words.
          Sparse co-occurrence matrix of term-document pairs.
          TfIdfTransformer that can extract a tf-idf matrix or the idf vector.
      """$SEP$# the sklearn tokenizer splits "open-source", nltk does not$SEP$# is a sparse matrix