title: Create Document Term Matrix with N-Grams in R
body: <p>I am using "tm" package to create DocumentTermMatrix in R. It works well for one - gram but i am trying to create a DocumenttermMatrix of N-Grams(N = 3 for now) using tm package and tokenize_ngrams function from "tokenizers" package.
  But im not able to create it.</p>
  
  <p>I searched for possible solution but i didnt get much help.
  For privacy reasons i can not share the data.
  Here is what i have tried,  </p>
  
  <pre><code>library(tm)  
  library(tokenizers)
  </code></pre>
  
  <p>data is a dataframe with around 4.5k rows and 2 columns namely "doc_id" and "text"</p>
  
  <pre><code>data_corpus = Corpus(DataframeSource(data))
  </code></pre>
  
  <p>custom function for n-gram tokenization :  </p>
  
  <pre><code>ngram_tokenizer = function(x){
    temp = tokenize_ngrams(x, n_min = 1, n = 3, stopwords = FALSE, ngram_delim = "_")
    return(temp)
  }
  </code></pre>
  
  <p>control list for DTM creation :<br>
  1-gram  </p>
  
  <pre><code>control_list_unigram = list(tokenize = "words",
                            removePunctuation = FALSE,
                            removeNumbers = FALSE, 
                            stopwords = stopwords("english"), 
                            tolower = T, 
                            stemming = T, 
                            weighting = function(x)
                              weightTf(x)
  )
  </code></pre>
  
  <p>for N-gram tokenization</p>
  
  <pre><code>control_list_ngram = list(tokenize = ngram_tokenizer,
                      removePunctuation = FALSE,
                      removeNumbers = FALSE, 
                      stopwords = stopwords("english"), 
                      tolower = T, 
                      stemming = T, 
                      weighting = function(x)
                        weightTf(x)
                      )
  
  
  dtm_unigram = DocumentTermMatrix(data_corpus, control_list_unigram)
  dtm_ngram = DocumentTermMatrix(data_cropus, control_list_ngram)
  
  dim(dtm_unigram)
  dim(dtm_ngram)
  </code></pre>
  
  <p>The dimension of both the dtm's were same.<br>
  Please correct me!</p>