title: R dynamic stop word list with terms of frequency one
body: <p>I am working on a text mining assignment and am stuck at the moment. The following is based on <code>Zhaos Text Mining</code> with Twitter. I cannot get it to work, maybe one of you has a good idea?</p>
  
  <p><strong>Goal:</strong> I would like to remove all terms from the corpus with a word count of one instead of using a stopword list.</p>
  
  <p><strong>What I did so far:</strong> I have downloaded the tweets and converted them into a data frame.</p>
  
  <pre><code>tf1 &lt;- Corpus(VectorSource(tweets.df$text))
  
  
  tf1 &lt;- tm_map(tf1, content_transformer(tolower))
  
  
  removeUser &lt;- function(x) gsub("@[[:alnum:]]*", "", x)
  tf1 &lt;- tm_map(tf1, content_transformer(removeUser))
  
  
  removeNumPunct &lt;- function(x) gsub("[^[:alpha:][:space:]]*", "", x)
  tf1 &lt;- tm_map(tf1, content_transformer(removeNumPunct))
  
  
  removeURL &lt;- function(x) gsub("http[[:alnum:]]*", "", x)
  tf1 &lt;- tm_map(tf1, content_transformer(removeURL))
  
  tf1 &lt;- tm_map(tf1, stripWhitespace)
  
  
  #Using TermDocMatrix in order to find terms with count 1, dont know any other way
  tdmtf1 &lt;- TermDocumentMatrix(tf1, control = list(wordLengths = c(1, Inf)))
  
  ones &lt;- findFreqTerms(tdmtf1, lowfreq = 1, highfreq = 1)
  
  tf1Copy &lt;- tf1
  
  tf1List &lt;- setdiff(tf1Copy, ones)
  
  
  tf1CList &lt;- paste(unlist(tf1List),sep="", collapse=" ")
  
  tf1Copy &lt;- tm_map(tf1Copy, removeWords, tf1CList)
  
  tdmtf1Test &lt;- TermDocumentMatrix(tf1Copy, control = list(wordLengths = c(1, Inf)))
  
  #Just to test success...
  ones2 &lt;- findFreqTerms(tdmtf1Test, lowfreq = 1, highfreq = 1)
  (ones2)
  </code></pre>
  
  <p><strong>The Error:</strong></p>
  
  <blockquote>
    <p>Error in gsub(sprintf("(*UCP)\b(%s)\b", paste(sort(words, decreasing = TRUE),  :   invalid regular expression '(*UCP)\b(senior data scientist global strategy firm <br>
    25.0010230541229 48 17 6 6 115 1 186 0 1 en kdnuggets poll primary programming language for analytics data mining data scienc <br>
    25.0020229816437 48 17 6 6 115 1 186 0 2 en iapa canberra seminar mining the internet of everything official statistics in the information age anu june <br> 25.0020229816437 48 17 6 6 115 1 186 0 3 en handling and processing strings in r an ebook in pdf format pages<br>
    25.0020229816437 48 17 6 6 115 1 186 0 4 en webinar getting your data into r by hadley wickham am edt june th <br>
    25.0020229816437 48 17 6 6 115 1 186 0 5 en  before loading the rdmtweets dataset please run librarytwitter to load required package <br>
    25.0020229816437 48 17 6 6 115 1 186 0 6 en an infographic on sas vs r vs python datascience via  <br>
    25.0020229816437 48 17 6 6 115 1 186 0 7 en r is again the kdnuggets poll on top analytics data mining science software <br>
    25.0020229816437 48 17 6 6 115 1 186 0 8 en i will run  </p>
  </blockquote>
  
  <p><strong>In Addition:</strong></p>
  
  <blockquote>
    <p>Warning message: In gsub(sprintf("(*UCP)\b(%s)\b", paste(sort(words, decreasing = TRUE),  :   PCRE pattern compilation error
            'regular expression is too large'
            at ''</p>
  </blockquote>
  
  <p>PS sorry for the bad format at the end could not get it fixed.</p>