title: How to write custom removePunctuation() function to better deal with Unicode chars?
body: <p>In the source code of the tm text-mining R-package, in file <a href="https://r-forge.r-project.org/scm/viewvc.php/pkg/R/transform.R?view=markup&amp;root=tm" rel="nofollow">transform.R</a>, there is the <code>removePunctuation()</code> function, currently defined as:</p>
  
  <pre><code>function(x, preserve_intra_word_dashes = FALSE)
  {
      if (!preserve_intra_word_dashes)
          gsub("[[:punct:]]+", "", x)
      else {
          # Assume there are no ASCII 1 characters.
          x &lt;- gsub("(\\w)-(\\w)", "\\1\1\\2", x)
          x &lt;- gsub("[[:punct:]]+", "", x)
          gsub("\1", "-", x, fixed = TRUE)
      }
  }
  </code></pre>
  
  <p>I need to parse and mine some abstracts from a science conference (fetched from their website as UTF-8). The abstracts contain some unicode characters that need to be removed, particularly at word boundaries. There are the usual ASCII punctuation characters, but also a few Unicode Dashes, Unicode Quotes, Math Symbols... </p>
  
  <p>There are also URLs in the text, and there the punctuation the intra-word punctuation characters need to be preserved. tm's built-in <code>removePunctuation()</code> function is too radical.</p>
  
  <p>So I need a custom <code>removePunctuation()</code> function to do removal according to my requirements.</p>
  
  <p>My custom Unicode function looks like this now, but it does not work as expected. I am using R only rarely, so getting things done in R takes some time, even for the simplest tasks. </p>
  
  <p><strong>My function:</strong></p>
  
  <pre><code>corpus &lt;- tm_map(corpus, rmPunc =  function(x){ 
  # lookbehinds 
  # need to be careful to specify fixed-width conditions 
  # so that it can be used in lookbehind
  
  x &lt;- gsub('(.*?)(?&lt;=^[[:punct:]’“”:±&lt;/&gt;]{5})([[:alnum:]])'," \\2", x, perl=TRUE) ;
  x &lt;- gsub('(.*?)(?&lt;=^[[:punct:]’“”:±&lt;/&gt;]{4})([[:alnum:]])'," \\2", x, perl=TRUE) ;
  x &lt;- gsub('(.*?)(?&lt;=^[[:punct:]’“”:±&lt;/&gt;]{3})([[:alnum:]])'," \\2", x, perl=TRUE) ;
  x &lt;- gsub('(.*?)(?&lt;=^[[:punct:]’“”:±&lt;/&gt;]{2})([[:alnum:]])'," \\2", x, perl=TRUE) ;
  x &lt;- gsub('(.*?)(?&lt;=^[[:punct:]’“”:±&lt;/&gt;])([[:alnum:]])'," \\2", x, perl=TRUE) ; 
  # lookaheads (can use variable-width conditions) 
  x &lt;- gsub('(.*?)(?=[[:alnum:]])([[:punct:]’“”:±]+)$',"\1 ", x, perl=TRUE) ;
  
  # remove all strings that consist *only* of punct chars 
  gsub('^[[:punct:]’“”:±&lt;/&gt;]+$',"", x, perl=TRUE) ;
  
  }
  </code></pre>
  
  <p>It does not work as expected. I think, it doesn't do anything at all.
  The punctuation is still inside the terms-document matrix, see:</p>
  
  <pre><code> head(Terms(tdm), n=30)
  
    [1] "&lt;&gt;&lt;/&gt;"                      "---"                       
    [3] "--,"                        ":&lt;/&gt;"                      
    [5] ":()"                        "/)."                       
    [7] "/++"                        "/++,"                      
    [9] "..,"                        "..."                       
   [11] "...,"                       "..)"                       
   [13] "“”,"                        "(|)"                       
   [15] "(/)"                        "(.."                       
   [17] "(..,"                       "()=(|=)."                  
   [19] "(),"                        "()."                       
   [21] "(&amp;)"                        "++,"                       
   [23] "(0°"                        "0.001),"                   
   [25] "0.003"                      "=0.005)"                   
   [27] "0.006"                      "=0.007)"                   
   [29] "000km"                      "0.01)" 
  ...
  </code></pre>
  
  <p><strong>So my questions are:</strong></p>
  
  <ol>
  <li>Why doesn't the call to my function(){} have the desired effect? How can my
  function be improved? </li>
  <li>Are Unicode regex pattern classes such as if
  <code>\P{ASCII}</code> or <code>\P{PUNCT}</code> supported in R's perl-compatible regular
  expressions? I think they aren't (by default) by <a href="http://www.regular-expressions.info/pcre.html" rel="nofollow">PCRE:</a>: " Only the support for various Unicode properties with \p is incomplete, though the most important ones are supported."</li>
  </ol>