title: Text mining and NLP: from R to Python
body: <p>First of all, saying that I am new to python. At the moment, I am "translating" a lot of R code into python and learning along the way. This question relates to this one <a href="https://stackoverflow.com/questions/22797393/exactly-replicating-r-text-preprocessing-in-python">replicating R in Python</a> (in there they actually suggest to wrap it up using <code>rpy2</code>, which I would like to avoid for learning purposes). </p>
  
  <p>In my case, rather than exactly replicating R in python, I would actually like to learn a "pythonian" way of doing what I am describing here:</p>
  
  <p>I have a long vector (40000 elements) in which each element is a piece of text, for example:</p>
  
  <pre><code>&gt; descr
  [1] "dress Silver Grey Printed Jersey Dress 100% cotton"
  [2] "dress Printed Silk Dress 100% Silk Effortless style."                                                                                                                                                                                    
  [3] "dress Rust Belted Kimono Dress 100% Silk relaxed silhouette, mini length" 
  </code></pre>
  
  <p>I then preprocess it as, for example: </p>
  
  <pre><code># customized function to remove patterns in strings. used later within tm_map
  rmRepeatPatterns &lt;- function(str) gsub('\\b(\\S+?)\\1\\S*\\b', '', str,
                                     perl = TRUE)
  
  # process the corpus
  pCorp &lt;- Corpus(VectorSource(descr))
  pCorp &lt;- tm_map(pCorp, content_transformer(tolower))
  pCorp &lt;- tm_map(pCorp, rmRepeatPatterns)
  pCorp &lt;- tm_map(pCorp, removeStopWords)
  pCorp &lt;- tm_map(pCorp, removePunctuation)
  pCorp &lt;- tm_map(pCorp, removeNumbers)
  pCorp &lt;- tm_map(pCorp, stripWhitespace)
  pCorp &lt;- tm_map(pCorp, PlainTextDocument)
  
  # create a term document matrix (control functions can also be passed here) and a table: word - freq
  Tdm1 &lt;- TermDocumentMatrix(pCorp)
  freq1 &lt;- rowSums(as.matrix(Tdm1))
  dt &lt;- data.table(terms=names(freq1), freq=freq1)
  
  # and perhaps even calculate a distance matrix (transpose because Dist operates on a row basis)
  D &lt;- Dist(t(as.matrix(Tdm1)))
  </code></pre>
  
  <p>Overall, I would like to know an adequate way of doing this in python, mainly the text processing. </p>
  
  <p>For example, I could remove stopwords and numbers as they describe here <a href="https://stackoverflow.com/questions/5541745/get-rid-of-stopwords-and-punctuation">get rid of StopWords and Numbers</a> (although seems a lot of work for such a simple task). But all the options I see imply processing the text itself rather than mapping the whole corpus. In other words, they imply "looping" through the <code>descr</code> vector. </p>
  
  <p>Anyway, any help would be really appreciated. Also, I have a bunch of customised functions like <code>rmRepeatPatterns</code>, so learning how to map these would be extremely useful. </p>
  
  <p>thanks in advance for your time. </p>