title: Apply a custom function on an entire column of data.table?
body: <p>I have a very large <code>Data Table</code> with two columns. And I wish to apply a custom function on a particular column. The code to generate the problem is as follows:</p>
  
  <pre><code>require(data.table)
  X &lt;- rep("This is just random text", 1e5)
  data &lt;- data.frame(1:1e5, replicate(1, X, simplify=FALSE), stringsAsFactors=FALSE)
  colnames(data) &lt;- paste("X", seq_len(ncol(data)), sep="")
  DT &lt;- as.data.table(data)
  </code></pre>
  
  <p>Now, we have a large data table which looks like</p>
  
  <pre><code>| X1 |            X2           |
  |----|-------------------------|
  | 1  | This is just random text|
  | 2  | This is just random text|
  | 3  | This is just random text|
  | 4  | This is just random text|
  | .. |            ...          |
  </code></pre>
  
  <p>What if I want to do some vector operation on any of this column considering in mind that this data.table will be of very large size (approx <code>~100M</code> rows). </p>
  
  <p>Let's take an example of X1 column. Suppose, I want to apply the following function on it:</p>
  
  <pre><code>Fun4X1 &lt;- function(x){return(x+x*2)}
  </code></pre>
  
  <p>And a very complex <code>NLP</code> function on X2 column which looks something like</p>
  
  <pre><code>Fun4X2 &lt;- function(x){
               require(stringr)
               return(str_split(x, " ")[[1]][1])
            }
  </code></pre>
  
  <p>How shall I go about doing this for a large dataset? Please suggest the min. time consuming approach as my <code>Function</code> is itself very complex.</p>
  
  <p>P.S. I have tried <code>foreach</code>, <code>sapply</code>, and of course <code>for-loop</code> and all are very slow on a pretty good hardware system.</p>