title: R+Hadoop: How to read CSV file from HDFS and execute mapreduce?
body: <p>In the following example:</p>
  
  <pre><code>  small.ints = to.dfs(1:1000)
    mapreduce(
      input = small.ints, 
      map = function(k, v) cbind(v, v^2))
  </code></pre>
  
  <p>The data input for mapreduce function is an object named small.ints which refered to blocks in HDFS. </p>
  
  <p>Now I have a CSV file already stored in HDFS as </p>
  
  <pre><code>"hdfs://172.16.1.58:8020/tmp/test_short.csv"
  </code></pre>
  
  <p>How to get an object for it? </p>
  
  <p>And as far as I know(which may be wrong), if I want data from CSV file as input for mapreduce, I have to first generate a table in R which contains all values in the CSV file. I do have method like:</p>
  
  <pre><code>data=from.dfs("hdfs://172.16.1.58:8020/tmp/test_short.csv",make.input.format(format="csv",sep=","))
  mydata=data$val
  </code></pre>
  
  <p>It seems OK to use this method to get mydata, and then do object=to.dfs(mydata), but the problem is the test_short.csv file is huge, which is around TB size, and memory can't hold output of from.dfs!!</p>
  
  <p>Actually, I'm wondering if I use "hdfs://172.16.1.58:8020/tmp/test_short.csv" as mapreduce input directly, and inside map function do the from.dfs() thing, am I able to get data blocks? </p>
  
  <p>Please give me some advice, whatever!</p>