title: s3fs on Amazon EMR: Will it scale for approx 100million small files?
body: <p>Please refer to the following questions already asked:
  <a href="https://stackoverflow.com/questions/14081123/write-100-million-files-to-s3">Write 100 million files to s3</a>
  and
  <a href="https://stackoverflow.com/questions/12953251/too-many-open-files-in-emr">Too many open files in EMR</a></p>
  
  <p>The size of data being handled here is atleast around 4-5TB. To be precise - 300GB with gzip compression.</p>
  
  <p>The size of input will grow gradually as this step aggregates the data over time.</p>
  
  <p>For example, the logs till December 2012 will contain:</p>
  
  <pre><code>UDID-1, DateTime, Lat, Lng, Location
  UDID-2, DateTime, Lat, Lng, Location
  UDID-3, DateTime, Lat, Lng, Location
  UDID-1, DateTime, Lat, Lng, Location
  </code></pre>
  
  <p>For this we would have to generate separate files with UDID(Unique device identifier) as filenames and records belonging to that UDID in the file in sorted order.</p>
  
  <p>Ex:</p>
  
  <pre><code>UDID-1.dat =&gt; File Contents
  DateTime1, Lat1, Lng1, Location1
  DateTime2, Lat2, Lng2, Location2
  DateTime3, Lat3, Lng3, Location3
  </code></pre>
  
  <p>Now when we have the logs for the month of Jan, 2013, this step will read both the old data, the files already generated for the older months by this step, and the newer logs to aggregate the data of UDIDs.</p>
  
  <p>Ex:</p>
  
  <pre><code>If the logs for month of Jan has a record as: UDID-1, DateTime4, Lat4, Lng4, Location4, the file UDID-1.dat would need to be updated with this data. Each UDID's file should be chronologically sorted.
  </code></pre>
  
  <p>For this step, we thought of writing the data to an EBS volume and keep it as-is for later use. But EBS volumes have a limit of 1TB. As already mentioned in the referenced questions, generating the files on s3 directly or generating on HDFS and then moving to s3 is not a viable option for this use case as there are around 100 million small files which needs to be moved. And moving such large number of files is way too slow even by using s3distcp.</p>
  
  <p>So, next we are going to try s3fs - FUSE-based file system backed by Amazon S3. Does anybody have any idea that how scalable is s3fs? Will it be able to handle 100 million small files? How much time will it take to move 3-5TB of data, spread across 100 million files, from s3 to local filesystem so that it can be used by the MR job? And how much time will it take to move the data back to s3? Will it have the same problem as was faced while using s3distcp?</p>
  
  <p>Thanks in advance !</p>