title: How to buffer outputs of spark stream to prevent generation of milions of small files?
body: <p>I am using spark stream to process files periodically from HDFS and produce results to HDFS. Each worker in each micro-batch generate a small file. I want to prevent from generating such small files (output format is sequence file). Here is some potential solutions:</p>
  
  <p>1- Each worker buffers outputs of itself. When its buffer reaches to the predefined threshold, it will writes it to hdfs.</p>
  
  <p>2- Using repartition in each micro batch to merge outputs of multiple workers, then writing them as a single file.</p>
  
  <p>3- Using another stream job to merge small files into bigger ones.</p>
  
  <p>4- Writing Key-Value pairs into Hive and exporting big files from it.</p>
  
  <p>but each one has its own drawbacks:</p>
  
  <p>1- buffering increases disk accesses. Moreover, in the case of failure large amount of input must be processed again.</p>
  
  <p>2- repartitioning increases the network traffic. Moreover, it may still be small, again.</p>
  
  <p>3- merging doubles number of read and writes to hdfs. </p>
  
  <p>4- according to <a href="https://stackoverflow.com/questions/32885825/persisting-spark-streaming-output">Persisting Spark Streaming output</a> its performance is not desirable.</p>
  
  <p>My question: Are there any other solutions to this problem. What is the best practice to such problems. </p>
  
  <p>Thanks</p>