title: How does Spark parallelize the processing of a 1TB file?
body: <p><strong>Imaginary problem</strong></p>
  
  <ul>
  <li>A gigantic CSV log file, let's say 1 TB in size, the file is located on a USB drive</li>
  <li>The log contains activities logs of users around the world, let's assume that the line contains 50 columns, among those there is Country.</li>
  <li>We want a line count per country, descending order.</li>
  <li>Let's assume the Spark cluster has enough nodes with RAM to process the entire 1TB in memory (20 nodes, 4 cores CPU, each node has 64GB RAM)</li>
  </ul>
  
  <p><strong>My Poorman's conceptual solution</strong>
  Using <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.SQLContext" rel="noreferrer">SparkSQL</a> &amp; <a href="https://github.com/databricks/spark-csv" rel="noreferrer">Databricks spark-csv</a></p>
  
  <pre><code>$ ./spark-shell --packages com.databricks:spark-csv_2.10:1.4.0
  </code></pre>
  
  <pre class="lang-scala prettyprint-override"><code>val dfBigLog = sqlContext.read
    .format("com.databricks.spark.csv")
    .option("header", "true")
    .load("/media/username/myUSBdrive/bogusBigLog1TB.log")
  
  dfBigLog.select("Country")
    .groupBy("Country")
    .agg(count($"Country") as "CountryCount")
    .orderBy($"CountryCount".desc).show
  </code></pre>
  
  <p><strong>Question 1: How does Spark parallelize the processing?</strong></p>
  
  <p>I suppose the majority of the execution time (99% ?) of the above solution is to read the 1TB file from the USB drive into the Spark cluster. Reading the file from the USB drive is not parallelizable. But after reading the entire file, what does Spark do under the hood to parallelize the processing?</p>
  
  <ul>
  <li><p>How many nodes used for creating the DataFrame? (maybe only one?)</p></li>
  <li><p>How many nodes used for <code>groupBy</code> &amp; <code>count</code>? Let's assume there are 100+ countries (but Spark doesn't know that yet). How would Spark partition to distribute the 100+ country values on 20 nodes?</p></li>
  </ul>
  
  <p><strong>Question 2: How to make the Spark application the fastest possible?</strong>
  I suppose the area of improvement would be to parallelize the reading of the 1TB file.</p>
  
  <ul>
  <li><p>Convert the CSV File into a Parquet file format + using Snappy compression. Let's assume this can be done in advance.</p></li>
  <li><p>Copy the Parquet file on HDFS. Let's assume the Spark cluster is within the same Hadoop cluster and the datanodes are independant from the 20 nodes Spark cluster.</p></li>
  <li><p>Change the Spark application to read from HDFS. I suppose Spark would now use several nodes to read the file as Parquet is splittable.</p></li>
  <li><p>Let's assume the Parquet file compressed by Snappy is 10x smaller, size = 100GB, HDFS block = 128 MB in size. Total 782 HDFS blocks.</p></li>
  </ul>
  
  <p>But then how does Spark manage to use all the 20 nodes for both creating the DataFrame and the processing (<code>groupBy</code> and <code>count</code>)? Does Spark use all the nodes each time?</p>