title: Apache Spark architecture
body: <p>Trying to find a complete documentation about an internal architecture of Apache Spark, but have no results there.</p>
  
  <p>For example I'm trying to understand next thing:
  Assume that we have 1Tb text file on HDFS (3 nodes in a cluster, replication factor is 1). This file will be spitted into 128Mb chunks and each chunk will be stored only on one node. We run Spark Workers on these nodes. I know that Spark is trying to work with data stored in HDFS on the same node (to avoid network I/O). For example I'm trying to do a word count in this 1Tb text file.</p>
  
  <p>Here I have next questions:</p>
  
  <ol>
  <li>Does Spark will load chuck (128Mb) into RAM, count words, and then delete it from memory and do it sequentially? What if there will be no available RAM?</li>
  <li>When does Spark will use not local data on HDFS?</li>
  <li>What if I will need to do more complex task, when a results of each iteration on each Worker need to be transferred to all other Workers (shuffling?), do I need to write them by my self to HDFS and then read them? For example I can't understand how does K-means clustering or Gradient descent works on Spark.</li>
  </ol>
  
  <p>I will appreciate any link to Apache Spark architecture guide.</p>