title: Efficiency of flatMap vs map followed by reduce in Spark
body: <p>I have a text file sherlock.txt containing multiple lines of text. I load it in spark-shell using:</p>
  
  <pre><code>val textFile = sc.textFile("sherlock.txt")
  </code></pre>
  
  <p>My purpose is to count the number of words in the file. I came across two alternative ways to do the job.</p>
  
  <p>First using flatMap:</p>
  
  <pre><code>textFile.flatMap(line =&gt; line.split(" ")).count()
  </code></pre>
  
  <p>Second using map followed by reduce:</p>
  
  <pre><code>textFile.map(line =&gt; line.split(" ").size).reduce((a, b) =&gt; a + b)
  </code></pre>
  
  <p>Both yield the same result correctly. I want to know the differences in time and space complexity of the above two alternative implementations, if indeed there is any ?</p>
  
  <p>Does the scala interpreter convert both into the most efficient form ?</p>