title: How do Spark Nodes communicate during a Shuffle?
body: <p>I've seen from <a href="https://stackoverflow.com/questions/38084293/spark-how-does-it-distribute-data-around-the-nodes">this question</a> that Spark nodes effectively "communicate directly", but I'm less concerned with the theory and more with the implementation. <a href="https://spark.apache.org/docs/2.1.0/configuration.html" rel="noreferrer">Here</a> it shows, in the "###Encryption" section near the bottom of the page, that you can configure Spark to use a number of SSL protocols for security, which would suggest, to me at least, that it uses some form of HTTP(s) for communication. My question is effectively two parts: what protocol do Spark nodes use to communicate, and how is the data formatted for this transfer?</p>