title: How Spark’s HDFS I/O architecture is different from S3?
body: <p>Let’s say I have a regular Hadoop-Yarn-Spark cluster. When Spark reads a big file from HDFS, the file is split into multiple partitions and they are processed by executors on worker nodes. Each executor then is doing physical reads and writes to/from HDFS in the same cluster. Is this correct?</p>
  
  <p>If so, then if I start processing remote S3 data and start reading from there - will only the spark master node do physical I/O (reads and writes) and then transmit the data to executors?</p>