 <p>When you run Spark in <strong>client mode</strong>, the driver process runs <strong>locally</strong>. In <strong>cluster mode</strong>, it runs <strong>remotely</strong> on an ApplicationMaster.</p> <p>In other words you will need all the nodes to see each other. Spark driver definitely needs to communicate with all the worker nodes. If this is a problem try to use the <code>yarn-cluster</code> mode, then the driver will run inside your cluster on one of the nodes.</p>