title: How do I load a local file onto Spark using a Jupyter notebook in an EMR cluster?
body: <p>I'm struggling to load a local file on an EMR core node into Spark and run a Jupyter notebook. I keep getting errors from the task nodes saying that the file doesn't exist, but I've tried setting the <code>spark</code> configuration to be <code>local</code>, so I'm not sure how to fix this. The following works when I spin up a 1-node cluster, but fails when I have anything larger than that:</p>
  
  <pre><code>spark = SparkSession.builder \
          .master("local") \
          .appName("Test") \
  
  df = spark.read.csv('/home/hadoop/dataset.csv')
  
  df.show(n=5)
  </code></pre>
  
  <p>I've tried restarting the Jupyter notebook kernel, but that hasn't fixed anything. So I'd like to know how to either broadcast/share a local file onto the entire cluster, or create a <code>SparkSession</code> instance that works. I'm also using the new AWS JupyterHub, if that makes any difference</p>