title: Connecting IPython notebook to spark master running in different machines
body: <p>I don't know if this is already answered in SO but I couldn't find a solution to my problem.</p>
  
  <p>I have an IPython notebook running in a docker container in Google Container Engine, the container is based on this image <a href="https://github.com/jupyter/docker-stacks/tree/master/all-spark-notebook" rel="noreferrer">jupyter/all-spark-notebook</a></p>
  
  <p>I have also a spark cluster created with <a href="https://cloud.google.com/dataproc/" rel="noreferrer">google cloud dataproc</a></p>
  
  <p>Spark master and the notebook are running in <strong>different VMs</strong> but in the <strong>same region and zone</strong>.</p>
  
  <p>My problem is that I'm trying to connect to the spark master from the IPython notebook but without success. I use this snippet of code in my python notebook</p>
  
  <pre><code>import pyspark
  conf = pyspark.SparkConf()
  conf.setMaster("spark://&lt;spark-master-ip or spark-master-hostname&gt;:7077")
  </code></pre>
  
  <p>I just started working with spark, so I'm sure I'm missing something (authentication, security ...), </p>
  
  <p>What I found over there is connecting a local browser over an <a href="https://cloud.google.com/dataproc/cluster-web-interfaces" rel="noreferrer">SSH tunnel</a></p>
  
  <p>Somebody already did this kind of set up?</p>
  
  <p>Thank you in advance</p>