title: Connecting Jupyter notebook to Spark
body: <p>I have a machine with Hadoop and Spark installed. Below is my current environment.</p>
  
  <p>python3.6</p>
  
  <p>spark1.5.2</p>
  
  <p>Hadoop 2.7.1.2.3.6.0-3796</p>
  
  <p>I was trying to connect jupyter notebook to connect to spark by building up ipython kernel.</p>
  
  <p>2 new files written.</p>
  
  <ol>
  <li><p>/root/.ipython/profile_pyspark/ipython_notebook_config.py</p></li>
  <li><p>/root/.ipython/profile_pyspark/startup/00-pyspark-setup.py</p></li>
  <li><p>/root/anaconda3/share/jupyter/kernels/pyspark/kernel.json</p></li>
  </ol>
  
  <p>kernel.json</p>
  
  <pre><code>{
      "display_name": "PySpark (Spark 2.0.0)",
      "language": "python",
      "argv": [
          "/root/anaconda3/bin/python3",
          "-m",
          "ipykernel",
          "--profile=pyspark"
      ],
      "env": {
          "CAPTURE_STANDARD_OUT": "true",
          "CAPTURE_STANDARD_ERR": "true",
          "SEND_EMPTY_OUTPUT": "false",
          "PYSPARK_PYTHON" : "/root/anaconda3/bin/python3",
          "SPARK_HOME": "/usr/hdp/current/spark-client/"
      }
  }
  </code></pre>
  
  <p>00-pyspark-setup.py</p>
  
  <pre><code>import os
  import sys
  os.environ["PYSPARK_PYTHON"] = "/root/anaconda3/bin/python"
  os.environ["SPARK_HOME"] = "/usr/hdp/current/spark-client"
  os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib"
  spark_home = os.environ.get('SPARK_HOME', None)
  sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.8.2.1-src.zip")
  sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")
  exec(open(os.path.join(spark_home, 'python/pyspark/shell.py')).read())
  </code></pre>
  
  <p>ipython_notebook_config.py</p>
  
  <pre><code>c = get_config()
  c.NotebookApp.port = 80
  </code></pre>
  
  <p>Then, when i run the following</p>
  
  <pre><code>jupyter notebook --profile=pyspark
  </code></pre>
  
  <p>The notebook is running well. Then, i change kernel to 'PySpark (Spark 2.0.0)', and it suppose to use the 'sc' spark context .  However, when i type 'sc', it does not shows anything.</p>
  
  <p>So, since sc cannot be initiliazed, if i want to run the following, it failed!</p>
  
  <pre><code>nums = sc.parallelize(xrange(1000000))
  </code></pre>
  
  <p>Can anybody help me how to configure jupyter notebook to talk to Spark?</p>