title: Py4JJavaError when testing Pyspark in Jupyter notebook on a single machine
body: <p>I am new to Spark and recently installed it on a mac (with Python 2.7 in the system) using homebrew:</p>
  <pre><code>brew install apache-spark
  </code></pre>
  <p>and then installed Pyspark using pip3 in my virtual environment where I have python 3.6 installed.</p>
  <pre><code>/Users/xxx/venv/bin/python /Users/xxx/venv/bin/pip3 install pyspark
  </code></pre>
  <p>When I ran the code below in my Jupyter Notebook to test if Spark functions on a single machine:</p>
  <pre><code>from pyspark.context import SparkContext
  sc = SparkContext.getOrCreate()
  
  import random
  num_samples = 100000000
  
  def inside(p):     
      x, y = random.random(), random.random()
      return x*x + y*y &lt; 1
  
  count = sc.parallelize(range(0, num_samples)).filter(inside).count()
  
  pi = 4 * count / num_samples
  print(pi)
  
  sc.stop()
  
  </code></pre>
  <p>I ran into the following error with <strong>sc.parallelize</strong>:</p>
  <pre><code>Py4JJavaError                             Traceback (most recent call last)
  &lt;ipython-input-3-482026ac7386&gt; in &lt;module&gt;
        8     return x*x + y*y &lt; 1
        9 
  ---&gt; 10 count = sc.parallelize(range(0, num_samples)).filter(inside).count()
       11 
       12 pi = 4 * count / num_samples
  ~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in count(self)
     1139         3
     1140         &quot;&quot;&quot;
  -&gt; 1141         return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
     1142 
     1143     def stats(self):
  
  ~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in sum(self)
     1130         6.0
     1131         &quot;&quot;&quot;
  -&gt; 1132         return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
     1133 
     1134     def count(self):
  
  ~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in fold(self, zeroValue, op)
     1001         # zeroValue provided to each partition is unique from the one provided
     1002         # to the final reduce call
  -&gt; 1003         vals = self.mapPartitions(func).collect()
     1004         return reduce(op, vals, zeroValue)
     1005 
  
  ~/venv/deep_learning/lib/python3.6/site-packages/pyspark/rdd.py in collect(self)
      887         
      888         with SCCallSiteSync(self.context) as css:
  -&gt; 889             sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
      890         return list(_load_from_socket(sock_info, self._jrdd_deserializer))
      891 
  
  ~/venv/deep_learning/lib/python3.6/site-packages/py4j/java_gateway.py in __call__(self, *args)
     1303         answer = self.gateway_client.send_command(command)
     1304         return_value = get_return_value(
  -&gt; 1305             answer, self.gateway_client, self.target_id, self.name)
     1306 
     1307         for temp_arg in temp_args:
  
  ~/venv/deep_learning/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
      326                 raise Py4JJavaError(
      327                     &quot;An error occurred while calling {0}{1}{2}.\n&quot;.
  -&gt; 328                     format(target_id, &quot;.&quot;, name), value)
      329             else:
      330                 raise Py4JError(
  
  Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2, 192.168.0.15, executor driver): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  
    File &quot;/Users/xxx/venv/deep_learning/lib/python3.6/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py&quot;, line 477, in main
      (&quot;%d.%d&quot; % sys.version_info[:2], version))
  
  Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
  
  </code></pre>
  <p>I have configured the Pyspark path as follows in the /.bash_profile:</p>
  <pre><code>export SPARK_HOME=/usr/local/Cellar/apache-spark/3.0.1/libexec
  export PYTHONPATH=/usr/local/Cellar/apache-spark/3.0.1/libexec/python/:$PYTHONP$
  export PYSPARK_PYTHON=/Users/xxx/venv/bin/python
  export PYSPARK_DRIVER_PYTHON=/Users/xxx/venv/bin/python
  </code></pre>
  <p>Does anyone know what I did wrong here? Any advice will be greatly appreciated.</p>