title: Using pyspark in Google Colab
body: <p>This is my first question here after using a lot of StackOverflow so correct me if I give inaccurate or incomplete info</p>
  <p>Up until  this week I had a colab notebook setup to run with pyspark following one of the many guides I found throughout the internet, but this week it started popping with a few different errors.</p>
  <p>The code used is pretty much this one:</p>
  <pre><code>!apt-get install openjdk-8-jdk-headless -qq &gt; /dev/null
  !wget -q https://downloads.apache.org/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz
  !tar -xvf spark-3.0.0-preview2-bin-hadoop2.7.tgz
  !pip install -q findspark
  import os
  os.environ[&quot;JAVA_HOME&quot;] = &quot;/usr/lib/jvm/java-8-openjdk-amd64&quot;
  os.environ[&quot;SPARK_HOME&quot;] = &quot;/content/spark-3.0.0-preview2-bin-hadoop2.7&quot;
  import findspark
  findspark.init()
  </code></pre>
  <p>I have tried changing the Java version from 8 to 11 and using all of the available Spark builds on <a href="https://downloads.apache.org/spark/" rel="noreferrer">https://downloads.apache.org/spark/</a> and changing the HOME paths accordingly. I used <code>pip freeze</code> as seen on one guide to check the Spark version used in colab and it said pyspark 3.0.0 so I tried all the ones on version 3.0.0 and all I keep getting is the error:</p>
  <pre><code>Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly
  </code></pre>
  <p>I don't understand much about the need of using Java for this, but I also tried installing pyj4 though <code>!pip install py4j</code> and it says its already installed when I do, and I tried every different guide on the internet, but I can't run my Spark code anymore. Does anyone know how to fix this?
  I only use colab for college purposes because my PC is quite outdated and I don't know much about it, but I really need to get this notebook running reliably and so how do I know it's time to update the imported builds?</p>