title: PySpark java.lang.ExceptionInInitializerError Caused by: java.lang.StringIndexOutOfBoundsException
body: <p>I followed <a href="https://sigdelta.com/blog/how-to-install-pyspark-locally/" rel="nofollow noreferrer">these instructions</a> and installed Apache Spark (PySpark) 2.3.1 on my machine which has the specifications:</p>
  
  <ul>
  <li>Ubuntu 18.04</li>
  <li>JDK 10</li>
  <li>Python 3.6</li>
  </ul>
  
  <p>When I create a <a href="https://spark.apache.org/docs/preview/api/python/pyspark.sql.html#pyspark.sql.SparkSession" rel="nofollow noreferrer"><code>SparkSession</code></a> either indirectly by calling <code>pyspark</code> from the shell or by directly creating a session in my app with:</p>
  
  <pre class="lang-py prettyprint-override"><code>spark = pyspark.sql.SparkSession.builder.appName('test').getOrCreate()
  </code></pre>
  
  <p>I get the following exception:</p>
  
  <pre><code>Exception in thread "main" java.lang.ExceptionInInitializerError
      at org.apache.hadoop.util.StringUtils.&lt;clinit&gt;(StringUtils.java:80)
      at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
      at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
  ....
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
      at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3107)
      at java.base/java.lang.String.substring(String.java:1873)
      at org.apache.hadoop.util.Shell.&lt;clinit&gt;(Shell.java:52)
      ... 22 more
  Traceback (most recent call last):
    File "/home/welshamy/tools/anaconda3/lib/python3.6/site-packages/pyspark/python/pyspark/shell.py", line 38, in &lt;module&gt;
      SparkContext._ensure_initialized()
    File "/home/welshamy/tools/anaconda3/lib/python3.6/site-packages/pyspark/context.py", line 292, in _ensure_initialized
      SparkContext._gateway = gateway or launch_gateway(conf)
    File "/home/welshamy/tools/anaconda3/lib/python3.6/site-packages/pyspark/java_gateway.py", line 93, in launch_gateway
      raise Exception("Java gateway process exited before sending its port number")
  Exception: Java gateway process exited before sending its port number
  </code></pre>
  
  <p>If I'm using a Jupyter notebook, I also get this exception in the notebook:</p>
  
  <pre><code>Exception: Java gateway process exited before sending the driver its port number
  </code></pre>
  
  <p>All the solutions I found and followed [<a href="https://stackoverflow.com/questions/31841509/pyspark-exception-java-gateway-process-exited-before-sending-the-driver-its-po">1</a>,<a href="https://github.com/jupyter/notebook/issues/743" rel="nofollow noreferrer">2</a>,<a href="https://github.com/jupyter/jupyter/issues/248" rel="nofollow noreferrer">3</a>] point toward environment variables definitions, but non of them worked for me.</p>