title: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.DataSourceV2 for Spark 3.0.0
body: <h2>Brief</h2>
  <p><strong>What are possible paths that can make me process data by pyspark 3.0.0 with success from the pure <code>pip</code> installation, well, at least loading data without downgrading the version of Spark?</strong></p>
  <p>When I attempted to load datasets of <code>parquet</code> and <code>csv</code>, I would get the exception message as the content below <em>Exception Message</em> displays. The initialization of Spark session is fine, yet when I wanted to load datasets, it just went wrong.</p>
  <h2>Some Information</h2>
  <ul>
  <li>Java: openjdk 11</li>
  <li>Python: 3.8.5</li>
  <li>Mode: local mode</li>
  <li>Operating System: Ubuntu 16.04.6 LTS</li>
  <li>Notes:
  <ol>
  <li>I executed <code>python3.8 -m pip install pyspark</code> to install Spark.</li>
  <li>When I looked up the jar of <code>spark-sql_2.12-3.0.0.jar</code> (which is under the Python site-package path, i.e., <code>~/.local/lib/python3.8/site-packages/pyspark/jars</code> in my case), there is no <code>v2</code> under <code>spark.sql.sources</code>, the most similar one I found is an interface called <code>DatSourceRegister</code> under the same package.</li>
  <li>The most similar question I found on Stackoverflow is <a href="https://stackoverflow.com/questions/61362841/pyspark-structured-streaming-kafka-error-caused-by-java-lang-classnotfoundex">PySpark structured Streaming + Kafka Error (Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.StreamWriteSupport )</a> where downgrading the Spark version is recommended throughout the information on that page.</li>
  </ol>
  </li>
  </ul>
  <h2>Exception Message</h2>
  <pre><code>Py4JJavaError: An error occurred while calling o94.csv.
  : java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2
      at java.base/java.lang.ClassLoader.defineClass1(Native Method)
      at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
      at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
      at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
      at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
      at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
      at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
      at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:575)
      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
      at java.base/java.lang.Class.forName0(Native Method)
      at java.base/java.lang.Class.forName(Class.java:398)
      at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1209)
      at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1220)
      at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1264)
      at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1299)
      at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1384)
      at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:43)
      at scala.collection.Iterator.foreach(Iterator.scala:941)
      at scala.collection.Iterator.foreach$(Iterator.scala:941)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
      at scala.collection.IterableLike.foreach(IterableLike.scala:74)
      at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
      at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255)
      at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249)
      at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
      at scala.collection.TraversableLike.filter(TraversableLike.scala:347)
      at scala.collection.TraversableLike.filter$(TraversableLike.scala:347)
      at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
      at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:644)
      at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:728)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:230)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:705)
      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.base/java.lang.Thread.run(Thread.java:834)
  Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.DataSourceV2
      at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
      at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
      at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
      ... 45 more
  </code></pre>