title: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/StructFilters in spark scala application in intellij
body: <p>My Pom.xml</p>
  <pre><code> &lt;dependencies&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
        &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
        &lt;version&gt;${scala.version}&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;junit&lt;/groupId&gt;
        &lt;artifactId&gt;junit&lt;/artifactId&gt;
        &lt;version&gt;4.4&lt;/version&gt;
        &lt;scope&gt;test&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.specs&lt;/groupId&gt;
        &lt;artifactId&gt;specs&lt;/artifactId&gt;
        &lt;version&gt;1.2.5&lt;/version&gt;
        &lt;scope&gt;test&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-core_2.12&lt;/artifactId&gt;
        &lt;version&gt;3.0.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-sql_2.12&lt;/artifactId&gt;
        &lt;version&gt;3.0.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.databricks/spark-xml --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.databricks&lt;/groupId&gt;
        &lt;artifactId&gt;spark-xml_2.12&lt;/artifactId&gt;
        &lt;version&gt;0.10.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;mysql&lt;/groupId&gt;
        &lt;artifactId&gt;mysql-connector-java&lt;/artifactId&gt;
        &lt;version&gt;5.1.29&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
        &lt;artifactId&gt;aws-java-sdk&lt;/artifactId&gt;
        &lt;version&gt;1.11.985&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3 --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
        &lt;artifactId&gt;aws-java-sdk-s3&lt;/artifactId&gt;
        &lt;version&gt;1.11.985&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-core --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
        &lt;artifactId&gt;aws-java-sdk-core&lt;/artifactId&gt;
        &lt;version&gt;1.11.985&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
        &lt;artifactId&gt;aws-java-sdk-dynamodb&lt;/artifactId&gt;
        &lt;version&gt;1.11.985&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-cloudwatch --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
        &lt;artifactId&gt;aws-java-sdk-cloudwatch&lt;/artifactId&gt;
        &lt;version&gt;1.11.985&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-kinesis --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
        &lt;artifactId&gt;aws-java-sdk-kinesis&lt;/artifactId&gt;
        &lt;version&gt;1.11.985&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-avro --&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-avro_2.12&lt;/artifactId&gt;
        &lt;version&gt;3.1.1&lt;/version&gt;
      &lt;/dependency&gt;
  </code></pre>
  <p>I want to read the avro file</p>
  <pre><code> val conf = new SparkConf().setAppName(&quot;Nightmare&quot;).setMaster(&quot;local&quot;)
  val sc = new SparkContext(conf)
  sc.setLogLevel(&quot;Error&quot;)
  val spark= SparkSession.builder().getOrCreate()
  import spark.implicits._
  //Step 1-2 read avro file
  println(&quot;Step 1-2&quot;)
  val df1 = spark.read
    .format(&quot;com.databricks.spark.avro&quot;)
    .option(&quot;multiline&quot;,&quot;true&quot;)
    .load(&quot;file:///D:/bigdata_tasks/nightmare.avro&quot;)
  //step 3-4 Hit the url --- Convert it to dataframe -- https://randomuser.me/api/0.8/?results=1000 - df2
  println(&quot;Step 3-5&quot;)
  val html =Source.fromURL(&quot; https://randomuser.me/api/0.8/?results=1000&quot;)
  val rdddata=html.mkString
  //convert string to rdd
  val paralleldata=sc.parallelize(List(rdddata))
  val df2= spark.read.json(paralleldata)
  df2.printSchema()
  df2.show()
  </code></pre>
  <p>After running it is giving exception :</p>
  <p>Exception in thread &quot;main&quot; java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/StructFilters</p>
  <p>I have tried with the following code also:</p>
  <pre><code>      val df1 = spark.read
        .format(&quot;avro&quot;)
        .option(&quot;multiline&quot;,&quot;true&quot;)
        .load(&quot;file:///D:/bigdata_tasks/nightmare.avro&quot;)
  </code></pre>
  <p>But still same exception is giving. My spark version is 2.12. Should I update spark version?</p>