title: Spark 2.3.1 structured streaming kafka ClassNotFound
body: <p>I am trying to use Spark 2.3.1 structured streaming with Kafka. Getting the following error:</p>
  
  <pre><code>java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at http://spark.apache.org/third-party-projects.html
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:164)
    ... 49 elided
  Caused by: java.lang.ClassNotFoundException: kafka.DefaultSource
    at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
    at scala.util.Try.orElse(Try.scala:84)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
    ... 51 more
  </code></pre>
  
  <p>Request advise.</p>
  
  <p><strong>Scala code (using IntelliJ 2018.1):</strong></p>
  
  <pre><code>import org.apache.log4j.Logger._
  import org.apache.spark.sql.SparkSession
  
  object test {
  
    def main(args: Array[String]): Unit = {
      println("test3")
      import org.apache.log4j._
      getLogger("org").setLevel(Level.ERROR)
      getLogger("akka").setLevel(Level.ERROR)
  
      val spark = SparkSession.
        builder.
        master("local").
        appName("StructuredNetworkWordCount").
        getOrCreate()
  
      import spark.implicits._
  
      val lines = spark
        .readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", "localhost:9092")
        .option("subscribe", "t_tweets")
        .load()
  
      lines.selectExpr("CAST(value AS STRING)")
        .as[(String)]
  
      // Split the lines into words
      val words = lines.as[String].flatMap(_.split(" "))
  
      // Generate running word count
      val wordCounts = words.groupBy("value").count()
  
      val query = wordCounts.writeStream
        .outputMode("complete")
        .format("console")
        .start()
  
      query.awaitTermination()
    }
  }
  </code></pre>
  
  <p><strong>Build.sbt :</strong></p>
  
  <pre><code>name := "scalaSpark3"
  
  version := "0.1"
  
  scalaVersion := "2.11.8"
  
  libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.1"
  libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.1"
  
  // https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10
  libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.3.1" % "provided"
  
  // https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients
  libraryDependencies += "org.apache.kafka" % "kafka-clients" % "1.1.0"
  
  // https://mvnrepository.com/artifact/org.apache.spark/spark-streaming
  libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.3.1" % "provided"
  </code></pre>
  
  <p><strong>Full error log:</strong></p>
  
  <pre><code>objc[7301]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/bin/java (0x10c6334c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10c6bf4e0). One of the two will be used. Which one is undefined.
  test3
  
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: kafka. Please find packages at http://spark.apache.org/third-party-projects.html
      at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:635)
      at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:159)
      at test$.main(test.scala:33)
      at test.main(test.scala)
  
  Caused by: java.lang.ClassNotFoundException: kafka.DefaultSource
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23$$anonfun$apply$15.apply(DataSource.scala:618)
      at scala.util.Try$.apply(Try.scala:192)
      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
      at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$23.apply(DataSource.scala:618)
      at scala.util.Try.orElse(Try.scala:84)
      at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:618)
      ... 3 more
  </code></pre>
  
  <p>My code is based the sample code here:
  <a href="https://spark.apache.org/docs/2.3.1/structured-streaming-programming-guide.html" rel="nofollow noreferrer">https://spark.apache.org/docs/2.3.1/structured-streaming-programming-guide.html</a>
  <a href="https://spark.apache.org/docs/2.3.1/structured-streaming-kafka-integration.html" rel="nofollow noreferrer">https://spark.apache.org/docs/2.3.1/structured-streaming-kafka-integration.html</a></p>