title: Spark Scala: Problems with sbt package/spark-submit/java.lang.ClassNotFoundException
body: <p>Others have posted questions before about creating the canonical Simple App application from the Scala-Spark Quick Start guide.  I can't get the jars created to include to the class I've tried create or they can't recognize the Spark Context objects I'm trying to import.  Everything I've tried has come from the suggestions made for similar inquiries about Spark Scala on Stack Overflow.  </p>
  
  <p>In the directory where I have Spark installed, I set up the following directory structure per the instructions on the Apache Spark website and general sbt guidelines:</p>
  
  <pre><code>simple.sbt
  \src
  \src\main\scala\
  \src\main\scala\SimpleApp.scala
  </code></pre>
  
  <p>This is the code for the SimpleApp.scala file. </p>
  
  <pre><code>/* SimpleApp.scala */
  import org.apache.spark.SparkContext
  import org.apache.spark.SparkContext._
  import org.apache.spark.SparkConf
  
  
  object SimpleApp {
    def main(args: Array[String]) {
    val logFile = "/Users/sidneygivens/spark-1.4.1/README.md" 
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line =&gt; line.contains("a")).count()
    val numBs = logData.filter(line =&gt; line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
    }
  }
  </code></pre>
  
  <p>This is the structure of the simple.sbt file:</p>
  
  <pre><code>name := "Simple Project"
  
  version := "1.0"
  
  scalaVersion := "2.10.4"
  
  libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"
  
  resolvers += "bintray-spark-packages" at "https://dl.bintray.com/spark-   packages/maven/"
  
  resolvers += "Akka Repository" at "http://repo.akka.io/releases/"
  </code></pre>
  
  <p>When I run sbt package from the directory where I have spark installed, I get error messages indicating a problem with "import" commands for the SparkContext. </p>
  
  <pre><code>[error] /Users/sidneygivens/src/main/scala/SimpleApp.scala:4: object apache  is not a member of package org
  [error] import org.apache.spark.SparkContext
  </code></pre>
  
  <p>If I go down to the directory src/main, I can get the sbt package command to create a jar.  However, it is not creating the SimpleApp classes. </p>
  
  <p>In previous questions posted on this issue, it has been suggested that this command should be run to find the class name.  </p>
  
  <pre><code>jar tf main_2.10-0.1-SNAPSHOT.jar
  </code></pre>
  
  <p>This is all that is returned.  </p>
  
  <pre><code>META-INF/MANIFEST.MF
  </code></pre>
  
  <p>No classes are present as is expected from a .jar build. </p>
  
  <p>As expected, an attempt to run a variant of the suggested spark-submit command from the Apache Spark quick start guide on the .jar file I managed to create does not work.</p>
  
  <pre><code>/Users/sidneygivens/spark-1.4.1/bin/spark-submit \
  --class "SimpleApp" \
  --master local[1] \
  /Users/sidneygivens/spark-1.4.1/src/main/target/scala-2.10/main_2.10-0.1-SNAPSHOT.jar
  </code></pre>
  
  <p>Gives me a ClassNotFound exception:</p>
  
  <pre><code>java.lang.ClassNotFoundException: SimpleApp
      at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  </code></pre>
  
  <p>Everything I've tried seems to conform with the Spark/sbt boilerplate, but I'm obviously doing something wrong.  Where should be running sbt pcakage from? Is there something wrong with my Spark/sbt setup? </p>
  
  <p>Here's the full output of the sbt package call: </p>
  
  <pre><code>[info] Set current project to sparkscala (in build      file:/Users/sidneygivens/spark-1.4.1/sparkscala/)
  [info] Compiling 1 Scala source to /Users/sidneygivens/spark-    1.4.1/sparkscala/target/scala-2.10/classes...
  [error] /Users/sidneygivens/spark-1.4.1/sparkscala/src/main/scala/SimpleApp.scala:4: object apache is not a member of package org
  [error] import org.apache.spark._
  [error]            ^
  [error] /Users/sidneygivens/spark- 1.4.1/sparkscala/src/main/scala/SimpleApp.scala:10: not found: type SparkConf
  [error]     val conf = new SparkConf().setAppName("Simple App")
  [error]                    ^
  [error] /Users/sidneygivens/spark-1.4.1/sparkscala/src/main/scala/SimpleApp.scala:11: not found: type SparkContext
  [error]     val sc = new SparkContext(conf)
  [error]                  ^
  [error] three errors found
  [error] (compile:compileIncremental) Compilation failed
  [error] Total time: 2 s, completed Aug 23, 2015 11:30:22 PM
  </code></pre>