title: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
body: <p>I want to create my first scala program using the scala example <a href="https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/HBaseTest.scala" rel="nofollow"><code>HBaseTest2.scala</code></a>, provided in <strong>Sparkd 1.4.1</strong>. The goal is to connect to HBase and do some basic stuff, such as counting rows or scan rows. However, when I tried to execute the program, I got an error. It seems that Spark couldn't find the class <code>HBaseConfiguration</code>. Assuming the we're located a the root path of my project HBaseTest2 <code>/usr/local/Cellar/spark/programs/HBaseTest2</code>. Here are some details for the exception :</p>
  
  <p><strong>./src/main/scala/com/orange/spark/examples/HBaseTest2.scala</strong>  </p>
  
  <pre><code>package com.orange.spark.examples
  
  import org.apache.hadoop.hbase.client.HBaseAdmin
  import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor, TableName}
  import org.apache.hadoop.hbase.mapreduce.TableInputFormat
  
  import org.apache.spark._
  
  
  object HBaseTest2 {
    def main(args: Array[String]) {
      val sparkConf = new SparkConf().setAppName("HBaseTest2")
      val sc = new SparkContext(sparkConf)
      val tableName = "personal-cloud-test"
  
      // please ensure HBASE_CONF_DIR is on classpath of spark driver
      // e.g: set it through spark.driver.extraClassPath property
      // in spark-defaults.conf or through --driver-class-path
      // command line option of spark-submit
  
      val conf = HBaseConfiguration.create()
  
      // Other options for configuring scan behavior are available. More information available at
      // http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/TableInputFormat.html
      conf.set(TableInputFormat.INPUT_TABLE, tableName)
  
      // Initialize hBase table if necessary
      val admin = new HBaseAdmin(conf)
      if (!admin.isTableAvailable(tableName)) {
        val tableDesc = new HTableDescriptor(TableName.valueOf(tableName))
        admin.createTable(tableDesc)
      }
  
      val hBaseRDD = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat],
        classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],
        classOf[org.apache.hadoop.hbase.client.Result])
  
      println("hbaseRDD.count()")
      println(hBaseRDD.count())
  
      sc.stop()
      admin.close()
    }
  }
  </code></pre>
  
  <p><strong>./build.sbt</strong><br>
  I've added dependencies in this file to ensure all classes called are included in the jar file.</p>
  
  <pre><code>name := "HBaseTest2"
  
  version := "1.0"
  
  scalaVersion := "2.11.7"
  
  libraryDependencies += "org.apache.spark" %% "spark-core" % "1.4.1"
  
  libraryDependencies ++= Seq(
      "org.apache.hadoop" % "hadoop-core" % "1.2.1",
      "org.apache.hbase" % "hbase" % "1.0.1.1",
      "org.apache.hbase" % "hbase-client" % "1.0.1.1",
      "org.apache.hbase" % "hbase-common" % "1.0.1.1",
      "org.apache.hbase" % "hbase-server" % "1.0.1.1"
  )
  </code></pre>
  
  <p><strong>Run application</strong></p>
  
  <pre><code>MacBook-Pro-de-Mincong:spark-1.4.1 minconghuang$ bin/spark-submit \
    --class "com.orange.spark.examples.HBaseTest2" \
    --master local[4] \ 
    ../programs/HBaseTest2/target/scala-2.11/hbasetest2_2.11-1.0.jar
  </code></pre>
  
  <p><strong>Exception</strong></p>
  
  <pre><code>15/08/18 12:06:17 INFO storage.BlockManagerMaster: Registered BlockManager
  Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
      at com.orange.spark.examples.HBaseTest2$.main(HBaseTest2.scala:21)
      at com.orange.spark.examples.HBaseTest2.main(HBaseTest2.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 11 more
  15/08/18 12:06:17 INFO spark.SparkContext: Invoking stop() from shutdown hook
  </code></pre>
  
  <p>The problem might come from the HBase configuration as mentioned in <code>HBaseTest2.scala</code> line 16 :  </p>
  
  <blockquote>
    <p>// please ensure HBASE_CONF_DIR is on classpath of spark driver<br>
        // e.g: set it through spark.driver.extraClassPath property<br>
        // in spark-defaults.conf or through --driver-class-path<br>
        // command line option of spark-submit</p>
  </blockquote>
  
  <p>But I don't know how to configure it... I've added the <code>HBASE_CONF_DIR</code> to <code>CLASSPATH</code> in my command line. The CLASSPATH is now <code>/usr/local/Cellar/hadoop/hbase-1.0.1.1/conf</code>. Nothing happened... T_T So what should I do to get this fixed ? I can add/delete details if needed. Thanks a lot !!</p>