title: Version Compatibility in Spark and Hbase-client
body: <p>I am trying to write a Spark batch job. I want to package it to a jar and use it with spark submit. My program works perfectly in spark-shell, however I get the following error when I try to run it with spark submit: </p>
  
  <pre><code>Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
      at HBaseBulkload$.saveAsHFile(ThereInLocationGivenTimeInterval.scala:103)
      at HBaseBulkload$.toHBaseBulk(ThereInLocationGivenTimeInterval.scala:178)
      at ThereInLocationGivenTimeInterval$.main(ThereInLocationGivenTimeInterval.scala:241)
      at ThereInLocationGivenTimeInterval.main(ThereInLocationGivenTimeInterval.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p>According to <a href="https://stackoverflow.com/a/30893383/4668959">this answer</a> the problem originates from version incompatibility. I also found <a href="https://community.hortonworks.com/questions/61968/spark-hbase-connector-for-spark-161-version.html" rel="nofollow noreferrer">this</a>, but my spark version is 1.6.0 Here is my .sbt file for the project:</p>
  
  <pre><code>name := "HbaseBulkLoad"
  
  version := "1.0"
  
  scalaVersion := "2.10.5"
  
  resolvers += "Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/"
  
  libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0"
  //libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.2.0-cdh5.9.0"
  //libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.2.0-cdh5.9.0"
  //libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.2.0-cdh5.9.0"
  
  libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.1.2"
  libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.1.2"
  libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.1.2"
  </code></pre>
  
  <p>My imports and the code segment that causes the error is as follows:
      /<em>SimpleApp.scala</em>/
      import org.apache.spark.SparkContext
      import org.apache.spark.SparkContext._
      import org.apache.spark.SparkConf</p>
  
  <pre><code>// HBaseBulkLoad imports
  import java.util.UUID
  
  import org.apache.hadoop.conf.Configuration
  import org.apache.hadoop.fs.permission.FsPermission
  import org.apache.hadoop.fs.{Path, FileSystem}
  import org.apache.hadoop.hbase.{KeyValue, TableName}
  import org.apache.hadoop.hbase.client._
  import org.apache.hadoop.hbase.io.ImmutableBytesWritable
  import org.apache.hadoop.hbase.mapreduce.{HFileOutputFormat2, LoadIncrementalHFiles}
  import org.apache.hadoop.hbase.util.Bytes
  import org.apache.hadoop.mapreduce.Job
  import org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner
  import org.apache.spark.rdd.RDD
  import org.apache.spark.Partitioner
  import org.apache.spark.storage.StorageLevel
  
  import scala.collection.JavaConversions._
  import scala.reflect.ClassTag
  
  // Hbase admin imports
  import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
  import org.apache.hadoop.hbase.client.HBaseAdmin
  import org.apache.hadoop.hbase.mapreduce.TableInputFormat
  import org.apache.hadoop.fs.Path;
  import org.apache.hadoop.hbase.HColumnDescriptor
  import org.apache.hadoop.hbase.util.Bytes
  import org.apache.hadoop.hbase.client.Put;
  import org.apache.hadoop.hbase.client.HTable;
  import java.util.Calendar
  
  val now = Calendar.getInstance.getTimeInMillis    
  //val filteredRdd = myRdd.filter(...
  val resultRdd= filteredRdd.map{ row =&gt;  (row(0).asInstanceOf[String].getBytes(), 
                                  scala.collection.immutable.Map("batchResults" -&gt; 
                                          Array( ( "batchResult1", ("true", now) ) ) 
                                  ) 
                          )
          }
  println( resultRdd.count )
  </code></pre>