title: NoSuchMethodError on apache spark. Sbt dispatcher library not found
body: <p>Currently, when i do a spark-submit of a jar file i am getting the following error:</p>
  
  <pre><code>Exception in thread "streaming-job-executor-0" java.lang.NoSuchMethodError: io.netty.handler.ssl.SslContextBuilder.protocols([Ljava/lang/String;)Lio/netty/handler/ssl/SslContextBuilder;
  at org.asynchttpclient.netty.ssl.DefaultSslEngineFactory.buildSslContext(DefaultSslEngineFactory.java:45)
  at org.asynchttpclient.netty.ssl.DefaultSslEngineFactory.init(DefaultSslEngineFactory.java:69)
  at org.asynchttpclient.netty.channel.ChannelManager.&lt;init&gt;(ChannelManager.java:116)
  at org.asynchttpclient.DefaultAsyncHttpClient.&lt;init&gt;(DefaultAsyncHttpClient.java:85)
  at dispatch.Http.client$lzycompute(execution.scala:16)
  at dispatch.Http.client(execution.scala:16)
  at dispatch.Http.client(execution.scala:11)
  at dispatch.HttpExecutor$class.apply(execution.scala:120)
  at dispatch.Http.apply(execution.scala:11)
  at dispatch.HttpExecutor$class.apply(execution.scala:115)
  at dispatch.Http.apply(execution.scala:11)
  at com.testing.streamstest$$anonfun$lookupHostNames$1$1.apply(streamstest.scala:121)
  at com.testing.streamstest$$anonfun$lookupHostNames$1$1.apply(streamstest.scala:111)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
  at com.testing.streamstest$.lookupHostNames$1(streamstest.scala:111)
  at com.testing.streamstest$.com$testing$streamstest$$processLine$1(streamstest.scala:169)
  at com.testing.streamstest$$anonfun$main$1.apply(streamstest.scala:221)
  at com.testing.streamstest$$anonfun$main$1.apply(streamstest.scala:221)
  at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)
  at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
  at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)
  at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:415)
  at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)
  at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
  at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)
  at scala.util.Try$.apply(Try.scala:192)
  at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)
  at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:254)
  at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
  at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:254)
  at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
  at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:253)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <p>I feel that it is the library dependencies that I am using that are causing the netty errors (most likely the dispatch library, but I tried using different versions of the library and this same error still occur). </p>
  
  <p>The library dependencies i am currently using are:</p>
  
  <pre><code>libraryDependencies += "net.databinder.dispatch" %% "dispatch-core"   % "0.13.2" 
  libraryDependencies += "ch.qos.logback"  %  "logback-classic" % "1.2.3"
  libraryDependencies += "dnsjava" % "dnsjava" % "2.1.8"
  </code></pre>
  
  <p>Are there any way to solve this error and make the spark job run?</p>
  
  <p><s>
  <strong>Edit:</strong>
  After testing around for a bit, i found out that there was an error when installing <code>libraryDependencies += "net.databinder.dispatch" %% "dispatch-core"   % "0.13.2"</code> using the tutorial stated under the SBT section in <a href="https://dispatchhttp.org/Dispatch.html" rel="nofollow noreferrer">dispatchhttp.org/Dispatch.html</a></p>
  
  <p>This was the error after running <code>sbt console</code> command (This was ran in Ubuntu 16.04):</p>
  
  <pre><code>sbt.ResolveException: unresolved dependency: net.databinder.dispatch#dispatch-core_2.10;0.13.2: not found
  </code></pre>
  
  <p>Wondering if there was a problem with the library dependencies version for dispatch in sbt?</p>
  
  <p><strong>Edit 2:</strong></p>
  
  <p>Here is the entire build.sbt file as requested:</p>
  
  <pre><code>name := "test"
  
  version := "1.0"
  
  scalaVersion := "2.11.8"
  
  libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" % "provided"
  libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" % "provided"
  libraryDependencies += "org.apache.spark" % "spark-streaming_2.11" % "2.1.0" % "provided"
  libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
  libraryDependencies += "org.scalaj" % "scalaj-http_2.11" % "2.3.0"
  libraryDependencies += "dnsjava" % "dnsjava" % "2.1.8"
  libraryDependencies += "ch.qos.logback"  %  "logback-classic" % "1.2.3"    
  libraryDependencies += "net.databinder.dispatch" %% "dispatch-core"   % "0.13.2"
  
  
  
  
  
  assemblyMergeStrategy in assembly := {
   case PathList("META-INF", xs @ _*) =&gt; MergeStrategy.discard
   case x =&gt; MergeStrategy.first
  }
  </code></pre>
  
  <p><strong>Edit 3:</strong>
  I went ahead and did a <code>show update</code> and this result was shown:</p>
  
  <pre><code>&gt;show update
  [info]  compile:
  [info]  io.netty:netty-all
  [info]      - 4.0.51.Final
  [info]          status: release
  [info]          publicationDate: Thu Aug 24 20:26:14 WIB 2017
  [info]          resolver: sbt-chain
  [info]          artifactResolver: sbt-chain
  [info]          evicted: false
  [info]          isDefault: false
  [info]          configurations: default, compile, runtime, default(compile), master
  [info]          licenses: (Apache License, Version 2.0,Some(http://www.apache.org/licenses/LICENSE-2.0))
  [info]          callers: streamingserver:streamingserver_2.11:1.0
  ...
  [info]  io.netty:netty
  [info]      - 3.8.0.Final
  [info]          status: release
  [info]          publicationDate: Thu Nov 07 16:23:12 WIB 2013
  [info]          resolver: sbt-chain
  [info]          artifactResolver: sbt-chain
  [info]          evicted: false
  [info]          homepage: http://netty.io/
  [info]          isDefault: false
  [info]          configurations: compile, runtime(*), master(compile), runtime, compile(*), master
  [info]          licenses: (Apache License, Version 2.0,Some(http://www.apache.org/licenses/LICENSE-2.0))
  [info]          callers: org.apache.spark:spark-core_2.11:2.1.0
  </code></pre>
  
  <p>The version 3.8.0 of netty was not evicted for some reason, maybe this was the cause of the error? if so, how can it be evicted so only the latest version remain? (Was it due to the MergeStrategy by any chance?)</p>
  
  <p>Or was it the problem with cloudera netty? went to see the netty versions in classpath.txt and this was what i got:</p>
  
  <pre><code>&gt; cat classpath.txt | grep netty
  /opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/netty-3.10.5.Final.jar
  /opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/netty-3.9.4.Final.jar
  /opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/netty-all-4.0.23.Final.jar 
  </code></pre>
  
  <p></s></p>
  
  <p><strong>Final Edit</strong>
  I manage to "solve" this problem by switching from using <a href="http://dispatchhttp.org/Dispatch.html" rel="nofollow noreferrer">dispatchhttp</a> to using <a href="https://github.com/scalaj/scalaj-http" rel="nofollow noreferrer">scalaj-http</a> instead. Scalaj-http does what i want and i did not encounter any of the netty errors.</p>