title: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V
body: <p>It looks like I am again stuck on the running a packaged spark app jar using spark submit. Following is my pom file:</p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;
      &lt;parent&gt;
          &lt;artifactId&gt;oneview-forecaster&lt;/artifactId&gt;
          &lt;groupId&gt;com.dataxu.oneview.forecast&lt;/groupId&gt;
          &lt;version&gt;1.0.0-SNAPSHOT&lt;/version&gt;
      &lt;/parent&gt;
      &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
      &lt;artifactId&gt;forecaster&lt;/artifactId&gt;
  
  &lt;dependencies&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.fasterxml.jackson.core&lt;/groupId&gt;
          &lt;artifactId&gt;jackson-databind&lt;/artifactId&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.fasterxml.jackson.module&lt;/groupId&gt;
          &lt;artifactId&gt;jackson-module-scala_${scala.binary.version}&lt;/artifactId&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
          &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
          &lt;version&gt;${scala.version}&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-streaming_${scala.binary.version}&lt;/artifactId&gt;
          &lt;version&gt;${spark.version}&lt;/version&gt;
          &lt;scope&gt;provided&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-sql_${scala.binary.version}&lt;/artifactId&gt;
          &lt;version&gt;${spark.version}&lt;/version&gt;
          &lt;!--&lt;scope&gt;provided&lt;/scope&gt;--&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive --&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-hive_2.11&lt;/artifactId&gt;
          &lt;version&gt;2.2.0&lt;/version&gt;
          &lt;!--&lt;scope&gt;provided&lt;/scope&gt;--&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
          &lt;artifactId&gt;hadoop-aws&lt;/artifactId&gt;
          &lt;version&gt;2.8.3&lt;/version&gt;
          &lt;!--&lt;scope&gt;provided&lt;/scope&gt;--&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
          &lt;artifactId&gt;aws-java-sdk&lt;/artifactId&gt;
          &lt;version&gt;1.10.60&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/joda-time/joda-time --&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;joda-time&lt;/groupId&gt;
          &lt;artifactId&gt;joda-time&lt;/artifactId&gt;
          &lt;version&gt;2.9.9&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common --&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
          &lt;artifactId&gt;hadoop-common&lt;/artifactId&gt;
          &lt;version&gt;2.8.0&lt;/version&gt;
          &lt;!--&lt;scope&gt;provided&lt;/scope&gt;--&gt;
      &lt;/dependency&gt;
  &lt;/dependencies&gt;
  
  &lt;build&gt;
      &lt;sourceDirectory&gt;src/main/scala&lt;/sourceDirectory&gt;
      &lt;testSourceDirectory&gt;src/test/scala&lt;/testSourceDirectory&gt;
      &lt;plugins&gt;
          &lt;plugin&gt;
              &lt;groupId&gt;net.alchim31.maven&lt;/groupId&gt;
              &lt;artifactId&gt;scala-maven-plugin&lt;/artifactId&gt;
              &lt;version&gt;${scala-maven-plugin.version}&lt;/version&gt;
              &lt;executions&gt;
                  &lt;execution&gt;
                      &lt;goals&gt;
                          &lt;goal&gt;compile&lt;/goal&gt;
                          &lt;goal&gt;testCompile&lt;/goal&gt;
                      &lt;/goals&gt;
                  &lt;/execution&gt;
              &lt;/executions&gt;
          &lt;/plugin&gt;
          &lt;plugin&gt;
              &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
              &lt;configuration&gt;
                  &lt;archive&gt;
                      &lt;manifest&gt;
                          &lt;mainClass&gt;com.dataxu.oneview.forecaster.App&lt;/mainClass&gt;
                      &lt;/manifest&gt;
                  &lt;/archive&gt;
                  &lt;descriptorRefs&gt;
                      &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
                  &lt;/descriptorRefs&gt;
              &lt;/configuration&gt;
              &lt;executions&gt;
                  &lt;execution&gt;
                      &lt;id&gt;make-assembly&lt;/id&gt;
                      &lt;phase&gt;package&lt;/phase&gt;
                      &lt;goals&gt;
                          &lt;goal&gt;single&lt;/goal&gt;
                      &lt;/goals&gt;
                  &lt;/execution&gt;
              &lt;/executions&gt;
          &lt;/plugin&gt;
      &lt;/plugins&gt;
  &lt;/build&gt;
  </code></pre>
  
  <p></p>
  
  <p>Following is a simple snippet of code which fetches data from s3 location and prints it:</p>
  
  <pre><code>def getS3Data(path: String): Map[String, Any] = {
      println("spark session start.........")
      val spark =  getSparkSession()
  
      val configTxt = spark.sparkContext.textFile(path)
          .collect().reduce(_ + _)
  
      val mapper = new ObjectMapper
      mapper.registerModule(DefaultScalaModule)
      mapper.readValue(configTxt, classOf[Map[String, String]])
  }
  </code></pre>
  
  <p>When I run it from intellij, everything works fine. the log is clear and looks good. However, when I package it using mvn package and try to run it using spark submit, I end up getting the following error at the <code>.collect.reduce(_ + _)</code>. Following is the error I encounter:</p>
  
  <pre><code> "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V
  at org.apache.hadoop.fs.s3a.S3AFileSystem.addDeprecatedKeys(S3AFileSystem.java:181)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.&lt;clinit&gt;(S3AFileSystem.java:185)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:348)
  ...
  </code></pre>
  
  <p>I am not understanding which dependency was not packaged or what might be the issue as I did set the versions correctly expecting the hadoop aws should have all of them.</p>
  
  <p>Any help will be appreciated.</p>