title: Upgraded the spark version, and during spark jobs encountering java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
body: <p>We recently made an upgrade from Spark 2.4.2 to 2.4.5 for our ETL project.</p>
  <p>After deploying the changes, and running the job I am seeing the following error:</p>
  <pre><code>    Exception in thread &quot;main&quot; java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
      at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
  Caused by: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
      at com.advisory.pic.etl.utils.OracleDialect$.&lt;init&gt;(OracleDialect.scala:12)
      at com.advisory.pic.etl.utils.OracleDialect$.&lt;clinit&gt;(OracleDialect.scala)
      at com.advisory.pic.etl.drivers.BaseDriver.$init$(BaseDriver.scala:19)
      at com.advisory.pic.etl.drivers.PASLoadDriver$.&lt;init&gt;(PASLoadDriver.scala:19)
      at com.advisory.pic.etl.drivers.PASLoadDriver$.&lt;clinit&gt;(PASLoadDriver.scala)
      at com.advisory.pic.etl.drivers.PASLoadDriver.main(PASLoadDriver.scala)
      ... 6 more
  </code></pre>
  <p>I read online that it could be possible due to library version mismatch, but I could not find any such violations in our build.gradle, except for <strong>testImplementation</strong>, which I upgraded to correct version, but I doubt this is the root cause of the issue.</p>
  <p>Following is a snippet of dependencies in build.gradle file.</p>
  <pre><code>dependencies {
      def hadoopClientVersion = '2.7.1'
      def hadoopCommonsVersion = '2.7.1'
      def sparkVersion = '2.4.5'
      def sparkTestingVersion = '2.4.5'
  
  
  
      provided group: 'org.apache.hadoop', name: 'hadoop-client', version: hadoopClientVersion
      provided group: 'org.apache.hadoop', name: 'hadoop-common', version: hadoopCommonsVersion
  
      implementation(&quot;org.apache.spark:spark-sql_2.12:$sparkVersion&quot;){
          //excluding is causing issues when running through IDE - as sparkLibraries are not available at run-time
          //We can comment while going for deployment if we face jar conflict issues
          //exclude module: 'spark-core_2.10'
          //exclude module: 'spark-catalyst_2.10'
      }
      implementation group: 'org.apache.spark', name: 'spark-core_2.12', version: sparkVersion
      testImplementation group: 'org.apache.spark', name: 'spark-core_2.12', version: sparkVersion
  
      // spark -sql with avro
      implementation(&quot;com.databricks:spark-avro_2.11:4.0.0&quot;)
  
      // joda-time
      implementation 'com.github.nscala-time:nscala-time_2.12:2.22.0'
  
      //configuration object
      implementation group: 'com.typesafe', name: 'config', version: '1.2.1'
  
      implementation &quot;ch.qos.logback:logback-classic:1.1.3&quot;
      implementation &quot;org.slf4j:log4j-over-slf4j:1.7.13&quot;
  
  
  
      // Libraries needed for scala api
      implementation 'org.scala-lang:scala-library:2.12.0'
      implementation 'org.scala-lang:scala-compiler:2.12.0'
  
  
      testImplementation 'org.scalatest:scalatest_2.12:3.0.5'
  
      implementation 'com.oracle:ojdbc7:12.1.0.1'
  
      testImplementation group: 'com.h2database', name: 'h2', version: '1.4.196'
  
      testImplementation 'com.holdenkarau:spark-testing-base_2.12:' + sparkTestingVersion + '_0.12.0'
      itestCompile 'org.scala-lang:scala-library:2.12.0'
      itestCompile 'org.scalatest:scalatest_2.12:3.0.5'
      testImplementation 'org.scalamock:scalamock_2.12:4.3.0'
  }
  </code></pre>
  <p>Any suggestions on why the above issue is happening, and how can I verify the version mismatch?</p>