title: SparkSQL key not found: scale
body: <p>Spark version is 1.6.0.</p>
  
  <p>I trying to do a simple SQL query to the remote Oracle 11g DB by using Spark SQL. </p>
  
  <p>Of course the ojdbc driver is added to the classpath and ping to the DB is also ok.</p>
  
  <pre><code>SparkConf conf = new SparkConf().setAppName(APP_NAME).setMaster("yarn-client");
  JavaSparkContext jsc = new JavaSparkContext(conf);
  SqlContext sqlContext = new SqlContext(jsc );
  
  Map&lt;String, String&gt; connectionProperties = new HashMap&lt;&gt;();
  connectionProperties.put("user", username);
  connectionProperties.put("password", password);
  connectionProperties.put("url", url);
  connectionProperties.put("dbtable", "(SELECT * FROM tableName)");
  connectionProperties.put("driver", "oracle.jdbc.OracleDriver");
  
  DataFrame result = sqlContext.read().format("jdbc").options(connectionProperties).load();
  </code></pre>
  
  <p>The error arise at the last line at <code>.load()</code> method.</p>
  
  <p>The resulted stacktrace is:</p>
  
  <pre><code>Exception in thread "main" java.util.NoSuchElementException: key not found: scale
      at scala.collection.MapLike$class.default(MapLike.scala:228)
      at scala.collection.AbstractMap.default(Map.scala:58)
      at scala.collection.MapLike$class.apply(MapLike.scala:141)
      at scala.collection.AbstractMap.apply(Map.scala:58)
      at org.apache.spark.sql.types.Metadata.get(Metadata.scala:108)
      at org.apache.spark.sql.types.Metadata.getLong(Metadata.scala:51)
      at org.apache.spark.sql.jdbc.OracleDialect$.getCatalystType(OracleDialect.scala:33)
      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:140)
      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.&lt;init&gt;(JDBCRelation.scala:91)
      at org.apache.spark.sql.execution.datasources.jdbc.DefaultSource.createRelation(DefaultSource.scala:57)
      at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
      at myapp.dfComparator.entity.OriginalSourceTable.load(OriginalSourceTable.java:74)
      at myapp.dfComparator.Program.main(Program.java:74)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p>I have no mind, what is wrong.</p>
  
  <p><strong>EDIT 01/27/2017</strong></p>
  
  <p>Additional information:</p>
  
  <p>Hadoop version is 2.6.0-cdh.5.8.3</p>
  
  <p>Spark version is 1.6.0 with Scala version 2.10.5</p>
  
  <p>I trying to reproduce the code above in scala and execute it using <code>spark-shell</code>:</p>
  
  <pre><code>val jdbcDF = sqlContext.read.format("jdbc").options(
  Map("url" -&gt; "jdbc:oracle:thin:system/system@db-host:1521:orcl", 
  "dbtable"-&gt; "schema_name.table_name", 
  "driver"-&gt; "oracle.jdbc.OracleDriver", 
  "username" -&gt; "user", 
  "password" -&gt; "pwd")).load()
  </code></pre>
  
  <p>The result of this code is the similiar stacktrace:</p>
  
  <pre><code>java.util.NoSuchElementException: key not found: scale
      at scala.collection.MapLike$class.default(MapLike.scala:228)
      at scala.collection.AbstractMap.default(Map.scala:58)
      at scala.collection.MapLike$class.apply(MapLike.scala:141)
      at scala.collection.AbstractMap.apply(Map.scala:58)
      at org.apache.spark.sql.types.Metadata.get(Metadata.scala:108)
      at org.apache.spark.sql.types.Metadata.getLong(Metadata.scala:51)
      at org.apache.spark.sql.jdbc.OracleDialect$.getCatalystType(OracleDialect.scala:33)
      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:140)
      at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.&lt;init&gt;(JDBCRelation.scala:91)
      at org.apache.spark.sql.execution.datasources.jdbc.DefaultSource.createRelation(DefaultSource.scala:57)
      at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:25)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:30)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:32)
      at $iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:34)
      at $iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:36)
      at $iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:38)
      at $iwC$$iwC.&lt;init&gt;(&lt;console&gt;:40)
      at $iwC.&lt;init&gt;(&lt;console&gt;:42)
      at &lt;init&gt;(&lt;console&gt;:44)
      at .&lt;init&gt;(&lt;console&gt;:48)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at .&lt;init&gt;(&lt;console&gt;:7)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at $print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
      at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
      at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
      at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
      at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
      at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
      at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
      at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
      at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
      at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
      at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
      at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
      at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
      at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
      at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
      at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
      at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)
      at org.apache.spark.repl.Main$.main(Main.scala:31)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p>So, I have strong perception that there is some mistakes in configuration of spark or(and) hadoop.</p>
  
  <p><strong>EDIT 02/01/2017</strong></p>
  
  <p>I investigate that such problem is arise only in case when the column type in oracle table is <code>NUMBER</code>. For example, if I cast the <code>id</code> column (which type is <code>NUMBER</code>) to <code>VARCHAR</code> in select statement, then all will works fine:</p>
  
  <pre><code>val jdbcDF = sqlContext.read.format("jdbc").options(
      Map("url" -&gt; "jdbc:oracle:thin:system/system@db-host:1521:orcl", 
      "dbtable"-&gt; "(SELECT CAST(id AS varchar(3))) FROM tableName", 
      "driver"-&gt; "oracle.jdbc.OracleDriver", 
      "username" -&gt; "user", 
      "password" -&gt; "pwd")).load()
  </code></pre>
  
  <p>In more detail - the staktrace show us, that problem is appear at <code>org.apache.spark.sql.types.Metadata.get</code> method. By investigate this method from <a href="https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Metadata.scala" rel="nofollow noreferrer">sources</a> we can see (or assume) that in case of <code>NUMBER</code> type it trying to cast it to Long and cannot find the scale of it.</p>
  
  <p>Thats why now I think that main issue is in cloudera distr of apache spark.</p>