title: Corrupt Decimal value when querying a hive on parquet table from spark
body: <p>Spark is returning garbage/incorrect values for decimal fields when querying an external hive table on parquet in Spark code using Spark SQL.</p>
  <p>In my application flow, a spark process originally writes data to these parquet files directly into HDFS on which external Hive table exists. Incorrect data is fetched when the second Spark process is trying to consume from Hive table using Spark-SQL.</p>
  <p>Scenario steps: This is a simple demo reproducing the issue:</p>
  <ol>
  <li><p>Write to Parquet: I am writing data to parquet file in HDFS, Spark itself assumes precision for decimal fields as <code>Decimal(28,26)</code>.</p>
  <pre><code>scala&gt; val df = spark.sql(&quot;select 'dummy' as name, 10.70000000000000000000000000 as value&quot;)
  df: org.apache.spark.sql.DataFrame = [name: string, value: decimal(28,26)]
  scala&gt; df.schema
  res0: org.apache.spark.sql.types.StructType = StructType(StructField(name,StringType,false), StructField(value,DecimalType(28,26),false))
  scala&gt; df.show
  +-----+--------------------+
  | name|               value|
  +-----+--------------------+
  |dummy|10.70000000000000...|
  +-----+--------------------+
  scala&gt; df.write.option(&quot;overwrite&quot;,true).parquet(&quot;/my/hdfs/location/test&quot;)
  </code></pre>
  </li>
  <li><p>Read parquet file: to see if value is correctly written.</p>
  <pre><code>scala&gt; val df_parq = spark.read.option(&quot;spark.sql.decimalOperations.allowPrecisionLoss&quot;,false).parquet(&quot;/tenants/gwm/morph/test/tablePrecisionTest/test&quot;)
  df_parq: org.apache.spark.sql.DataFrame = [name: string, value: decimal(28,26)]
  scala&gt; df_parq.show
  +-------+--------------------+
  |   name|               value|
  +-------+--------------------+
  |  dummy|10.70000000000000...|
  +-------+--------------------+ 
  </code></pre>
  </li>
  <li><p>Create external hive table: on top of parquet location with Decimal field as <code>Decimal(18,6)</code>.</p>
  <pre><code>hive&gt; create external table db1.test_precision(name string, value Decimal(18,6)) STORED As PARQUET LOCATION '/my/hdfs/location/test';
  </code></pre>
  </li>
  <li><p>Run Hive query in beeline: to verify that correct data is returned.</p>
  <pre><code>hive&gt; select * from db1.test_precision;
  +----------------------+-----------------------+--+
  | test_precision.name  | test_precision.value  |
  +----------------------+-----------------------+--+
  | dummy                | 10.7                  |
  +----------------------+-----------------------+--+
  </code></pre>
  </li>
  <li><p>Run same query using Spark Sql: Incorrect decimal values are produced.</p>
  <pre><code>scala&gt; val df_hive = spark.sql(&quot;select * from db1.test_precision&quot;)
  df_hive: org.apache.spark.sql.DataFrame = [name: string, value: decimal(18,6)]
  scala&gt; df_hive.show
  +-----+-----------+
  | name|      value|
  +-----+-----------+
  |dummy|-301.989888|
  +-----+-----------+
  </code></pre>
  </li>
  </ol>
  <p><strong>Note -</strong> I am aware that storing the value to parquet with an explicit <code>cast(value as Decima(18,6))</code> on first step can fix the issue, but I already have historical data that I can't reload right away.</p>
  <p>Is there a way I can fix this while reading the value at step 5?</p>