title: Spark not providing the updated results as available in Hive
body: <p>I am using a pyspark code to process some data which is fetched from Hive View. We are currently using HDP 3.1 Version. The issue is i can see that Hive View contains data, for example: Data for date 5th June 2019, where as when i try to fetch the data from Spark SQL, I get the data latest of date 4th June 2019. </p>
  
  <p>I have tried selecting complete data in spark and then ordering it by Date DESC just to be sure about the issue. I have also checked that the data for 5th June is available in Hive.</p>
  
  <p>I came to know that spark and Hive manages different catalog in HDP 3+. But as per my knowledge, Admin has already taken care of pointing Spark to Hive Catalog. Is it the catalog issue, that i need to check again?</p>
  
  <p><strong>Update:</strong>
  Ask asked in comment , here is the sample code i ran in Hive and spark with sample results i got - </p>
  
  <p><strong>Hive Query</strong></p>
  
  <pre><code>select number,v_date, location, region  from View1 where Upper(category) ="I" and lower(region) ="cochin" and to_date(visit_date) between "2019-06-02" AND "2019-06-06"  order by  to_date(visit_date) desc limit 500;
  </code></pre>
  
  <p><strong>Sample Result:</strong></p>
  
  <p><a href="https://i.stack.imgur.com/cXObY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cXObY.png" alt="enter image description here"></a></p>
  
  <p>As shown above, I get results for 5th June in Hive. Now when i execute the same in <strong>PySpark:</strong></p>
  
  <pre><code>data_for_region = sparkSession.sql('select number, v_date, location, region  from View1 where Upper(category) ="I" and lower(region) ="cochin" and date_format(visit_date,"yyyy-MM-dd") between "2019-06-02" AND "2019-06-06" order by visit_date desc limit 50')
  </code></pre>
  
  <p><strong>Sample Result</strong> </p>
  
  <p><a href="https://i.stack.imgur.com/fR2PV.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fR2PV.png" alt="enter image description here"></a></p>
  
  <p>As you can see, Its just not showing me the results for 5th June , but giving the data from 4th June. I am just not aware of how to resolve this discrepancy.</p>