title: Refresh cached dataframe?
body: <p>We have a small hive table (around 50000 records) which is updated once, daily. </p>
  
  <p>We have a cached Data-frame for this table and is being joined with spark streaming data. How do we refresh the data frame when new data is loaded in base hive?</p>
  
  <pre><code>DataFrame tempApp = hiveContext.table("emp_data");
  
  //Get Max Load-Date
  Date max_date =  max_date = tempApp.select(max("load_date")).collect()[0].getDate(0);
  
  //Get data for latest date and cache. This will be used to join with stream data.
  DataFrame emp= hiveContext.table("emp_data").where("load_date='" + max_date + "'").cache();
  
  // Get message from Kafka Stream
  JavaPairInputDStream&lt;String, String&gt; messages  = KafkaUtils.createDirectStream(....);
  
  JavaDStream&lt;MobileFlowRecord&gt; rddMobileFlorRecs = messages.map(Record::parseFromMessage);
  
  kafkaRecs.foreachRDD(rdd-&gt;{DataFrame recordDataFrame = hiveContext.createDataFrame(rdd, Record.class);
  
  DataFrame  joinedDataSet = recordDataFrame.join(emp, 
  recordDataFrame.col("application").equalTo(app.col("emp_id"));
  joinedDataSet. &lt;Do furthur processing&gt;
  });
  </code></pre>