title: How does computing table stats in hive or impala speed up queries in Spark SQL?
body: <p>For increasing performance (e.g. for joins) it is recommended to compute table statics first.  </p>
  
  <p>In Hive I can do::</p>
  
  <pre><code>analyze table &lt;table name&gt; compute statistics;
  </code></pre>
  
  <p>In Impala:</p>
  
  <pre><code>compute stats &lt;table name&gt;;
  </code></pre>
  
  <p>Does my spark application (reading from hive-tables) also benefit from pre-computed statistics? If yes, which one do I need to run? Are they both saving the stats in the hive metastore? I'm using spark 1.6.1 on Cloudera 5.5.4</p>
  
  <p>Note:
  In the Docs of spark 1.6.1 (<a href="https://spark.apache.org/docs/1.6.1/sql-programming-guide.html" rel="noreferrer">https://spark.apache.org/docs/1.6.1/sql-programming-guide.html</a>) for the parameter <code>spark.sql.autoBroadcastJoinThreshold</code> I found a hint: </p>
  
  <blockquote>
    <p>Note that currently statistics are only supported for Hive Metastore
    tables where the command ANALYZE TABLE  COMPUTE STATISTICS
    noscan has been run.</p>
  </blockquote>