title: sparksql drop hive table
body: <p>I want to drop a hive table through sparksql.</p>
  
  <p>In a cluster with hadoop 2.6, hive 2.0, spark 1.6 and spark 2.0 installed. I tried the following code in pyspark shell and spark-submit job with both version.</p>
  
  <pre><code>sqlContext.sql('drop table test')  //spark 1.6
  spark.sql('drop table test')       //spark 2.0
  </code></pre>
  
  <p>Both code works fine in pyspark-shell , I can see from the hive cli that the <strong><em>test</em></strong> table no longer exist.</p>
  
  <p>However, if the code was in a python file and later submitted to cluster using spark-submit, the code never took effect. </p>
  
  <p>spark 2.0 even gave error about </p>
  
  <pre><code>pyspark.sql.utils.AnalysisException: u"Table to drop '`try`' does not exist;"
  </code></pre>
  
  <p>I have copied <strong>hive-site.xml</strong> into the conf directory in spark.</p>
  
  <p>What would be the correct way to drop a hive table through sparksql?</p>
  
  <p>Update:</p>
  
  <p>I tried compared the spark environment between the spark-shell and the job I submitted using the following code</p>
  
  <pre><code>spark-submit --master yarn --deploy-mode cluster try_spark_sql.py
  </code></pre>
  
  <p>In the spark-shell environment, I can see <strong>spark.sql.catalogImplementation</strong>   is set to <strong>hive</strong></p>
  
  <p>IN the job submitted using the above code. The environment doesn't contain <strong>spark.sql.catalogImplementation</strong>
  I tried setting it using the following code:</p>
  
  <pre><code>spark = SparkSession.builder.appName("PythonSQL").config("spark.sql.catalogImplementation","hive").
  </code></pre>
  
  <p>But it doesn't have any effect on the environment.</p>
  
  <p>One workaround I found is submitting the job using <strong>client</strong> mode instead of <strong>cluster</strong> mode. Then the hive table can be successfully dropped.</p>