title: Spark Partitioning Hive Table
body: <p>I am trying to partition the hive table with distinct timestamps. I have a table with timestamps in it but when I execute the hive partition query, it says that it is not a valid partition column. Here's the table:</p>
  
  <pre><code>+---+-----------------------+
  |id |rc_timestamp           |
  +---+-----------------------+
  |1  |2017-06-12 17:18:39.824|
  |2  |2018-06-12 17:18:39.824|
  |3  |2019-06-12 17:18:39.824|
  +---+-----------------------+
  </code></pre>
  
  <pre><code>spark.sql("SET hive.exec.dynamic.partition.mode=nonrestrict")
  
  val tempTable  = spark.sql("SELECT * FROM partition_table")
  
  val df = tempTable.select("rc_timestamp")
  
  val a = x.toString().replaceAll("[\\[\\]]","")
  
  df.collect().foreach(a =&gt; {
    spark.sql(s"ALTER TABLE mydb.partition_table ADD IF NOT EXISTS PARTITION 
    (rc_timestamp = '$a')").show()
  )}
  </code></pre>
  
  <p>Here's the error which I'm getting:</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: rc_timestamp is not a valid partition column 
  in table mydb.partition_table.;
  </code></pre>