title: Hive doesn't read partitioned parquet files generated by Spark
body: <p>I'm having a problem to read partitioned parquet files generated by Spark in Hive. I'm able to create the external table in hive but when I try to select a few lines, hive returns only an "OK" message with no rows.</p>
  
  <p>I'm able to read the partitioned parquet files correctly in Spark, so I'm assuming that they were generated correctly.
  I'm also able to read these files when I create an external table in hive without partitioning.</p>
  
  <p>Does anyone have a suggestion?</p>
  
  <p><strong>My Environment is:</strong></p>
  
  <ul>
  <li>Cluster EMR 4.1.0</li>
  <li>Hive 1.0.0 </li>
  <li>Spark 1.5.0 </li>
  <li>Hue 3.7.1 </li>
  <li>Parquet files are stored in a S3 bucket (s3://staging-dev/test/ttfourfieldspart2/year=2013/month=11)</li>
  </ul>
  
  <p><strong>My Spark config file has the following parameters(/etc/spark/conf.dist/spark-defaults.conf):</strong></p>
  
  <pre><code>spark.master yarn
  spark.driver.extraClassPath /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
  spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
  spark.executor.extraClassPath /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/*:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
  spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
  spark.eventLog.enabled true
  spark.eventLog.dir hdfs:///var/log/spark/apps
  spark.history.fs.logDirectory hdfs:///var/log/spark/apps
  spark.yarn.historyServer.address ip-10-37-161-246.ec2.internal:18080
  spark.history.ui.port 18080
  spark.shuffle.service.enabled true
  spark.driver.extraJavaOptions    -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M -XX:OnOutOfMemoryError='kill -9 %p'
  spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
  spark.executor.memory 4G
  spark.driver.memory 4G
  spark.dynamicAllocation.enabled true
  spark.dynamicAllocation.maxExecutors 100
  spark.dynamicAllocation.minExecutors 1
  </code></pre>
  
  <p><strong>Hive config file has the following parameters(/etc/hive/conf/hive-site.xml):</strong></p>
  
  <pre><code>&lt;configuration&gt;
  
  &lt;!-- Hive Configuration can either be stored in this file or in the hadoop configuration files  --&gt;
  &lt;!-- that are implied by Hadoop setup variables.                                                --&gt;
  &lt;!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive    --&gt;
  &lt;!-- users do not have to edit hadoop configuration files (that may be managed as a centralized --&gt;
  &lt;!-- resource).                                                                                 --&gt;
  
  &lt;!-- Hive Execution Parameters --&gt;
  
  
  &lt;property&gt;
    &lt;name&gt;hbase.zookeeper.quorum&lt;/name&gt;
    &lt;value&gt;ip-10-xx-xxx-xxx.ec2.internal&lt;/value&gt;
    &lt;description&gt;http://wiki.apache.org/hadoop/Hive/HBaseIntegration&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;hive.execution.engine&lt;/name&gt;
    &lt;value&gt;mr&lt;/value&gt;
  &lt;/property&gt;
  
    &lt;property&gt;
      &lt;name&gt;fs.defaultFS&lt;/name&gt;
      &lt;value&gt;hdfs://ip-10-xx-xxx-xxx.ec2.internal:8020&lt;/value&gt;
    &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;hive.metastore.uris&lt;/name&gt;
    &lt;value&gt;thrift://ip-10-xx-xxx-xxx.ec2.internal:9083&lt;/value&gt;
    &lt;description&gt;JDBC connect string for a JDBC metastore&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;javax.jdo.option.ConnectionURL&lt;/name&gt;
      &lt;value&gt;jdbc:mysql://ip-10-xx-xxx-xxx.ec2.internal:3306/hive?createDatabaseIfNotExist=true&lt;/value&gt;
      &lt;description&gt;username to use against metastore database&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;javax.jdo.option.ConnectionDriverName&lt;/name&gt;
      &lt;value&gt;org.mariadb.jdbc.Driver&lt;/value&gt;
      &lt;description&gt;username to use against metastore database&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;javax.jdo.option.ConnectionUserName&lt;/name&gt;
    &lt;value&gt;hive&lt;/value&gt;
    &lt;description&gt;username to use against metastore database&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
    &lt;name&gt;javax.jdo.option.ConnectionPassword&lt;/name&gt;
    &lt;value&gt;1R72JFCDG5XaaDTB&lt;/value&gt;
    &lt;description&gt;password to use against metastore database&lt;/description&gt;
  &lt;/property&gt;
  
    &lt;property&gt;
      &lt;name&gt;datanucleus.fixedDatastore&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
    &lt;/property&gt;
  
    &lt;property&gt;
      &lt;name&gt;mapred.reduce.tasks&lt;/name&gt;
      &lt;value&gt;-1&lt;/value&gt;
    &lt;/property&gt;
  
    &lt;property&gt;
      &lt;name&gt;mapred.max.split.size&lt;/name&gt;
      &lt;value&gt;256000000&lt;/value&gt;
    &lt;/property&gt;
  
    &lt;property&gt;
      &lt;name&gt;hive.metastore.connect.retries&lt;/name&gt;
      &lt;value&gt;5&lt;/value&gt;
    &lt;/property&gt;
  
    &lt;property&gt;
      &lt;name&gt;hive.optimize.sort.dynamic.partition&lt;/name&gt;
      &lt;value&gt;true&lt;/value&gt;
    &lt;/property&gt;
  
    &lt;property&gt;&lt;name&gt;hive.exec.dynamic.partition&lt;/name&gt;&lt;value&gt;true&lt;/value&gt;&lt;/property&gt;
    &lt;property&gt;&lt;name&gt;hive.exec.dynamic.partition.mode&lt;/name&gt;&lt;value&gt;nonstrict&lt;/value&gt;&lt;/property&gt;
    &lt;property&gt;&lt;name&gt;hive.exec.max.dynamic.partitions&lt;/name&gt;&lt;value&gt;10000&lt;/value&gt;&lt;/property&gt;
    &lt;property&gt;&lt;name&gt;hive.exec.max.dynamic.partitions.pernode&lt;/name&gt;&lt;value&gt;500&lt;/value&gt;&lt;/property&gt;
  
  &lt;/configuration&gt;
  </code></pre>
  
  <p><strong>My python code that reads the partitioned parquet file:</strong></p>
  
  <pre><code>from pyspark import *
  from pyspark.sql import *
  from pyspark.sql.types import *
  from pyspark.sql.functions import *
  
  df7 = sqlContext.read.parquet('s3://staging-dev/test/ttfourfieldspart2/')
  </code></pre>
  
  <p><strong>The parquet file schema printed by Spark:</strong></p>
  
  <pre><code>&gt;&gt;&gt; df7.schema
  StructType(List(StructField(transactionid,StringType,true),StructField(eventts,TimestampType,true),StructField(year,IntegerType,true),StructField(month,IntegerType,true)))
  
  &gt;&gt;&gt; df7.printSchema()
  root
   |-- transactionid: string (nullable = true)
   |-- eventts: timestamp (nullable = true)
   |-- year: integer (nullable = true)
   |-- month: integer (nullable = true)
  
  &gt;&gt;&gt; df7.show(10)
  +--------------------+--------------------+----+-----+
  |       transactionid|             eventts|year|month|
  +--------------------+--------------------+----+-----+
  |f7018907-ed3d-49b...|2013-11-21 18:41:...|2013|   11|
  |f6d95a5f-d4ba-489...|2013-11-21 18:41:...|2013|   11|
  |02b2a715-6e15-4bb...|2013-11-21 18:41:...|2013|   11|
  |0e908c0f-7d63-48c...|2013-11-21 18:41:...|2013|   11|
  |f83e30f9-950a-4b9...|2013-11-21 18:41:...|2013|   11|
  |3425e4ea-b715-476...|2013-11-21 18:41:...|2013|   11|
  |a20a6aeb-da4f-4fd...|2013-11-21 18:41:...|2013|   11|
  |d2f57e6f-889b-49b...|2013-11-21 18:41:...|2013|   11|
  |46f2eda5-408e-44e...|2013-11-21 18:41:...|2013|   11|
  |36fb8b79-b2b5-493...|2013-11-21 18:41:...|2013|   11|
  +--------------------+--------------------+----+-----+
  only showing top 10 rows
  </code></pre>
  
  <p><strong>The create table in Hive:</strong></p>
  
  <pre><code>create external table if not exists t3(
    transactionid string,
    eventts timestamp)
  partitioned by (year int, month int)
  stored as parquet
  location 's3://staging-dev/test/ttfourfieldspart2/';
  </code></pre>
  
  <p><strong>When I try to select some rows in Hive, it doesn't return any rows:</strong></p>
  
  <pre><code>hive&gt; select * from t3 limit 10;
  OK
  Time taken: 0.027 seconds
  hive&gt; 
  </code></pre>