title: How to save a partitioned parquet file in Spark 2.1?
body: <p>I am trying to test how to write data in HDFS 2.7 using Spark 2.1. My data is a simple sequence of dummy values and the output should be partitioned by the attributes: <em>id</em> and <em>key</em>.</p>
  
  <pre><code> // Simple case class to cast the data
   case class SimpleTest(id:String, value1:Int, value2:Float, key:Int)
  
   // Actual data to be stored
   val testData = Seq(
      SimpleTest("test", 12, 13.5.toFloat, 1),
      SimpleTest("test", 12, 13.5.toFloat, 2),
      SimpleTest("test", 12, 13.5.toFloat, 3),
      SimpleTest("simple", 12, 13.5.toFloat, 1),
      SimpleTest("simple", 12, 13.5.toFloat, 2),
      SimpleTest("simple", 12, 13.5.toFloat, 3)
   )
  
   // Spark's workflow to distribute, partition and store
   // sc and sql are the SparkContext and SparkSession, respectively
   val testDataP = sc.parallelize(testData, 6)
   val testDf = sql.createDataFrame(testDataP).toDF("id", "value1", "value2", "key")
   testDf.write.partitionBy("id", "key").parquet("/path/to/file")
  </code></pre>
  
  <p>I am expecting to get the following tree structure in HDFS:</p>
  
  <pre><code>- /path/to/file
     |- /id=test/key=1/part-01.parquet
     |- /id=test/key=2/part-02.parquet
     |- /id=test/key=3/part-03.parquet
     |- /id=simple/key=1/part-04.parquet
     |- /id=simple/key=2/part-05.parquet
     |- /id=simple/key=3/part-06.parquet
  </code></pre>
  
  <p>But when I run the previous code I get the following output:</p>
  
  <pre><code>/path/to/file/id=/key=24/
   |-/part-01.parquet
   |-/part-02.parquet
   |-/part-03.parquet
   |-/part-04.parquet
   |-/part-05.parquet
   |-/part-06.parquet
  </code></pre>
  
  <p>I do not know if there is something wrong in the code, or is there something else that Spark is doing.</p>
  
  <p>I'm executing <code>spark-submit</code> as follows:</p>
  
  <blockquote>
    <p>spark-submit --name APP --master local --driver-memory 30G --executor-memory 30G --executor-cores 8 --num-executors 8 --conf spark.io.compression.codec=lzf --conf spark.akka.frameSize=1024 --conf spark.driver.maxResultSize=1g --conf spark.sql.orc.compression.codec=uncompressed --conf spark.sql.parquet.filterPushdown=true --class myClass myFatJar.jar</p>
  </blockquote>