title: Best Method of copying partition in HDFS
body: <p>So, I have a use case where I want to copy the last partition of the table and insert it in the same table with new partition column value.</p>
  
  <p>Its important to note that last partition value is always known.</p>
  
  <p>I can think of two ways of doing this</p>
  
  <ol>
  <li>Get <code>where partitionColumn = partitionValue</code> in a <code>DataFrame</code> and change the partition value using <code>.withColumn</code> and then <code>saveAsTable</code> and then do Invalidate/Refresh</li>
  <li>OR,
  
  <ol>
  <li>Copy the <code>parquet</code> files from the last partition directory</li>
  <li>Create new partition, paste it in new directory with <code>/newPatritionValue/</code> at the end</li>
  <li>Paste the parquet files</li>
  <li>Run <code>ALTER TABLE</code></li>
  </ol></li>
  </ol>
  
  <p>Which would run faster?</p>