title: What does MSCK REPAIR TABLE do behind the scenes and why it's so slow?
body: <p>I know that <code>MSCK REPAIR TABLE</code> updates the metastore with the current partitions of an external table.</p>
  
  <p>To do that, you only need to do <code>ls</code> on the root folder of the table (given the table is partitioned by only one column), and get all its partitions, clearly a &lt; 1s operation. </p>
  
  <p>But in practice, the operation can take a <strong>very long</strong> time to execute (or even <a href="https://docs.aws.amazon.com/athena/latest/ug/msck-repair-table.html" rel="noreferrer">timeout if ran on AWS Athena</a>).</p>
  
  <p>So my question is, what does <code>MSCK REPAIR TABLE</code> actually do behind the scenes and why?  </p>
  
  <p>How does MSCK REPAIR TABLE find the partitions?</p>
  
  <hr>
  
  <p>Additional data in case it's relevant:</p>
  
  <p><sub>Our data is all on S3, it's both slow when running on EMR (Hive) or Athena (Presto), there are ~450 partitions in the table, every partition has on avg 90 files, overall 3 Gigabytes for a partition, files are in Apache parquet format</sub></p>