title: running interactive sql queries over millions of parquet files
body: <p>I have millions of streaming parquet files being written . I want to support running ad hoc interactive queries for debugging and analytics purpose ( added bonus if i can run streaming queries for some real time monitoring of key metrics as well).
  What is a scalable solution for supporting this.
  The two ways I have observed is running spark sql interactively over millions of parquet files (not too familiar with spark ecosystem but does this mean running a spark job for every sql user submits or do i need to run some streaming job and submit queries somehow) and second being using a presto sql engine on top of parquet (not exactly sure how presto ingests new incoming parquet files). 
  Any recommendations or pros and cons of either approach . Any better solutions considering i have > ~10Tb data produced every day .</p>