title: Copy parquet files then query them with Athena
body: <p>I use aws-data-wrangler (<a href="https://github.com/awslabs/aws-data-wrangler" rel="nofollow noreferrer">https://github.com/awslabs/aws-data-wrangler</a>) to process pandas dataframes. Once they are processed, I export them to parquet files with:</p>
  
  <pre><code>wr.pandas.to_parquet(
      dataframe=my_dataframe,
      description=DESCRIPTION,
      columns_comments=COLUMN_COMMENTS,
      parameters=DATASET_TAGS,
      database=my_database,
      table=f"{table}_{latest_refresh_date}",
      path=f"s3://{bucket_out}/{sub_path}/{latest_refresh_date}/",
      procs_cpu_bound=1,
      partition_cols=["date"],
      mode="overwrite_partitions",
      preserve_index=False,
  )
  </code></pre>
  
  <p>By doing it this way, it also creates a Glue table. The parquet files happily live in a S3 bucket, and I can query the data with Athena using the name of the Glue table, like this:</p>
  
  <pre><code>select * from {table}_{latest_refresh_date}
  </code></pre>
  
  <p>Now let's say that I get new data. The new data should be stored in a new S3 path: <code>s3://{bucket_out}/{sub_path}/{other_refresh_date}/</code> for example. I process the new data as before, but I don't want to re-process the old data. So I copy the parquet files from the old path to the new one, with a simple S3 copy. The old data isn't handled with <code>wr.pandas.to_parquet</code>.</p>
  
  <p>Now when I want to query the data living in <code>s3://{bucket_out}/{sub_path}/{other_refresh_date}/</code>, I can only access the new data.</p>
  
  <pre><code>select * from {table}_{other_refresh_date}
  </code></pre>
  
  <p>It seems I can only query the data that was added to the Glue table. I naively thought that Athena would query a S3 path but apparently it's more complicated than that.</p>
  
  <p>Could you explain to me why this is happening, and maybe suggest a work around? Do I need to somehow register the old files to the new Athena table?</p>