title: AWS Glue Crawler creates each tables for each parquet data
body: <p>In the glue job, I created dynamic_frame from AWS RDS PSQL using glue connection and write the data into S3 after doing some ETL jobs. After that, I created and run Glue Crawler for the parquet file wrote in S3. Here is the sample code.</p>
  
  <pre><code>     dynamic_frame = glueContext.create_dynamic_frame_from_options('postgresql',
                                                                           connection_options)
  
  
      glueContext.write_dynamic_frame.from_options(
           frame=cleaned_dynamic_frame,
           connection_type="s3",
           connection_options={"path": export_path},
           format="parquet")
  
  
  
     response = glue.create_crawler(
          Name=crawler_name,
          Role=role,
          DatabaseName=db_name,
          Targets={
              'S3Targets': [
                  {
                      'Path': path
                  }
              ]
          },
          TablePrefix='{}_'.format(table_name)
      )
  
  
  ### S3 Objects list in the path (20 parquet files)
  &lt;bucket&gt;/&lt;path&gt;/part-0000-a.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0001-b.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0002-c.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0003-d.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0004-e.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0005-f.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0006-g.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0007-h.snappy.parquet
  &lt;bucket&gt;/&lt;path&gt;/part-0008-i.snappy.parquet
  </code></pre>
  
  <p>Problems: 
  The glue crawler supposed to create one Table with the same schema for those parquet data. However, somehow glue crawler created multiple tables for each parquet data and each table's location is for each parquet data. Not Combined. This happens sometime.. some times work correctly...</p>
  
  <p>My guess is that spark creates _temporary file to s3 before it finish glue job... and at the moment where the temp file exists, glue crawler is running and it creates each table for each file.</p>
  
  <p>Please comment below if you have any sense what is going on.
  Thank you for your great help.</p>