title: AWS Glue Crawler updating existing catalog tables is (painfully) slow
body: <p>I am continuously receiving and storing multiple feeds of <strong>uncompressed</strong> JSON objects, <strong>partitioned</strong> to the day, to different locations of an Amazon S3 bucket (hive-style: <code>s3://bucket/object=&lt;object&gt;/year=&lt;year&gt;/month=&lt;month&gt;/day=&lt;day&gt;/object_001.json</code>), and was planning to incrementally batch and load this data to a Parquet data lake using AWS Glue:</p>
  
  <ul>
  <li>Crawlers would update manually created Glue tables, one per object feed, for schema and partition (new files) updates;</li>
  <li>Glue ETL Jobs + Job bookmarking would then batch and map all new partitions per object feed to a Parquet location now and then.</li>
  </ul>
  
  <p>This design pattern &amp; architecture seemed to be quite a safe approach as it was backed up by many AWS blogs, <a href="https://aws.amazon.com/blogs/big-data/build-and-automate-a-serverless-data-lake-using-an-aws-glue-trigger-for-the-data-catalog-and-etl-jobs/" rel="noreferrer">here</a> and <a href="https://aws.amazon.com/blogs/big-data/load-data-incrementally-and-optimized-parquet-writer-with-aws-glue/" rel="noreferrer">there</a>.</p>
  
  <p>I have a crawler configured as so: </p>
  
  <pre><code>{
    "Name": "my-json-crawler",
    "Targets": {
      "CatalogTargets": [
        {
          "DatabaseName": "my-json-db",
          "Tables": [
            "some-partitionned-json-in-s3-1",
            "some-partitionned-json-in-s3-2",
            ...
          ]
        }
      ]
    },
    "SchemaChangePolicy": {
      "UpdateBehavior": "UPDATE_IN_DATABASE",
      "DeleteBehavior": "LOG"
    },
    "Configuration": "{\"Version\":1.0,\"Grouping\":{\"TableGroupingPolicy\":\"CombineCompatibleSchemas\"}}"
  }
  </code></pre>
  
  <p>And each table was "manually" initialized as so:</p>
  
  <pre><code>{
    "Name": "some-partitionned-json-in-s3-1",
    "DatabaseName": "my-json-db",
    "StorageDescriptor": {
      "Columns": [] # i'd like the crawler to figure this out on his first crawl,
      "Location": "s3://bucket/object=some-partitionned-json-in-s3-1/",
      "PartitionKeys": [
        {
          "Name": "year",
          "Type": "string"
        },
        {
          "Name": "month",
          "Type": "string"
        },
        {
          "Name": "day",
          "Type": "string"
        }
      ],
      "TableType": "EXTERNAL_TABLE"
    }
  }
  </code></pre>
  
  <p>First run of the crawler is, as expected, an hour-ish long, but it successfully figures out the table schema and existing partitions. Yet from that point onward, re-running the crawler takes the exact same amount of time as the first crawl, if not longer; which lead me to believe that the crawler is not only crawling for new files / partitions, but recrawling all the entire S3 locations each time.</p>
  
  <p>Note that the delta of new files between two crawls is very small (few new files are to be expected each time).</p>
  
  <p><a href="https://aws.amazon.com/premiumsupport/knowledge-center/long-running-glue-crawler/" rel="noreferrer">AWS Documentation</a> suggests running multiple crawlers, but I am not convinced that this would solve my problem on the long run. I also considered updating the crawler exclude pattern after each run, but then I would see too few advantages using Crawlers over manually updating Tables partitions through some Lambda boto3 magic.</p>
  
  <p>Am I missing something there ? Maybe an option I would have misunderstood regarding crawlers updating existing data catalogs rather than crawling data stores directly ?</p>
  
  <p>Any suggestions to improve my data cataloging ? Given that indexing these JSON files in Glue tables is only necessary to me as I want my Glue Job to use bookmarking.</p>
  
  <p>Thanks ! </p>