Expiring Amazon S3 Objects Based on Last Accessed Date to Decrease Costs
AWS Architecture BlogExpiring Amazon S3 Objects Based on Last Accessed Date to Decrease Costsby Hareesh Singireddy, Shivam Patel, and Uday Narayanan | on03 AUG 2021| inAmazon Simple Storage Service (S3),Architecture,AWS Lambda|Permalink|ShareOrganizations are usingAmazon Simple Storage Service (S3)for building their data lakes, websites, mobile applications, and enterprise applications. As the number of objects within your S3 bucket increases, you may want to move older objects into lower-cost tiers of Amazon S3. In some cases you may want to delete the objects altogether to further reduce S3 storage costs. A common practice is to useS3 Lifecycle rulesto achieve this. These rules can be applied to objects based on their creation date. In certain situations, you may want to keep objects available that are still being accessed, but transition or delete objects that are no longer in use.In this post, we will demonstrate how you can create custom object expiry rules for Amazon S3 based on the last accessed date of the object. We will first walk through the various features used within the workflow, followed by an architecture diagram outlining the process flow.Amazon S3 server access loggingS3 Server access loggingprovides detailed records of the requests that are made to objects in Amazon S3 buckets. Amazon S3 periodically collects access log records, consolidates the records in log files, and then uploads log files to your target bucket as log objects. Each log record consists of information such as bucket name, the operation in the request, and the time at which the request was received.S3 Server Access Log Formatprovides more details about the format of the log file.Amazon S3 inventoryAmazon S3 inventoryprovides a list of your objects and the corresponding metadata on a daily or weekly basis, for an S3 bucket or a shared prefix. The inventory lists are stored as a comma-separated value (CSV) file compressed with GZIP, as anApache optimized row columnar (ORC)file compressed with ZLIB, or as anApache Parquetfile compressed with Snappy.Amazon S3 LifecycleAmazon S3 Lifecycle policies help you manage your objects through two types of actions, Transition and Expiration. In the architecture shown following in Figure 1, we create anS3 Lifecycle configurationrule that expires objects after ‘x’ days. It has a filter for an object tag of“delete=True”. You can configure the value of ‘x’ based on your requirements.If you are using an S3 bucket to store short lived objects with unknown access patterns, you might want to keep the objects that are still being accessed, but delete the rest. This will let you retain objects in your S3 bucket even after their expiry date as per the S3 lifecycle rules, while saving you costs by deleting objects that are not needed anymore. The following diagram shows an architecture that considers the last accessed date of the object before deleting S3 objects.Figure 1. Object expiry architecture flowThis architecture uses native S3 features mentioned earlier in combination with other AWS services to achieve the desired outcome.Here is the architecture flow:The S3 server access logs capture S3 object requests. These are generated and stored in the target S3 bucket.An S3 inventory report is generated for the source bucket daily. It is written to the S3 inventory target bucket.AnAmazon EventBridgerule is configured that will initiate anAWS Lambdafunction once a day, or as desired.The Lambda function initiates anS3 Batch Operationjob to tag objects in the source bucket. These must be expired using the following logic:Capture thenumber of days(x) configuration from the S3 Lifecycle configuration.Run anAmazon Athenaquery that will get the list of objects from the S3 inventory report and server access logs. Create a delta list with objects that were created earlier than ‘x’ days, but not accessed during that time.Write a manifest file with the list of these objects to an S3 bucket.Create anS3 Batch operation jobthat will tag all objects in the manifest file with a tag of“delete=True”.The Lifecycle rule on the source S3 bucket will expire all objects that were created prior to ‘x’ days. They will have the tag given via the S3 batch operation of“delete=True”.The preceding architecture is built for fault tolerance. If a particular run fails, all the objects that must be expired will be picked up during the next run. You can configureerror handling and automatic retriesin your Lambda function. AnAmazon Simple Notification Service (SNS)topic will send out a notification in the event of a failure.Cost considerationsS3 server access logs, S3 inventory lists, and manifest files can accumulate many objects over time. We recommend you configure an S3 Lifecycle policy on the target bucket to periodically delete older objects. Although following the guidelines in this post can decrease some of your costs, S3 requests, S3 inventory, S3 Object Tagging, and Lifecycle transitions also have costs associated with them. Additional details can be found on theS3 pricing page.Amazon Athena charges you based on the amount of data scanned by each query. But Amazon S3 inventory can also output files in Apache ORC or Apache Parquet format, which can reduce the amount data scanned by Athena. TheAthena pricing pagewould be helpful to review.AWS Lambda has a free usage tier of 1M free requests per month and 400,000 GB-seconds of compute time per month. However, you are charged based on the number of requests, the amount of memory allocated, and the runtime duration of the function. See more at theLambda pricing page.ConclusionIn this blog post, we showed how you can create a custom process to delete objects from your S3 bucket based on the last time the object was accessed. You can use this architecture to customize your object transitions, clean up your S3 buckets for any unnecessary objects, and keep your S3 buckets cost-effective. This architecture can also be used on versioned S3 buckets with some minor modifications.We hope you found this blog post useful and welcome your feedback!Read more about queries, rules, and tags:Querying Amazon S3 inventory with Amazon AthenaQuerying access logs for requests using Amazon AthenaCreate a rule in Amazon EventBridgeAdding and removing object tags with Amazon S3 Batch OperationsHareesh SingireddyHareesh Singireddy is a Solutions Architect at Amazon Web Services. He helps large enterprises design and build Well-Architected solutions on AWS Cloud. Hareesh holds 8 AWS certifications including AWS Certified Solutions Architect Professional.Shivam PatelShivam Patel is a Solutions Architect at AWS. He helps customers design, deploy, and scale solutions to achieve business outcomes. His core areas of focus are End-User Computing, Media & Entertainment, and VMWare Cloud on AWS. Outside of work, Shivam is an avid food connoisseur, New York Yankees fanatic, and globetrotter (conditions permitting).Uday NarayananUday Narayanan is a Sr. Solutions Architect at AWS. He enjoys helping customers find innovative solutions to complex business challenges. His core areas of focus are Data Analytics, Big Data Systems and Machine Learning. In his spare time he enjoys playing sports, binge-watching TV shows, and traveling.
Organizations are using Amazon Simple Storage Service (S3) for building their data lakes, websites, mobile applications, and enterprise applications. As the number of objects within your S3 bucket increases, you may want to move older objects into lower-cost tiers of Amazon S3. In some cases you may want to delete the objects altogether to further […]
Hareesh Singireddy
Amazon Simple Storage Service (S3) Architecture AWS Lambda