Get Started with Amazon S3 Event Driven Design Patterns
AWS Architecture BlogGet Started with Amazon S3 Event Driven Design Patternsby Micah Walter, Aditya Pendyala, and Divyesh Sah | on27 SEP 2021| inAmazon Rekognition,Amazon Simple Notification Service (SNS),Amazon Simple Queue Service (SQS),Architecture,AWS Lambda|Permalink|ShareEvent driven programs use events to initiate succeeding steps in a process. For example, the completion of an upload job may then initiate an image processing job. This allows developers to create complex architectures by using the principle ofdecoupling. Decoupling is preferable for many workflows, as it allows each component to perform its tasks independently, which improves efficiency. Examples are ecommerce order processing, image processing, and other long running batch jobs.Amazon Simple Storage Service (S3)is an object-based storage solution fromAmazon Web Services (AWS)that allows you to store and retrieve any amount of data, at any scale.Amazon S3 Event Notificationsprovides users a mechanism for initiating events when certain actions take place inside an S3 bucket.In this blog post, we will illustrate how you can use Amazon S3 Event Notifications in combination with a powerful suite of Amazon messaging services. This will allow you to implement an event driven architecture for a variety of common use cases.Setting up Amazon S3 Event NotificationsWe first must understand the types of events that can be initiated with Amazon S3 Event Notifications. Events can be initiated by uploading, modifying, deleting an object, or other actions. When an event is initiated, a payload is created containing the event metadata. This includes information about the object that initiated the event itself.To enable notifications, you must first add a notification configuration that identifies the events you want Amazon S3 to publish. Specify the destinations where you want Amazon S3 to send the notifications. This configuration is stored in thenotificationsubresource, which you can find under thePropertiestab within your S3 bucket, see Figure 1.Figure 1. Properties tab showing S3 Event Notifications subresourceAn event notification can be initiated anytime an object is uploaded, modified, or deleted, depending on your configuration details. You can create multiple notification configurations for different scenarios, shown in Figure 2. For example, one configuration can handle new or modified objects, and another configuration can handle deletions. You can specify that events will only be initiated when objects contain a specific prefix, or following the restoration of an object. For a complete listing of all the configuration options and event types, read documentation onsupported event types.Figure 2. S3 Event Notifications subresource details and optionsWhen all of the conditions in your configuration have been met, a new event will be initiated and sent to the destination you specify. An S3 event destination can be anAWS Lambdafunction, anAmazon Simple Queue Service (SQS)queue, or anAmazon Simple Notification Service (SNS)topic, see Figure 3.Figure 3. S3 Event Notifications subresource destination settingsEvent driven design patternsThere are many common design patterns for building event driven programs with Amazon S3 Event Notifications. Once you have set up your notification configuration, the next step is to consume the event. The following describes a few typical architectures you might consider, depending on the needs of your application.Synchronous and reliable point-to-point processingFigure 4. Point-to-point processing with S3 and Lambda as a destinationOne common use case for event driven processing, is when synchronous and reliable information is required. For example, a mobile application processes images uploaded by users and automatically tags the images with the detected objects using Artificial Intelligence/Machine Learning (AI/ML). From an architectural perspective (Figure 4), an image is uploaded to an S3 bucket, which generates an event notification. This initiates a Lambda function that sends the details of the uploaded image toAmazon Rekognitionfor tagging. Results from Amazon Rekognition could be further processed by the Lambda function and stored in a database likeAmazon DynamoDB.With this type of architecture, there is no contingency for dealing with multiple images arriving simultaneously in the S3 bucket. If this application sends too many requests to Lambda, events can start to pile up. This can cause a failure to process some of the images. To make our program more fault tolerant, adding an Amazon SQS queue would help, as shown in Figure 5.Asynchronous and queued point-to-point processingFigure 5. Queued point-to-point processing with S3, SQS, and LambdaArchitectures that require the processing of information in an asynchronous fashion can use this pattern. Building off the first example, a mobile application might provide a solution to allow end users to bulk upload thousands of images simultaneously. It can then use AWS Lambda to send the images to Amazon Rekognition for tagging.By providing a queue-based asynchronous solution, the Lambda function can retrieve work from the SQS queue at its own pace. This allows it to control the processing flow by processing files sequentially without risk of being overloaded. This is especially useful if the application must handle incomplete or partial uploads when a connection is temporarily lost.Currently, Amazon S3 Event Notifications only work with standard SQS queues, and first-in-first-out (FIFO) SQS queues are not supported. Read more abouthow to configure S3 event notificationwith an SQS queue as a destination. Your Lambda function in this architecture must be adjusted to handle the message payload arriving from SQS. This is because it will have a slightly different form than the original event notification body generated from S3.Parallel processing with “Fan Out” architectureFigure 6. Fan out design pattern with S3, SNS, and SQS before sending to a Lambda functionTo create a “fan out” style architecture where a single event is propagated to many destinations in parallel, SNS is combined with SQS. Configure your S3 event notification to use an SNS topic as its destination, as shown in Figure 6. You can then direct multiple subsequent processes to act on the same event. This is especially useful if you aim to do parallel processing on the same object in S3.For example, if you wanted to process a source image into multiple target resolutions, you could create a Lambda function. The function will use the “fan-out” pattern to process all images at the same time, at each resolution. You could then subscribe an SQS queue to your SNS topics. This ensures that Event Notifications sent to SNS are verified as complete by SQS, once they’ve been processed by your Lambda function.Figure 7. Fan out design pattern including secondary pipeline for deleting imagesTo extend the use case of image processing even further, you could create multiple SNS topics to handle different types of events from the same S3 bucket. As depicted in Figure 7, this architecture would allow your program to handle creations and updates differently than deletions. You could also process images differently based on their S3 prefix.Adjust your Lambda code to handle messages making their way through SNS and SQS. Their payloads will be slightly different than the original S3 Event Notification payload.Real-time notificationsFigure 8. Event driven design pattern for real-time notificationsIn addition toapplication-to-applicationmessaging, Amazon SNS providesapplication-to-person(A2P) communication (see Figure 8). Amazon SNS can send SMS text messages to mobile subscribers in over 100 countries. It can also send push notifications to Android and Apple devices and emails over SMTP. Using A2P, uploading an image to an Amazon S3 bucket can generate a notification to a group of users via their choice of Amazon SNS A2P platform.ConclusionIn this blog post, we’ve shown you the basic design patterns for developing an event driven architecture using Amazon S3 Event Notifications. You can create many more complicated architecture patterns to suit your needs. By using Amazon SQS, Amazon SNS, and AWS Lambda, you can design an event driven program that is fault tolerant, scalable, and smartly decoupled. But don’t stop there! Consider expanding your program further by utilizingAWS Lambda destinations. Or combine parallel image processing with highly scalable A2P notifications, which will alert your users when a task is complete.For further reading:Amazon S3 Event NotificationsAmazon SQSAmazon SNSTAGS:A2PMicah WalterMicah Walter is a Sr. Solutions Architect supporting enterprise customers in the New York City region and beyond. He advises executives, engineers, and architects at every step along their journey to the cloud, with a deep focus on sustainability and practical design. In his free time, Micah enjoys the outdoors, photography, and chasing his son around the house.Aditya PendyalaAditya is a Senior Solutions Architect at AWS based out of NYC. He has extensive experience in architecting cloud based applications. He is currently working with large enterprises to help them craft highly scalable, flexible, and resilient cloud architectures, and guides them on all things cloud. He has a Master of Science degree in Computer Science from Shippensburg University and believes in the quote “When you cease to learn, you cease to grow”.Divyesh SahDivyesh Sah is a Sr. Enterprise Solutions Architect in AWS focusing on financial services customers. He helps them with cloud transformation initiatives in the areas of migrations, application modernization, and cloud native solutions. He has over 18 years of technical experience specializing in AI/ML, databases, big data, containers, and BI and analytics. Prior to AWS, he has experience in areas of sales, program management, and professional services.
Event driven programs use events to initiate succeeding steps in a process. For example, the completion of an upload job may then initiate an image processing job. This allows developers to create complex architectures by using the principle of decoupling. Decoupling is preferable for many workflows, as it allows each component to perform its tasks […]
Micah Walter
Amazon Rekognition Amazon Simple Notification Service (SNS) Amazon Simple Queue Service (SQS) Architecture AWS Lambda