Application Integration Using Queues and Messages
AWS Architecture BlogApplication Integration Using Queues and Messagesby Mithun Mallick | on24 MAR 2020| inAdvanced (300),Amazon Simple Queue Service (SQS),Intermediate (200),Messaging,Serverless|Permalink|ShareIn previous blog posts in this messaging series, we provided anoverview of messagingand we also explained the common characteristics to consider whenevaluating messaging channel technologies. In this post, we will explain some of the semantics of queue-based processing, its use in designing flexible systems, and how to apply it to your use cases. AWS offers two queue-based services:Amazon Simple Queue Service(SQS) andAmazon MQ. We will focus on SQS in this blog.Building BlocksIn the digital world, even the most basic design of web based systems requires the use of queues to integrate applications. SQS is a secure, serverless, durable, and highly available message queue service. It provides a simple REST API to create a queue as well as send, receive, and delete messages.Producing MessagesMessage producers are processes that call theSendMessageAPIs of SQS. SQS supports two types of queues: standard and first-in-first-out (FIFO). Standard queues provide best effort ordering while FIFO provides first-in-first-out ordering. Messages can be sent either as single messages or in batches. Standard queues can support unlimited throughput by adding as many concurrent producers as needed, whereas FIFO queues can support up to 300 TPS without batching and 3000 TPS with batching.In terms of message delivery, SQS standard supports “at-least-once” delivery, meaning that messages will be delivered at least once but occasionally, more than one copy of the message will be delivered. SQS FIFO provides “exactly once” delivery, meaning it can detect duplicate messages.Consuming MessagesMessage consumers are processes that make theReceiveMessageAPI call on SQS. Messages from queues can be processed either in batch or as a single message at a time. Each approach has its advantages and disadvantages.Batch processing: This is where each message can be processed independently, and an error on a single message is not required to disrupt the entire batch. Batch processing provides the most throughput for processing, and also provides the most optimizations for resources involved in reading messages.Single message processing: Single message processing is commonly used in scenarios where each message may trigger multiple processes within the consumer. In case of errors, the retry is confined to the single message.To maintain the order of processing, FIFO queues are typically consumed by a single process. Messages across message groups can still be processed in parallel. However, the overall throughput limit will still apply for FIFO queues.SQS supports long and short polling on queues. Short polling will sample a subset of servers and will return the messages or provide an empty response if there are no messages in those servers. Long polling will query all servers and return immediately if there are messages. Long polling can reduce cost by reducing the number of calls in cases of empty responses. Get more details and sample code forsending and receiving messages.It’s simple to poll for messages but it does have an overhead of running a process continuously. In some situations it may be hard to monitor such processes and troubleshoot in case of errors. SQS integration withAWS Lambdatakes the undifferentiated heavy lifting of polling logic into a Lambda agent.Get more details for SQS as an event source for Lambda in this blog post:AWS Lambda Adds Amazon Simple Queue Service to Supported Event Sources. This is supported for both standard as well as FIFO queues.BenefitsQueue-based processing can protect backend systems like relational databases from unexpected surge in front-end traffic. You can decouple the processing logic by using an intermediate queue.Consider a scenario that involves a web application to order products related to a TV show. It’s possible that the ordering and payment systems rely on a traditional relational database. During peak show seasons, queues can be used to control the traffic to the ordering and payment system. In the following diagram we show how the web application requests are staged in the queue, while the consumption of the users on the backend is based on the capacity of the database.Error HandlingAsynchronous message processing presents unique challenges with error handling since they just manifest as log entries on the producer or consumer and create backlogs in processing. In order to handle errors elegantly, the first requirement is to address the nature of the error. In case the error is transient, it may help to retry after a brief delay and eventually move to a dead letter queue if repeated attempts fail. The other error scenario can be cases where the data itself is bad and the error won’t fix, even after repeated attempts. In such cases, consumer process needs to make this determination and needs to move the message to an error queue.In some cases, due to its highly distributed architecture, Standard SQS can deliver a duplicate message, which it can result in duplicate key errors for the consumer. The best way to mitigate it is to make the consumersidempotentso that processing the same message multiple times produces the same result.ConclusionIn this blog, we explained the importance of messaging in building distributed applications, various aspects of queue based processing using SQS like sending and receiving messages, FIFO versus Standard SQS, and common error handling scenarios. We also covered using SQS as an event source for AWS Lambda. Please refer to following blog posts for using messaging in different integration patterns:Implementing enterprise integration patterns with AWS messaging services: point-to-point channelsImplementing enterprise integration patterns with AWS messaging services: publish-subscribe channelsUnderstanding asynchronous messaging for microservicesApplication integration patterns for microservices: Fan-out strategiesIn this messaging seriesNow that you’re understanding application integration using queues and messages, be sure to read the first two posts in this series:Event-Based Processing for Asynchronous CommunicationIntroduction to Messaging for Modern Cloud ArchitectureNext up:Serverless Stream-based Processing for Real-time InsightsTAGS:Amazon SQS,application integration,messaging,serverlessMithun MallickMithun is a Senior Solutions Architect at AWS, focused on messaging services. Mithun is responsible for helping customers design and operate integration solutions using Amazon SQS, Amazon SNS, and Amazon MQ. He also has deep experience in the healthcare tech industry and works closely with it to build interoperability solutions. Mithun holds an MBA from Colorado State University in Fort Collins, Colorado.
In previous blog posts in this messaging series, we provided an overview of messaging and we also explained the common characteristics to consider when  evaluating messaging channel technologies. In this post, we will explain some of the semantics of queue-based processing, its use in designing flexible systems, and how to apply it to your use […]
Mithun Mallick
Advanced (300) Amazon Simple Queue Service (SQS) Intermediate (200) Messaging Serverless