Rate Limiting Strategies for Serverless Applications
AWS Architecture BlogRate Limiting Strategies for Serverless Applicationsby Sharon Li, Akhil Aendapally, and Ashish Lagwankar | on09 JUN 2021| inAmazon API Gateway,Amazon CloudWatch,AWS Fargate,AWS Lambda,AWS WAF,AWS X-Ray,Kinesis Data Streams,Serverless|Permalink|ShareServerless technologies reduce the work needed to set up and maintain computing resources, provide built-in scalability, and optimize agility, performance, cost, and security. The pay-as-you-go model is particularly liberating for developers. You can fail fast, experiment more, and do it fairly cheaply. However, serverless brings its own challenges. In this blog, we’ll examine how to avoid unnecessary costs and help you limit the impact on your applications:Runaway costs.How do you keep track of and control costs in this development model? In a typical on-premises environment, the limits of your exposure are controlled by the hardware you provision. Serverless scales quickly, and a piece of poorly written code could rapidly scale up and consume resources, making it difficult to control runaway costs.Overwhelming downstream components.A typical application combines serverless and non-serverless technologies. The serverless piece of the architecture can scale rapidly to match demand. But downstream components with fixed capacities (like a relational database with a set capacity of connections) could be overwhelmed. Implementing a throttling mechanism can prevent events from propagating to multiple downstream components with unexpected side effects.Rate limiting on serverless applicationsIn this section, we show you two common strategies to use when rate limiting serverless applications.Strategy 1 – Synchronous invocationFor synchronous invocation, Figure 1 shows a classic serverless architecture.Figure 1. Synchronous invocationOn the compute layer,AWS Lambda, a serverless compute service, lets you run code without provisioning or managing servers. It automatically scales your application by running code in response to each event. However, the downstream systems that this Lambda function calls may not be scalable. To prevent the downstream systems from being overwhelmed by a large flood of events, you can put a rate limiting mechanism in front. This throttling mechanism can be implemented for different components as a cost saving solution, which will help make your system more robust.Note that there is a concurrency limit for Lambda functions per Region. This is especially true if you have many Lambda functions for different workloads deployed in one Region. You will get throttled if you exceed the concurrency limit. TheManaging AWS Lambda Function Concurrencyblog post further explains the Lambda concurrency limit. Generally speaking, you want to configure the reserved concurrency limit per Lambda function to control throughput based on different workloads. Then you will use the unreserved limit as a common pool for non-critical workloads.When you want to expose your Lambda function, you can create a web API with an HTTP endpoint for your Lambda function. You can configure rate limiting to your API throughAPI Gateway throttles. This prevents your API from being overwhelmed by too many requests.Amazon API Gatewayprovides two basic types of throttling-related settings:Server-side throttling limits are applied across all clients. These limit settings exist to prevent your API and your account from being overwhelmed by too many requests.Per-client throttling limits are applied to clients that use API keys associated with your usage policy as client identifier.Additionally, as shown in Figure 2, you may want to continuously enhance protection to your API by enablingAWS WAF. With AWS WAF, you can set up rate-based rules to specify the number of web requests that are allowed by each client IP in a trailing, continuously updated, 5-minute period.Figure 2. Enabling AWS WAF in front of APIStrategy 2 – Asynchronous invocationWhen you don’t need a response right away (like firing events), you can use anAmazon Simple Queue Service (Amazon SQS)queue as an event source. As shown in Figure 3, this prompts the Lambda function to process the messages in the queue.Figure 3. Asynchronous invocation with SQSWhen configuringAWS Lambda event source mapping, there is a configurable maximumbatch size, which is the number of messages that are delivered on each function invocation. The SQS queue message processing rate is determined by event source maximum batch size and Lambda concurrency limit. When an SQS trigger is initially enabled, Lambda begins with a maximum of five concurrent invocations. Lambda functions with an SQS queue trigger scale up to a maximum of 1,000 concurrent invocations, the account concurrency limit, or the reserved concurrency limit if you have it configured. When Lambda invokes the target function, the event can contain multiple items (up to a configurable maximum batch size). To control the message consumption rate and avoid throttling failure, follow thesebest practices.Figure 4 shows another approach: using anAmazon Kinesis Data Streamas an event source to prompt the Lambda function.Figure 4. Asynchronous invocation with Kinesis Data StreamTo control the processing rate of data from the stream, you can configure the following two parameters:Thenumber of shardsin the Kinesis Data StreamLambda event source batch sizeFigure 5 shows another commonly used serverless compute service:AWS Fargate. Lambda is not designed for functions that last for more than 15 minutes. If you foresee batch jobs taking several minutes or hours to complete, you can set up anAmazon Elastic Container Service (Amazon ECS)Fargate clusterto process messages from a queue.Figure 5. Serverless compute with FargateIn this architecture, the message processing rate is driven by the size of the cluster. We can also use anAmazon CloudWatchalarm that will automatically scale the size of the cluster based on the depth of the queue.When launching a Fargate task using the Amazon ECS run-task API, the call is throttled at 1 transaction per second by default with a burst rate of 10. This means that you can, at most, launch 10 tasks every 10 seconds. Because of this, we recommend that you use a backoff strategy (likeretriesandexponential backoffs) when launching tasks. Alternatively, you can use an Amazon ECS create-service call where Amazon ECS will ensure that all tasks are run in time while maintaining the throttle rate. Essentially, you could run 30 tasks concurrently. However, you couldn’t start all 30 tasks at the same time because the run-task API for Fargate will be throttled.Proactive monitoring to track scaling overrunsBefore applying any rate limiting configurations, it is important to understand your workloads and traffic. AWS provides many services to help you monitor and understand your applications. For example, enablingAWS X-Rayon API Gateway gives you a full view of the request coming from the user into your application. CloudWatch collects monitoring and operational data in logs, metrics, and events, providing you with a unified view of AWS resources, applications, and services. You can understand the normal state of your applications byeffectively monitoring them. Once you understand your workloads, you can determine which layers and attributes are best suited to be rate limited.ConclusionIn this blog post, we examined various methods and patterns to throttle your serverless applications as a means to control costs as well as avoid overloading downstream components. Consider using the strategies in this blog to implement throttling mechanisms when designing your serverless applications.Sharon LiSharon Li is a solutions architect at AWS, based in the Boston, MA area. She works with enterprise customers helping them solve difficult problems and build on AWS. Outside of work, she likes to spend time with her family and explore local restaurants.Akhil AendapallyAkhil Aendapally is an AWS Solutions Architect focused on helping customers with their AWS adoption. He holds a master’s degree in Network and Computer Security. Akhil has 8+ years of experience working with different cloud platforms, Infrastructure automation, and Microsoft technologies.Ashish LagwankarAshish Lagwankar is a Senior Enterprise Solutions Architect at AWS. His core interests include AI/ML, serverless, and container technologies. Ashish is based in the Boston, MA, area and enjoys reading, outdoors, and spending time with his family.
Serverless technologies reduce the work needed to set up and maintain computing resources, provide built-in scalability, and optimize agility, performance, cost, and security. The pay-as-you-go model is particularly liberating for developers. You can fail fast, experiment more, and do it fairly cheaply. However, serverless brings its own challenges. In this blog, we’ll examine how to […]
Sharon Li
Amazon API Gateway Amazon CloudWatch AWS Fargate AWS Lambda AWS WAF AWS X-Ray Kinesis Data Streams Serverless