Use direct service integrations to optimize your architecture
AWS Architecture BlogUse direct service integrations to optimize your architectureby Jerome Van Der Linden | on16 MAY 2022| inAmazon API Gateway,Amazon DynamoDB,Amazon EventBridge,Architecture,AWS Lambda,AWS Step Functions,AWS X-Ray|Permalink|ShareWhen designing an application, you must integrate and combine several AWS services in the most optimized way for an effective and efficient architecture:Optimize for performanceby reducing the latency between servicesOptimize for costs operability and sustainability, by avoiding unnecessary components and reducing workload footprintOptimize for resiliencyby removing potential point of failuresOptimize for securityby minimizing the attack surfaceAs stated in theServerless Application Lensof theWell-Architected Framework, “If yourAWS Lambdafunction is not performing custom logic while integrating with other AWS services, chances are that it may be unnecessary.” In addition,Amazon API Gateway,AWS AppSync,AWS Step Functions,Amazon EventBridge, and Lambda Destinations can directly integrate with a number of services. These optimizations can offer you more value and less operational overhead.This blog post will show how to optimize an architecture with direct integration.Workflow example and initial architectureFigure 1 shows a typical workflow for the creation of an online bank account. The customer fills out a registration form with personal information and adds a picture of their ID card. The application then validates ID and address, and scans if there is already an existing user by that name. If everything checks out, a backend application will be notified to create the account. Finally, the user is notified of successful completion.Figure 1. Bank account application workflowThe workflow architecture is shown in Figure 2 (click on the picture to get full resolution).Figure 2. Initial account creation architectureThis architecture contains 13 Lambda functions. If you look at the code onGitHub, you can see that:Five of these Lambda functions are basic and perform simple operations:One function to execute the Step Functions state machineTwo functions to performcreate, read, update, and delete(CRUD) operations onAmazon DynamoDBTwo functions to send messages onAmazon Simple Queue Service (Amazon SQS)or EventBridgeAdditional Lambda functions perform other tasks, such as verification and validation:One function generates a presigned URL to upload ID card pictures toAmazon Simple Storage Service (Amazon S3)One function uses theAmazon TextractAPI to extract information from the ID cardOne function verifies the identity of the user against the information extracted from the ID cardOne function performs simple HTTP request to a third-party API to validate the addressFinally, four functions concern the websocket (connect,message, anddisconnect) and notifications to the user.Opportunities for improvementIf you further analyze the code of the five basic functions (seestartWorkflowon GitHub, for example), you will notice that there are actually three lines of fundamental code that start the workflow. The others 38 lines involve imports, input validation, error handling, logging, and tracing. Remember that all this code must be tested and maintained.import os
import json
import boto3
from aws_lambda_powertools import Tracer
from aws_lambda_powertools import Logger
import re

logger = Logger()
tracer = Tracer()

sfn = boto3.client('stepfunctions')

PATTERN = re.compile(r"^arn:(aws[a-zA-Z-]*)?:states:[a-z]{2}((-gov)|(-iso(b?)))?-[a-z]+-\d{1}:\d{12}:stateMachine:[a-zA-Z0-9-_]+$")

if ('STATE_MACHINE_ARN' not in os.environ
    or os.environ['STATE_MACHINE_ARN'] is None
    or not PATTERN.match(os.environ['STATE_MACHINE_ARN'])):
    raise RuntimeError('STATE_MACHINE_ARN env var is not set or incorrect')

STATE_MACHINE_ARN = os.environ['STATE_MACHINE_ARN']

@logger.inject_lambda_context
@tracer.capture_lambda_handler
def handler(event, context):
    try:
        event['requestId'] = context.aws_request_id

        sfn.start_execution(
            stateMachineArn=STATE_MACHINE_ARN,
            input=json.dumps(event)
        )

        return {
            'requestId': event['requestId']
        }
    except Exception as error:
        logger.exception(error)
        raise RuntimeError('Internal Error - cannot start the creation workflow') from errorAfter running this workflow several times and reviewing theAWS X-Raytraces (Figure 3), we can see that it takes about 2–3 seconds when functions are warmed:Figure 3. X-Ray traces when Lambda functions are warmedBut the process takes around 10 seconds with cold starts, as shown in Figure 4:Figure 4. X-Ray traces when Lambda functions are coldWe use an asynchronous architecture to avoid waiting time for the user, as this can be a long process. We also use WebSockets to notify the user when it’s finished. This adds some complexity, new components, and additional costs to the architecture. Now let’s look at how we can optimize this architecture.Improving the initial architectureDirect integration with Step FunctionsStep Functions can directlyintegrate with some AWS services, including DynamoDB, Amazon SQS, and EventBridge, andmore than 10,000 APIsfrom 200+ AWS services. With these integrations, you can replace Lambda functions when they do not provide value. We recommend using Lambda functions to transform data, not to transport data from one service to another.In our bank account creation use case, there are four Lambda functions we can replace with direct service integrations (see large arrows in Figure 5):Query a DynamoDB table to search for a userSend a message to an SQS queue when the extraction failsCreate the user in DynamoDBSend an event on EventBridge to notify the backendFigure 5. Lambda functions that can be replacedIt is not as clear that we need to replace the other Lambda functions. Here are some considerations:To extract information from the ID card, we use Amazon Textract. It is available through the SDK integration in Step Functions. However, the API’s response provides too much information. We recommend using a library such asamazon-textract-response-parserto parse the result. For this, you’ll need a Lambda function.The identity cross-check performs a simple comparison between the data provided in the web form and the one extracted in the ID card. We can perform this comparison in Step Functions using aChoicestate and several conditions. If the business logic becomes more complex, consider using a Lambda function.To validate the address, we query a third-party API. Step Functions cannot directly call a third-party HTTP endpoint, but because it’s integrated with API Gateway, we cancreate a proxy for this endpoint.If you only need to retrieve data from an API or make a simple API call, use the direct integration. If you need to implement some logic, use a Lambda function.Direct integration with API GatewayAPI Gateway also provides service integrations. In particular, we can start the workflow without using a Lambda function. In the console, select the integration type “AWS Service”, the AWS service “Step Functions”, the action “StartExecution”, and “POST” method, as shown in Figure 6.Figure 6. API Gateway direct integration with Step FunctionsAfter that, use amapping templatein the integration request to define the parameters as shown here:{"stateMachineArn":"arn:aws:states:eu-central-1:123456789012:stateMachine: accountCreationWorkflow","input":"$util.escapeJavaScript($input.json('$'))"}We can go further and remove the websockets and associated Lambda functionsconnect,message, anddisconnect. By usingSynchronous Express Workflowsand theStartSyncExecutionAPI, we can start the workflow and wait for the result in a synchronous fashion. API Gateway will then directly return the result of the workflow to the client.Final optimized architectureAfter applying these optimizations, we have the updated architecture shown in Figure 7. It uses only two Lambda functions out of the initial 13. The rest have been replaced by direct service integrations or implemented in Step Functions.Figure 7. Final optimized architectureWe were able to remove 11 Lambda functions and their associated fees. In this architecture, the cost is mainly driven by Step Functions, and the main price difference will be your use ofExpress Workflows instead of Standard Workflows. If you need to keep some Lambda functions, useAWS Lambda Power Tuningto configure your function correctly and benefit from the best price/performance ratio.One of the main benefits of this architecture is performance. With the final workflow architecture, it now takes about 1.5 seconds when the Lambda function is warmed and 3 seconds on cold starts (versus up to 10 seconds previously), see Figure 8:Figure 8. X-Ray traces for the final architectureThe process can now be synchronous. It reduces the complexity of the architecture and vastly improves the user experience.An added benefit is that by reducing the overall complexity and removing the unnecessary Lambda functions, we have also reduced the risk of failures. These can be errors in the code, memory or timeout issues due to bad configuration, lack of permissions, network issues between components, and more. This increases the resiliency of the application and eases its maintenance.TestingTestability is an important consideration when building your workflow. Unit testing a Lambda function is straightforward, and you can use your preferred testing framework and validate methods. Adopting ahexagonal architecturealso helps remove dependencies to the cloud.When removing functions and using an approach with direct service integrations, you are by definition directly connected to the cloud. You still must verify that the overall process is working as expected, and validate these integrations.You can achieve this kind of tests locally usingStep Functions Local, and the recently announcedMocked Service Integrations. By mocking service integrations, for example, retrieving an item in DynamoDB, you can validate the different paths of your state machine.You also have to perform integration tests, but this is true whether you use direct integrations or Lambda functions.ConclusionThis post describes how to simplify your architecture and optimize for performance, resiliency, and cost by using direct integrations in Step Functions and API Gateway. Although many Lambda functions were reduced, some remain useful for handling more complex business logic and data transformation. Try this out now by visiting theGitHub repository.For further reading:Building well-architected serverless applications: Optimizing application performance – part 3AWS Step Functions Supports 200 AWS Services To Enable Easier Workflow AutomationJerome Van Der LindenJerome is a Solutions Architect at AWS, where he works for different customers in Switzerland, helping them on their journey to the cloud. As a former developer, he built a strong experience on software craftsmanship and devops practices. He's also passionate about serverless. In his spare time, and when snow is here, he enjoys skiing with his kids. Reach out to him on twitter: @jeromevdl.
When designing an application, you must integrate and combine several AWS services in the most optimized way for an effective and efficient architecture: Optimize for performance by reducing the latency between services Optimize for costs operability and sustainability, by avoiding unnecessary components and reducing workload footprint Optimize for resiliency by removing potential point of failures […]
Jerome Van Der Linden
Amazon API Gateway Amazon DynamoDB Amazon EventBridge Architecture AWS Lambda AWS Step Functions AWS X-Ray