A serverless solution for invoking AWS Lambda at a sub-minute frequency
AWS Architecture BlogA serverless solution for invoking AWS Lambda at a sub-minute frequencyby Emanuele Menga | on11 MAY 2018| inAmazon CloudWatch,Architecture,AWS Step Functions,Serverless|Permalink|ShareIf you’ve usedAmazon CloudWatch Eventsto schedule the invocation of a Lambda function at regular intervals, you may have noticed that the highest frequency possible is one invocation per minute. However, in some cases, you may need to invoke Lambda more often than that. In this blog post, I’ll cover invoking a Lambda function every 10 seconds, but with some simple math you can change to whatever interval you like.To achieve this, I’ll show you how to leverage AWS Step Functions.The SolutionFor this example, I’ve created a Step Functions State Machine that invokes our Lambda function 6 times, 10 seconds apart. Such State Machine is then executed once per minute by a CloudWatch Events Rule. The result is our Lambda function being invoked every 10 seconds, indefinitely.Below is a diagram illustrating how the various services work together.Step 1:My sampleLambda function doesn’t actually do anything, it just simulates an execution for a few seconds. This is the (Python) code of my dummy function:import time
import random

rand = random.randint(1, 3)
    print('Running for {} seconds'.format(rand))
    time.sleep(rand)
    return TrueStep 2:The next step is to create a second Lambda function, that I called Iterator, which has two duties:It keeps track of the current number of iterations, since Step Function doesn’t natively have a state we can use for this purpose.It asynchronously invokes our Lambda function at every loops.This is the code of the Iterator, adapted fromhere.import boto3

client = boto3.client('lambda')

def lambda_handler(event, context):
    index = event['iterator']['index'] + 1
    response = client.invoke(
        FunctionName='LAMBDA_TO_INVOKE',
        InvocationType='Event'
    )
return {
        'index': index,
        'continue': index < event['iterator']['count'],
        'count': event['iterator']['count']
    }Now we can create the Step Functions State Machine; the definition is, again, adapted fromhere.{
    "Comment": "Invoke Lambda every 10 seconds",
    "StartAt": "ConfigureCount",
    "States": {
        "ConfigureCount": {
            "Type": "Pass",
            "Result": {
                "index": 0,
                "count": 6
            },
            "ResultPath": "$.iterator",
            "Next": "Iterator"
        },
        "Iterator": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:Iterator",
            "ResultPath": "$.iterator",
            "Next": "IsCountReached"
        },
        "IsCountReached": {
            "Type": "Choice",
            "Choices": [
                {
                    "Variable": "$.iterator.continue",
                    "BooleanEquals": true,
                    "Next": "Wait"
                }
            ],
            "Default": "Done"
        },
        "Wait": {
            "Type": "Wait",
            "Seconds": 10,
            "Next": "Iterator"
        },
        "Done": {
            "Type": "Pass",
            "End": true
        }
    }
}This is how it works:The state machine starts and sets the index at 0 and the count at 6.Iterator function is invoked.If the iterator function reached the end of the loop, the IsCountReached state terminates the execution, otherwise the machine waits for 10 seconds.The machine loops back to the iterator.Step 3:Create an Amazon CloudWatch Events rule scheduled to trigger every minute and add the state machine as its target. I’ve actually prepared an AWS CloudFormation template that creates the whole stack and starts the Lambda invocations, you can find it in thisGitHub Repository.PerformanceLet’s have a look at a sample series of invocations and analyse how precise the timing is. In the following chart I reported the delay (in excess of the expected 10-second-wait) of 30 consecutive invocations of my dummy function, when the Iterator is configured with a memory size of 1024MB.Invocations DelayNotice the delay increases by a few hundred milliseconds at every invocation. The good news is it accrues only within the same loop, 6 times; after that, a new CloudWatch Events kicks in and it resets.This delay is due to the work that AWS Step Function does outside of the Wait state, the main component of which is the Iterator function itself, that runs synchronously in the state machine and therefore adds up its duration to the 10-second-wait.As we can easily imagine, the memory size of the Iterator Lambda function does make a difference. Here are the Average and Maximum duration of the function with 256MB, 512MB, 1GB and 2GB of memory.Average DurationMaximum DurationGiven those results, I’d say that a memory of 1024MB is a good compromise between costs and performance.CaveatsAs mentioned, in our Amazon CloudWatch Eventsdocumentation, in rare cases a rule can be triggered twice, causing two parallel executions of the state machine. If that is a concern, we can add atask stateat the beginning of the state machine that checks if any other executions are currently running. If the outcome is positive, then achoice statecan immediately terminate the flow. Since the state machine is invoked every 60 seconds and runs for about 50, it is safe to assume that executions should all be sequential and any parallel executions should be treated as duplicates. The task state that checks for current running executions can be a Lambda function similar to the following:import boto3

client = boto3.client('stepfunctions')

def lambda_handler(event, context):
    response = client.list_executions(
        stateMachineArn='arn:aws:states:REGION:ACCOUNTID:stateMachine:LambdaSubMinute',
        statusFilter='RUNNING'
    )
    return {
        'alreadyRunning': len(response['executions']) > 1
    }Emanuele MengaEmanuele Menga is a Cloud Support Engineer.
If you’ve used Amazon CloudWatch Events to schedule the invocation of a Lambda function at regular intervals, you may have noticed that the highest frequency possible is one invocation per minute. However, in some cases, you may need to invoke Lambda more often than that. In this blog post, I’ll cover invoking a Lambda function every […]
Emanuele Menga
Amazon CloudWatch Architecture AWS Step Functions Serverless