Search This Blog

AWS Lambda vs EC2



best fit for all use cases? 



What is AWS EC2?



  • It can scale up and down instantly based on the computing requirement.
  • It provides an interface to configure capacity with minimal effort.
  • It allows complete admin access to the servers, making infrastructure management straightforward. It also enables monitoring, security and support for multiple instance types (wide variety of Operating Systems, Memory, and CPUs).

What is AWS Lambda?






  • Lambda logs can be monitored the same way as EC2, through CloudWatch.
  • Unlike EC2, it is charged based on the execution time and memory used.


of performance, cost, security, and other aspects:

Setup & Management





So here, we can see Lambda is a clear winner.

On-Demand vs. Always Available







in the long run.



them one by one.

1. Concurrency and Scaling








2. Dependencies




CPU as well.






EC2 is a clear winner here.

3. Latency









latency will be less.


4. Timeout



such kind of restriction. 






of examples:






will cost us only $7.62. 







Lambda as it is stateless in nature.




track it.

check alarm to notify you when a status check fails.


its resource usage until eventually hitting a ceiling and becoming unavailable.



that's where we would need to use APM tools.

missing in CloudWatch Logs.

still, a few gaps to be filled.

health of the Lambda function out of the box:

- Invocation Count - Execution Duration - Error Count - Throttled Count

CloudWatch Metrics. 



Use Cases 

Recovery. Again, EC2 would be the only choice. 

one to use. 

and is an ideal fit for these kinds of requirements.

as well.

like increased cold start time, limited concurrent executions, etc.

is no need to run the server all the time.

Framework are addressing those concerns. Local testing is another aspect you need to consider





PCF 2.6 Feature - Tag your Deployments with Revisions


            Developers, who have been using PCF, have always been asking this question that how do we know which version is running on PCF. We used to check this in on-premise by going to that VM and check the jar version. Also, how to do rollback smoothly for PCF deployments as doing it at SCM (Github) or CI/CD level is painful and risk-prone. It involves verification, approval and other deployment processes.
With PCF 2.6, Pivotal has come up with a feature called "App Revisions". Using this feature, we can rollback the deployment for an application very easily. In this article, we will try to understand this feature and walk through the steps for rollback.

What is App Revisions -

A revision represents code and configuration used by an app at a specific time. In PCF terms, it is an object which will contain references to a droplet, environment variables, and custom start command. Have a look at a simple example below:

In the sample above, we have retrieved all the revisions for an application based on it's GUID using PCF CAPI. In line 15, it has a "resources" object. This object contains all the revisions for an application. Resources will have an array of objects and each object will have a guid which represents a revision. The "description" node will provide a PCF configured description of the revision. For example, in this case, the first revision description at line 30 tells that it is the "Initial version" which got deployed for this application.
Now, whenever there is a change in the application deployment, it will create a new revision with a new GUID. Following are the events which will trigger the new revision:
  • A new droplet is created which happens when you do "cf push" after making a change to the code.
  • An environment variable is added or changed for the application.
  • A custom start command is added or changed in the manifest file.
  • An app rolls back to a previous version.

Steps to Follow to Enable, View, and Rollback Revisions -

Step 1 - Take an application that can run on PCF. Deploy it on PCF using cf push command.
Step 2 - Now, get the guid of the running app by executing this command:

Step 3 - Run this command to enable the revision for this app:

Step 4 - Let's check what is the current revision for this deployed application. You will notice an "Initial revision" as a description for the first revision after enabling it:

Step 5 -  Now, we will use the environment variable event to trigger the new revision for this example. Add an environment variable to the application using below command and then restage the application to reflect the changes:

Step 6 -  Lets, check if there is any new revision that has been added for the application. You will notice a new guid is added in the resources node with a description "New droplet deployed. New environment variables deployed.":

That's the way it keep creating new revisions for each event mentioned above in the article.
Step 7 - If we observe that the description node of the revision is predefined text configured by PCF. We don't have control to change it.  Let's suppose we do multiple "New Droplet deployed" type of events. How would we be able to recognize the particular revision? To overcome this, we need to tag each revision with a proper Release version and description. We can use the "metadata" object for this purpose. PCF v3 CAPI provides access to revisions to update the metadata object using the CAPI PATCH request. See the example below:

You will see output like below:

It has been observed that the key_value under metadata does not accept space. It should be Alphanumeric characters only. Though, Empty values are allowed.

Step 8 -  Let's set up one more env variable to create one more revision so that we can try to rollback.

Step 9 - Let's now see how to rollback the changes and go back to a previous revision. Going back to the previous version is very easy and can be done with one command:

We should now be able to see the new revision with the application pointing to the revision configured in the command. You can run below command to check the current revision:

Findings -

  1. If an environment variable is added in the manifest file and pushed to PCF using "cf push", it will create 2 revisions. One with "New environment variables deployed" and second with "New Droplet Deployed" descriptionThis may create confusion while doing rollback.
    To overcome this issue, we can set the environment variable using the "cf set-env" command and then restage the application. This will create only one revision with "New droplet deployed. New environment variables deployed." description.

  2. If you try to roll back to the current version itself, it will not throw any error and create a new revision with a new GUID. This should be raised as a bug to PCF.

  3. When you do roll back to the previous version, it ensures that it has a currently configured number of instances running. For example, if your application has 2 instances. it will first spin up 1 new instance with revised revision and then destroy the existing instance containers. So, It is ensuring that there is no downtime caused while doing the rollback. However, we need to ensure the capacity for 1 new instance is considered before doing this activity. Also, Users may experience different behavior of the application momentarily as both an old and new version of the apps will be live for a very small period.

  4. While doing a rollback, keep in mind that by default, PCF retains only the 5 most recent staged droplets in its droplets bucket. So, it cannot go back until this value is increased. Please note every revision does not mean a change in the droplet. PCF retains max 100 revisions by default.


           In this article, we have seen how the App Revision feature works and how we can use this feature to do a rollback. I believe, it will further smoothen the process of deployments in PCF. I would recommend to put all these above commands in your CI/CD pipeline and use this feature to tag your deployments in PCF.

AWS Step Functions Lesson Learned


Many articles have been written about AWS Step Functions since it was first introduced in 2016.
Most of them create the impression that the service is simply an extension of the Lambda function
that allows us to stitch together multiple Lambda functions to call each other.

But actually, it is much more than that. Step Functions allows us to design and build the flow of execution
of modules in our application in a simplistic manner. This enables a developer to focus solely on ensuring
that each module performs its intended task, without having to worry about connecting each module with others.

In this article, we will explore the what, the how and the why of Step Functions, before walking through
some use cases, limitations and best practices around using the service.

What is AWS Step Functions?


Step Functions is built on two main concepts Tasks and State Machine



Can you imagine if you had to do it yourself using a Messaging Queue, Istio or App Mesh? It would be a big task,
and that’s without considering the overhead of actually maintaining that component.

It's really great to see what features it provides out of the box. However, it would have been even better if AWS had
added the ability to design it visually rather than through JSON.

How Step Functions works

As discussed earlier, the state machine is a core component of the AWS Step Functions service. It defines communication
between states and how data is passed from one state to another.


A state is referred to by its name, which can be any string but must be unique within the scope of the entire state machine.
It does the following functions:
  • Performs some work in the state machine (a Task state).
  • Makes a choice between branches of execution (a Choice state).
  • Stops execution with failure or success (a Fail or Succeed state).
  • Simply passes its input to its output or injects some fixed data (a Pass state).
  • Provides a delay for a certain amount of time or until a specified time/date (a Wait state).
  • Begins parallel branches of execution (a Parallel state).

 Here is an example of a state definition for Task type:

"States": {
    "FirstState": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:ap-southeast-2:710187714096:function:DivideNumbers",
      "Next": "ChoiceState"

Input and Output Processing

For Step Functions, the input is always passed as a JSON file to the first state. However, it has
to pass through InputPath, ResultPath, and OutputPath before the final output is generated.
The JSON output is then passed to the next state. 

source: AWS

InputPath - selects which parts of the JSON input to pass to the task of the Task state (for example, an AWS Lambda function).

ResultPath then selects what combination of the state input and the task result to pass to the output.

OutputPath can filter the JSON output to further limit the information that's passed to the output.

Let's take a look at an example to better understand this in detail:

For Lambda execution, Input is described as JSON like above. That input is bound to the symbol $ and passed
on as the input to the first state in the state machine.

By default, the output of each state would be bound to $ and becomes the input of the next state. Each state,
we have InputPath, ResultPath and OutputPath attributes that filter the input and provide the final output.
In the above scenario, “ExamResults” state is filtering “lambda” node, appending the result of a state execution to
results” node and the final output is just “result” node rather than the whole JSON object: 

Hence, the final output will be:
  "math": 80,
  "eng": 93,
  "total": 173

Step Functions can be triggered in four ways :

As I mentioned earlier, Step Functions is not only about Lambda Functions. It has support for several other
Integration Patterns like SQS, DynamoDB, SNS, ECS and many others.

Step Functions use cases

There are many use cases that can be resolved using Step Functions. However, we’ll restrict ourselves to a
few major ones here:

1. Sequence Batch Processing Job

If you have many Batch Jobs to be processed sequentially and need to coordinate the data between those,
this is the best solution. For example, an e-commerce website can first read the product data and the next job
will find out which products are running out of stock soon and then, the third job can send a notification to all the
vendors to expedite the supply process. 

2. Easy Integration with Manual Tasks

If a workflow needs manual approval/intervention, Step Function would be the best solution to coordinate it.
For example, the Employee promotion process - It needs approval from the manager. So the Step function can
send the email using AWS SES service with Approve or Reject link and once receives it, can trigger the next action
using lambda or ECS jobs.

3. Coordinate Container Task in Serverless Workflow

A step function can help to make a decision about how best to process data. Based on the file size, you can decide
to use either lambda, ECS or on-premise activity to optimize the cost and runtime both

Step Functions benefits

  • Retry: Before Step Functions, there was no easy way to retry in the event of a timeout error or runtime error or

any other type of failure. It also provides an exponential backoff feature.

"Retry": [ {
    "ErrorEquals": [ "States.Timeout" ],
    "IntervalSeconds": 3,
    "MaxAttempts": 2,
    "BackoffRate": 1.5
 } ]

  • Error Handling: It provides an easy way of error handling at each state. It can handle several types of errors
a state can throw out, like:

  • States.Timeout - When a Task state cannot finish the job within the TimeoutSeconds or does not send
heartbeat using SendTaskHeartbeat within HeartbeatSeconds value
  • States.TaskFailed - When a Task state fails for any reason.
  • States.Permissions - When a Task state does not have sufficient privileges to call the Lambda/Activity code.
  • States.All - It captures any known error name. 

It can also catch Lambda service exceptions(Lambda.ServiceException) and even the unhandled errors (Lambda.Unknown).
A typical example of error handling: 
"Catch": [ {
    "ErrorEquals": [ "States.TaskFailed", “States.Permission” ],
     "Next": “state x”
 } ]
You can bet that it was never this easy to implement error handling like this with any other workflow solution.

  • Parallelization: You can parallelize the work declaratively. A step machine can have a state calling multiple states in
parallel. This will make the workflow complete faster.
  • High Execution Time: Step Functions has one year as max execution time so if some of the tasks of the workflow are high (more than 15 minutes), they can be run on ECS or EC2 or as an Activity hosted outside of AWS.
  • Pricing: Step Functions counts a state transition each time a step of a workflow is executed. You are charged for the total number of state transitions across all your state machines. Charges are a little on the higher side, but it would almost certainly be costlier to come up with your own solution for orchestrating the different services, Error Handling, Retries, and the many other features Step Function provide.


Despite all the powerful features Step Functions offers, there are still a few things missing:
  • Shorter Execution History: The maximum limit for keeping execution history logs is 90 days. It cannot be extended and
that may preclude the use of Step Functions for businesses that have longer retention requirements.
  • Missing Triggers: Some Event Sources and Triggers are still missing, such as DynamoDB and Kinesis.
  • State machine Execution name: Each Execution name for a state machine has to be unique (not used in the last 90 days).
This can be very tricky.
  • AWS Step Functions does not horizontally scale to dynamically launch AWS Lambda functions in parallel.
For example, if my state 1 generates 10 messages, it cannot spin up 10 AWS Lambda invocations at state
2 to process those messages individually (This feature is available if you use Lambda function directly with concurrent
execution). State 2 is statically defined in the state machine and provisions the same number of task
executors each time. Therefore, it cannot scale dynamically. Hopefully, this feature may be added in the future.

Step Function Best Practices

  • In a workflow sometimes, we would like to resume the process from the fail state as opposed to re-running it from the
beginning. This isn’t provided as a built-in feature, but there is a workaround to achieve this.

  • Beware: State Machine can run infinitely. It has a max execution time of one year. That itself is too high. On top of that,
it provides a feature "Continue as new Execution". This allows you to start a new execution before terminating your current
running execution. This opens up the possibility of it running infinitely by mistake. Monitoring Execution metrics is a good way
to identify and fix those mistakes.

  • AWS Step Functions has a hard limit of 25,000 event entries in the execution history (with a retention of 90 days),
which is going to fill very quickly for a long-running execution. To overcome this limitation, we can implement
“Continue as new Execution” pattern, where we can spin up the new execution from an existing running execution.
For example, if a long running execution has 10 steps and you’re expecting to have 40,000 event entries in your execution
history, you may configure to start a new execution at step 5. And it will create a total of two executions of the state machine.
Therefore, it will distribute the entries between those with steps 1-5 and step 6-10.

  • By default, the Amazon State Language doesn't set timeouts in state machine definitions. In a scenario where a
Lambda Function or Activity has a problem and keeps running without responding back to Step Functions, it will keep waiting
for a year (max timeout) at least. To prevent this, set the timeout using TimeoutSeconds like this:
"ExamResults": {
      "Type": "Task",
       "TimeoutSeconds": 200,
       "HeartbeatSeconds": 30,
      "End": true

  • Using TimeoutSeconds & HeartbeatSeconds, we can design a long-running workflow alive.

HeartbeatSeconds value should be less than TimeoutSeconds. And, we need to use
SendTaskHeartbeat periodically within the time we set in HeartbeatSeconds in our state machine
task definition to keep the task from timing out.

Let’s take an example where HeartbeatSeconds value is 30 seconds and TimeoutSeconds is
400 seconds for a long activity worker process. When the state machine and activity worker
process starts, the execution pauses at the activity task state and waits for your activity worker to
poll for a task. Once a taskToken is provided to your activity worker, your workflow will wait for
SendTaskSuccess or SendTaskFailure to provide a status.  If the execution doesn't receive either
of these or a SendTaskHeartbeat call before the time configured in TimeoutSeconds, the execution
will fail and the execution history will contain an ExecutionTimedOut event. So, by configuring these,
we can design a long-running workflow effectively.

  • For longer workflows, keep in mind that API Gateway calls the state machine in an
asynchronous way and responds back only to the execution ARN. You can define another
method in API Gateway to call "DescribeExecution" API. This method will have to call this
API periodically (poll) to get the output until the returned status is no longer "RUNNING".
Some developers use Step Functions for their API calls and want to send the response of
the state machine output to a UI or other system. Firstly, developers should avoid using
Step Functions for Microservice API calls. Microservices are supposed to be small and
respond back in 2-3 seconds, if not milliseconds.

  • Some developers use Step Functions for their API calls and want to send the response of the
state machine output to a UI or other system. Firstly, developers should avoid using Step Functions
for Microservice API calls. Microservices are supposed to be small and respond back
in 2-3 seconds, if not milliseconds.
  • Like any other application code, handle the exception. Use the Retry feature to handle occasional transient service errors.

Logging and Monitoring Step Functions

Similar to Lambda functions, Step Functions also sends logs to CloudWatch and it generates several metrics around it.
For example, Execution metrics, Activity metrics, Lambda metrics, etc. Below is an example of Execution Metrics:


Visual Workflow panel shows the current status of the execution. Look at the right side of the panel (below picture).
We can see the details of any state by clicking on the state in the workflow diagram. It shows the input, output, and
an exception (if any) for the state.  


It also logs the execution history and status for each state. AWS Console does provide a nice
visual of the states from start to end. We can also click on CloudWatch Logs to go to LogGroups
and see detail logs.

One recommendation is to create a Unique Trace ID which should be passed to all the integration
services these states connect to. It will help to track the transactions easily.
It also has integration with CloudTrail to log the events.


In this article, we explored the basic concepts of Step Functions and how it works. We also talked
about how with the Visualization panel, Error Handling and Retry features, it makes the workflow
creation process much smoother. Step Functions should properly be described as state-as-a-service.
Without it, we would not be able to maintain the state of each execution having multiple lambda

Just keep in mind that you need to keep a watch on your bills as it can burn a hole in your pocket very fast.
And the best way to do that is to ensure that proper monitoring and metrics are in place.

Most Viewed Posts