Optimize Cost by Automating the Start/Stop of Resources in Non-Production Environments
AWS Architecture BlogOptimize Cost by Automating the Start/Stop of Resources in Non-Production Environmentsby Ashutosh Pateriya | on23 DEC 2021| inArchitecture,AWS CloudFormation,AWS Step Functions,Technical How-to|Permalink|ShareCo-authored with Nirmal Tomar, Principal Consultant, Infosys Technologies Ltd.Ease of creating on-demand resources on AWS can sometimes lead to over-provisioning or under-utilization of AWS resources likeAmazon EC2andAmazon RDS. This can lead to higher costs that can often be avoided with proper planning and monitoring.  Non-critical environments, like development and test are often not monitored on a regular basis and can result in under-utilization of AWS resources.In this blog, we discuss a common AWS cost optimization strategy, which is an automated and deployable solution to schedule the start/stop of AWS resources. For this example, we are considering non-production environments because in most scenarios these do not need to be available at all time. By following this solution, cloud architects can automate the start/stop of services per their usage pattern and can save up to 70% of costs while running their non-production environment.The solution outlined here is designed to automatically stop and start the following AWS services based on your needs; Amazon RDS,Amazon Aurora, Amazon EC2,Auto Scaling groups,AWS Beanstalk, andAmazon EKS. The solution is automated usingAWS Step functions,AWS Lambda,AWS Cloud​Formation Templates,Amazon EventBridgeandAWS Identity and Access Management (IAM).In this solution, we also provide an option to exclude specificAmazon Resource Names (ARNs)of the aforementioned services. This helps cloud architects to exclude the start/stop function for various use cases like in a QA environment when they don’t want to stop Aurora or they want to start RDS in a Development environment. The solution can be used to start/stop the services mentioned previously on a scheduled interval but can also be used for other applicable services likeAmazon ECS,Amazon SageMaker Notebook Instances,Amazon Redshiftand many more.Note–Don’tset up this solution in a production or other environment where you require continuous service availability.PrerequisitesFor this walkthrough, you should have the following prerequisites:AnAWS accountwith permission to spin up required resources.A runningAmazon Aurorainstance in the source AWS account.WalkthroughTo set up this solution, proceed with the following two steps:Set up the step function workflow to stop services using aCloudFormation template. On a scheduled interval, this workflow will run and stop the chosen services.Set up the step function workflow to start services using aCloudFormation template. On a scheduled interval, this workflow will run and start services as configured during the CloudFormation setup.1. Stop Services using the Step Function workflow for a predefined durationFigure 1 – Architecture showing the AWS Step Functions Workflow to stop servicesThe AWS Lambda functions involved in this workflow:StopAuroraCluster:  This Lambda function will stop all Aurora Cluster setup across Region including read replica.StopRDSInstances:  This Lambda function will stop all RDS Instances except the Aurora setup across the Region.ScaleDownEKSNodeGroups: This Lambda function will downsize all nodegroups to zero instance across the Region.ScaleDownASG: This Lambda function will downsize all Auto Scaling groups including the Elastic Beanstalk Auto Scaling group to zero instance across Region. We can edit CloudFormation templates to include the custom value.StopEC2Instances: This Lambda function will stop all EC2 instances set up across the Region.Using the following AWS CloudFormation Template, we set up the required services and workflow:a. Launch the template in the source account and source Region:b.     Fill out the preceding form with the following details and selectNext.Stack name:Stack Name which you want to create.ExcludeAuroraClusterArnListInCommaSeprated:Comma separated Aurora clusters ARN which enterprises don’t want to stop, keep the default value if there is no exclusion list.e.g.  arn:aws:rds:us-east-1:111111111111:cluster:aurorcluster1, arn:aws:rds:us-east-2:111111111111:cluster:auroracluster2ExcludeRDSDBInstancesArnListInCommaSeprated: Comma separated DB instances ARN which enterprises don’t want to stop, keep the default value if there is no exclusion list.e.g.   arn:aws:rds:us-east-1:111111111111:db:rds-instance-1, arn:aws:rds:us-east-2:111111111111:db:rds-instance-2ExcludeEKSClusterNodeGroupsArnListInCommaSeprated:Comma separated EKS Clusters ARN which enterprises don’t want to stop, leave it with default value if there is no exclusion list.e.g.   arn:aws:eks:us-east-2:111111111111:cluster/testclusterExcludeAutoScalingGroupIncludingBeanstalkListInCommaSeprated:Comma separated Beanstalk and other Auto Clusters groups ARN (except Amazon EKS) which enterprises don’t want to stop, keep the default value if there is no exclusion list.e.g.  arn:aws:autoscaling:us-east-1:111111111111:autoScalingGroup:6d5af669-eb3b-4530-894b-e314a667f2e7:autoScalingGroupName/test-0-ASGExcludeEC2InstancesIdListInCommaSeprated:Comma separated EC2 instance ID’s which you don’t want to stop, keep the default value if there is no exclusion list.e.g.  i-02185df0872f0f852, 0775f7e39513c50ddScheduleExpression:Schedule a cron expression when you want to run this workflow. Sample expressions are available in this guide,Schedule expressions using rate or cron.c.     SelectIAM roleto launch this template.  As a best practice, select theAWS CloudFormation service roleto manage AWS services and resources  available to each user.d.     Acknowledge that you want to create various resources including IAM roles/policies and selectCreate Stack.2. Start Services using the Step Function workflow in pre-configured timeFigure 2 – Architecture showing the AWS Step Functions Workflow to start servicesThe  Lambda functions involved in this workflow:StartAuroraCluster:This Lambda function will start all Aurora Cluster setup across Region including read-replica.StartRDSInstances:This Lambda function will start all RDS Instances except for the Aurora setup across the Region.ScaleUpEKSNodeGroups:This Lambda function will upsize all nodegroups to minimum 2 and maximum 4 instances across Region. We can edit CloudFormation templates for custom value.ScaleUpASG:This Lambda function will Scale up all Auto Scaling group including Elastic Beanstalk Auto Scaling group to minimum 2 and maximum 4 instances across the Region. We can edit CloudFormation templates for custom value.StartEC2Instances:This Lambda function will start all EC2 instances setup across the Region.Using the following AWS CloudFormation template, we set up the required services and workflow:a. Launch the template in the source account and source Region:b. Fill out the preceding form with the following details and selectNext.Stack name:Stack Name which you want to create.ExcludeAuroraClusterArnListInCommaSeprated:Comma separated Aurora clusters the ARN which you don’t want to start, keep the default value if there is no exclusion list.For example:arn:aws:rds:us-east-1:111111111111:cluster:aurorcluster1, arn:aws:rds:us-east-2:111111111111:cluster:auroracluster2ExcludeRDSDBInstancesArnListInCommaSeprated:Comma separated databaseinstances ARN which you don’t want to start, keep default value if there is no exclusion list.For example:arn:aws:rds:us-east-1:111111111111:db:rds-instance-1, arn:aws:rds:us-east-2:111111111111:db:rds-instance-2ExcludeEKSClusterNodeGroupsArnListInCommaSeprated:Comma separated EKS Clusters ARN which you don’t want to start, keep the default value if there is no exclusion list.For example:arn:aws:eks:us-east-2:111111111111:cluster/testclusterExcludeAutoScalingGroupIncludingBeanstalkListInCommaSeprated:Comma separated Beanstalk and other Auto Clusters groups ARN (except EKS) which you don’t want to start, keep the default value if there is no exclusion list.For example:arn:aws:autoscaling:us-east-1:111111111111:autoScalingGroup:6d5af669-eb3b-4530-894b-e314a667f2e7:autoScalingGroupName/test-0-ASGExcludeEC2InstancesIdListInCommaSeprated:Comma separated EC2 instance ID s you  don’t want to start, keep the default value if there is no exclusion list.For example:i-02185df0872f0f852, 0775f7e39513c50ddScheduleExpression:Schedule a cron expression when you want to run this workflow. Sample expressions are available in this guide,Schedule expressions using rate or cron.c.     SelectIAM roleto launch this template.  As a best practice,select the AWS CloudFormation service roleto manage AWS services and resources available to each user.d. Acknowledge that you want to create various resources including IAM roles and policies and selectCreate Stack.Cleaning upDelete any unused resources to avoid incurring future charges.ConclusionIn this blog post, we outlined a solution to help you optimize cost by automating the stop/start of AWS services in non-production environments. Cost Optimization and Cloud Financial Management are ongoing initiatives. We hope you found this solution helpful and encourage you to explore additional ways to optimize cost on theAWS Architecture Center.Nirmal TomarNirmal is a principal consultant with Infosys, assisting vertical industries on application migration and modernization to the public cloud. He is the part of the Infosys Cloud CoE team and leads various initiatives with AWS. Nirmal has a keen interest in industry and cloud trends, and in working with hyperscalers to build differentiated offerings. Nirmal also specializes in cloud native development, managed containerization solutions, data analytics, data lakes, IoT, AI, and machine learning solutions.Ashutosh PateriyaAshutosh Pateriya is a Solutions Architect with Amazon Web Services. He is passionate about Networking, Security and helping customers leverage the power of AWS cloud.
Co-authored with Nirmal Tomar, Principal Consultant, Infosys Technologies Ltd. Ease of creating on-demand resources on AWS can sometimes lead to over-provisioning or under-utilization of AWS resources like Amazon EC2 and Amazon RDS. This can lead to higher costs that can often be avoided with proper planning and monitoring.  Non-critical environments, like development and test are […]
Ashutosh Pateriya
Architecture AWS CloudFormation AWS Step Functions Technical How-to