A multi-dimensional approach helps you proactively prepare for failures, Part 3: Operations and process resiliency
AWS Architecture BlogA multi-dimensional approach helps you proactively prepare for failures, Part 3: Operations and process resiliencyby Piyali Kamra, Aish Gopalan, Isael Pimentel, and Aditi Sharma | on02 SEP 2022| inAmazon CloudWatch,Amazon EventBridge,Amazon Route 53,Architecture,AWS CloudFormation,AWS CloudTrail,AWS CodeCommit,AWS Config,AWS Lambda|Permalink|ShareInPart 1andPart 2of this series, we discussed how to build application layer and infrastructure layer resiliency.In Part 3, we explore how to develop resilient applications, and the need to test and break our operational processes and run books. Processes are needed to capture baseline metrics and boundary conditions. Detecting deviations from accepted baselines requires logging, distributed tracing, monitoring, and alerting. Testing automation and rollback are part of continuous integration/continuous deployment (CI/CD) pipelines. Keeping track of network, application, and system health requires automation.In order to meet recovery time and point objective (RTO and RPO, respectively) requirements of distributed applications, we need automation to implement failover operations across multiple layers. Let’s explore how a distributed system’s operational resiliency needs to be addressed before it goes into production, after it’s live in production, and when a failure happens.Pattern 1: Standardize and automate AWS account setupCreate processes and automation for onboarding users and providing access to AWS accounts according to their role and business unit, as defined by the organization. Federated access to AWS accounts and organizations simplifies cost management, security implementation, and visibility. Having astrategyfor a suitable AWS account structure can reduce the blast radius in case of a compromise.Have auditing mechanisms in place.AWS CloudTrailmonitors compliance, improving security posture, and auditing all the activity records across AWS accounts.Practice theleast privilegesecurity model when setting up access to the CloudTrail audit logs plus network and applications logs. Followbest practiceson service control policies and IAM boundaries to help ensure your AWS accounts stay within your organization’s access control policies.ExploreAWS Budgets,AWS Cost Anomaly Detection, andAWS Cost Explorerfor cost-optimizing techniques. TheAWS Compute OptimizerandInstance Scheduler on AWSresource resizing and auto-shutdown for non-working hours.A Beginner’s Guide to AWS Cost Managementexplores multiple cost-optimization techniques.UseAWS CloudFormationandAWS Configto detect infrastructure drift and take corrective actions to make resources compliant, as demonstrated in Figure 1.Figure 1. Compliance control and drift detectionPattern 2: Documenting knowledge about the distributed systemDocument high-level infrastructure and dependency maps.Define availability characteristics of distributed system.Systems have components withvaryingRTO and RPO needs. Document application component boundaries and capture dependencies with other infrastructure components, including Domain Name System (DNS), IAM permissions; and access patterns, secrets, and certificates. Discover dependencies through solutions, such asWorkload Discovery on AWS, to plan resiliency methods and ensure the order of execution of various steps during failover are correct.Capture non-functional requirements (NFRs), such as business key performance indicators (KPIs), RTO, and RPO, for your composing services. NFRs are quantifiable and define system availability, reliability, and recoverability requirements. They should include throughput, page-load, and response time requirements. Quantify the RTO and RPO of different components of the distributed system by defining them. The KPIs measure if you are meeting the business objectives. As mentioned inPart 2: Infrastructure layer, RTO and RPO help define the failover and data recovery procedures.Pattern 3: Define CI/CD pipelines for application code and infrastructure componentsEstablish abranching strategy. Implement automated checks for version and tagging compliance in feature/sprint/bug fix/hot fix/release candidate branches, according to your organization’s policies. Define appropriate release management processes and responsibility matrices, as demonstrated in Figures 2 and 3.Test at all levels as part of an automated pipeline. This includes security, unit, and system testing. Create a feedback loop that provides the ability to detect issues and automate rollback in case of production failures, which are indicated by business KPI negative impact and other technical metrics.Figure 2. Define the release management processFigure 3. Sample roles and responsibility matrixPattern 4: Keep code in a source control repository, regardless of GitOpsMerge requests and configuration changes follow the same process as application software. Just like application code, manage infrastructure as code (IaC) by checking the code into a source control repository, submitting pull requests, scanning code for vulnerabilities, alerting and sending notifications, running validation tests on deployments, and having an approval process.You can audit your infrastructure drift, design reusable and repeatable patterns, and adhere to your distributed application’s RTO objectives by building your IaC (Figure 4). IaC is crucial for operational resilience.Figure 4. CI/CD pipeline for deploying IaCPattern 5: Immutable infrastructureAn immutable deployment pipeline launches a set of new instances running the new application version. You can customize immutability at different levels of granularity depending on which infrastructure part is being rebuilt for new application versions, as in Figure 5.The more immutable infrastructure components being rebuilt, the more expensive deployments are in both deployment time and actual operational costs. Immutable infrastructure also is easier to rollback.Figure 5. Different granularity levels of immutable infrastructurePattern 6: Test early, test oftenIn ashift-left testingapproach, begin testing in the early stages, as demonstrated in Figure 6. This can surface defects that can be resolved in a more time- and cost-effective manner compared with after code is released to production.Figure 6. Shift-left test strategyContinuous testingis an essential part of CI/CD. CI/CD pipelines can implement various levels of testing to reduce the likelihood of defects entering production. Testing can include: unit, functional, regression, load, and chaos.Continuous testing requires testing and breaking existing boundary conditions, and updating test cases if the boundaries have changed. Test cases should test distributed systems’ idempotency. Chaos testing benefits our incidence response mechanisms for distributed systems that have multiple integration points. By testing our auto scaling and failover mechanisms, chaos testing improves application performance and resiliency.AWS Fault Injection Simulator (AWS FIS)is a service for chaos testing. An experiment template contains actions, such as StopInstance and StartInstance, along with targets on which the test will be performed. In addition, you can mention stop conditions and check if they triggered the requiredAmazon CloudWatchalarms, as demonstrated in Figure 7.Figure 7. AWS Fault Injection Simulator architecture for chaos testingPattern 7: Providing operational visibilityIn production, operational visibility across multiple dimensions is necessary for distributed systems (Figure 8). To identify performance bottlenecks and failures, useAWS X-Rayand other open-source libraries for distributed tracing.Write application, infrastructure, and security logs to CloudWatch. When metrics breach alarm thresholds, integrate the corresponding alarms with Amazon Simple Notification Service or a third-party incident management system for notification.Monitoring services, such asAmazon GuardDuty, are used to analyze CloudTrail, virtual private cloud flow logs, DNS logs, andAmazon Elastic Kubernetes Serviceaudit logs to detect security issues. MonitorAWS Health Dashboardfor maintenance, end-of-life, and service-level events that could affect your workloads. Follow theAWS Trusted Advisorrecommendations to ensure your accounts follow best practices.Figure 8. Dimensions for operational visibility(click the image to enlarge)Figure 9 explores various application and infrastructure components integrating with AWS logging and monitoring components for increased problem detection and resolution, which can provide operational visibility.Figure 9. Tooling architecture to provide operational visibilityHaving an incident response management plan is an important mechanism for providing operational visibility. Successful execution of this requires educating the stakeholders on the AWS shared responsibility model, simulation of anticipated and unanticipated failures, documentation of the distributed system’s KPIs, and continuous iteration. Figure 10 demonstrates the features of a successful incidence response management plan.Figure 10. An incidence response management plan(click the image to enlarge)ConclusionIn Part 3, we discussed continuous improvement of our processes by testing and breaking them. In order to understand the baseline level metrics, service-level agreements, and boundary conditions of our system, we need to capture NFRs. Operational capabilities are required to capture deviations from baseline, which is where alerting, logging, and distributed tracing come in. Processes should be defined for automating frequent testing in CI/CD pipelines, detecting network issues, and deploying alternate infrastructure stacks in failover regions based on RTOs and RPOs. Automating failover steps depends on metrics and alarms, and by using chaos testing, we can simulate failover scenarios.Prepare for failure, and learn from it. Working to maintain resilience is an ongoing task.Want to learn more?Best practices for developing cloud applications with AWS CDKMulti-branch CodePipeline strategy with event-driven architecturePiyali KamraPiyali Kamra is a seasoned enterprise architect and a hands-on technologist who has over 21 years of experience building and executing large scale enterprise IT projects across geographies.She believes that building large scale enterprise systems is not an exact science but more like an art, where you can’t always choose the best technology that comes to one’s mind but rather tools and technologies must be carefully selected based on the team’s culture , strengths , weaknesses and risks , in tandem with having a futuristic vision as to how you want to shape your product a few years down the road.Aish GopalanAish Gopalan is a Senior Solutions Architect at AWS based out of the beautiful city of Atlanta, Georgia. Primarily an individual contributor, she has worn multiple hats ranging from Solution Architect, Application Architect, Cloud Transformation Delivery Lead and Developer in her 16yr software journey. Always chipper, she enjoys and follows Crossfit and believes in living for the moment.Isael PimentelIsael Pimentel is a Senior Technical Account Manager at AWS with over 15 years of experience in developing and managing complex infrastructures, and holds several certifications including AWS Solution Architect, AWS Network Specialty, AWS Security Speciality, MSCA, and CCNA.Aditi SharmaAditi Sharma is a Technical Account Manager in AWS and started her journey as a Cloud Support Engineer in the RDS Databases team where she worked on various DB engines. She has since then held several roles in AWS before moving to Technical Account Manager role. Prior to joining AWS, she was working as a Software Development Engineer in a Banking industry. Aditi hold Master’s in Management Information systems and Bachelor’s in Computer Science Engineering and also Certified Scrum Master, Product Owner and AWS Associate Solutions Architect.
In Part 1 and Part 2 of this series, we discussed how to build application layer and infrastructure layer resiliency. In Part 3, we explore how to develop resilient applications, and the need to test and break our operational processes and run books. Processes are needed to capture baseline metrics and boundary conditions. Detecting deviations […]
Piyali Kamra
Amazon CloudWatch Amazon EventBridge Amazon Route 53 Architecture AWS CloudFormation AWS CloudTrail AWS CodeCommit AWS Config AWS Lambda