Minimizing Dependencies in a Disaster Recovery Plan
AWS Architecture BlogMinimizing Dependencies in a Disaster Recovery Planby Randy DeFauw, Marcin Bednarz, and Brian Zambrano | on25 JAN 2022| inAmazon Route 53,Architecture,AWS Identity and Access Management (IAM),AWS Organizations|Permalink|ShareTheAvailability and Beyondwhitepaper discusses the concept of static stability for improving resilience. What does static stability mean with regard to a multi-Region disaster recovery (DR) plan? What if the very tools that we rely on for failover are themselves impacted by a DR event?In this post, you’ll learn how to reduce dependencies in your DR plan and manually control failover even if critical AWS services are disrupted. As a bonus, you’ll see how to use service control policies (SCPs) to help simulate a Regional outage, so that you can test failover scenarios more realistically.Failover plan dependencies and considerationsLet’s dig into the DR scenario in more detail. UsingAmazon Route 53for Regional failover routing is a common pattern for DR events. In the simplest case, we’ve deployed an application in a primary Region and a backup Region. We have a Route 53 DNS record set with records for both Regions, and all traffic goes to the primary Region. In an event that triggers our DR plan, we manually or automatically switch the DNS records to direct all traffic to the backup Region.Relying on an automated health check to control Regional failover can be tricky. A health check might not be perfectly reliable if a Region is experiencing some type of degradation. Often, we prefer to initiate our DR plan manually, which then initiates with automation.What are the dependencies that we’ve baked into this failover plan? First, Route 53, our DNS service, has to be available. It must continue to serve DNS queries, and we have to be able to change DNS records manually. Second, if we do not have a full set of resources already deployed in the backup Region, we must be able to deploy resources into it.Both dependencies might violate static stability, because we are relying on resources in our DR plan that might be affected by the outage we’re seeing. Ideally, we don’t want to depend on other services running so we can failover and continue to serve our own traffic. How do we reduce additional dependencies?Static stabilityLet’s look at our first dependency on Route 53 –control planesanddata planes. Briefly, a control plane is used to configure resources, and the data plane delivers services (seeUnderstanding Availability Needsfor a more complete definition.)The Route 53 data plane, which responds to DNS queries, is highly resilient across Regions. We can safely rely on it during the failure of any single Region. But let’s assume that for some reason we are not able to call on the Route 53 control plane.Amazon Route 53 Application Recovery Controller(Route 53 ARC) was built to handle this scenario. It provisions a Route 53 health check that we can manually control with a Route 53 ARC routing control, and is a data plane operation. The Route 53 ARC data plane is highly resilient, using a cluster of five Regional endpoints. You can revise the health check if three of the five Regions are available.Figure 1. Simple Regional failover scenario using Route 53 Application Recovery ControllerThe second dependency, being able to deploy resources into the second Region, is not a concern if we run a fully scaled-out set of resources. We must make sure that our deployment mechanism doesn’t rely only on the primary Region. Most AWS services have Regional control planes, so this isn’t an issue.TheAWS Identity and Access Management (IAM)data plane is highly available in each Region, so you can authorize the creation of new resources as long as you’ve already defined the roles. Note: If you use federated authentication through an identity provider, you should test that the IdP does not itself have a dependency on another Region.Testing your disaster recovery planOnce we’ve identified our dependencies, we need to decide how to simulate a disaster scenario. Two mechanisms you can use for this are network access control lists (NACLs) and SCPs. The first one enables us to restrict network traffic to our service endpoints. However, the second allows defining policies that specify the maximum permissions for the target accounts. It also allows us to simulate a Route 53 or IAM control plane outage by restricting access to the service.For the end-to-end DR simulation, we’ve published anAWS samples repositoryon GitHub that you can use to deploy. This evaluates Route 53 ARC capabilities if both Route 53 and IAM control planes aren’t accessible.By deploying test applications across us-east-1 and us-west-1 AWS Regions, we can simulate a real-world scenario that determines the business continuity impact, failover timing, and procedures required for successful failover with unavailable control planes.Figure 2. Simulating Regional failover using service control policiesBefore you conduct the test outlined in our scenario, we strongly recommend that you create a dedicated AWS testing environment with anAWS Organizationssetup. Make sure that you don’t attach SCPs to your organization’s root but instead create a dedicated organization unit (OU). You can use this pattern to test SCPs and ensure that you don’t inadvertently lock out users from key services.Chaos engineeringChaos engineeringis the discipline of experimenting on a system to build confidence in its capability to withstand turbulent production conditions. Chaos engineering and its principles are important tools when you plan for disaster recovery. Even a simple distributed system may be too complex to operate reliably. It can be hard or impossible to plan for every failure scenario in non-trivial distributed systems, because of the number of failure permutations. Chaos experiments test these unknowns by injecting failures (for example, shutting down EC2 instances) or transient anomalies (for example, unusually high network latency.)In the context of multi-Region DR, these techniques can help challenge assumptions and expose vulnerabilities. For example, what happens if a health check passes but the system itself is unhealthy, or vice versa? What will you do if your entire monitoring system is offline in your primary Region, or too slow to be useful? Are there control plane operations that you rely on that themselves depend on a single AWS Region’s health, such as Amazon Route 53? How does your workload respond when 25% of network packets are lost? Does your application set reasonable timeouts or does it hang indefinitely when it experiences large network latencies?Questions like these can feel overwhelming, so start with a few, then test and iterate. You might learn that your system can run acceptably in a degraded mode. Alternatively, you might find out that you need to be able to failover quickly. Regardless of the results, the exercise of performing chaos experiments and challenging assumptions is critical when developing a robust multi-Region DR plan.ConclusionIn this blog, you learned about reducing dependencies in your DR plan. We showed how you can use Amazon Route 53 Application Recovery Controller to reduce a dependency on the Route 53 control plane, and how to simulate a Regional failure using SCPs. As you evaluate your own DR plan, be sure to take advantage of chaos engineering practices. Formulate questions and test your static stability assumptions. And of course, you can incorporate these questions into a custom lens when you run a Well-Architected review using theAWS Well-Architected Tool.TAGS:Disaster recoveryRandy DeFauwI’m an electrical engineer by training who’s been working in technology for 23 years at companies ranging from start-ups to large defense firms. A fascination with distributed consensus systems led me into the Big Data space, where I discovered a passion for analytics and machine learning. I started using AWS in my Hadoop days, where I saw how easy it was to set up large complex infrastructure, and then realized that the cloud solved some of the challenges I saw with Hadoop. I picked up an MBA so I could learn how business leaders think and talk, and found that the ‘soft skill’ classes were some of the most interesting ones I took. Lately I’ve been dabbling with reinforcement learning as a way to tackle optimization problems, and re-reading Martin Kleppmann’s book on data intensive design.Marcin BednarzMarcin Bednarz is a Senior Solutions Architect at Amazon Web Services. He works with AWS customers to provide guidance and technical assistance, helping them improve the value of their solutions when using AWS.Brian ZambranoBrian Zambrano is a software engineer by training. He has a background in cloud-based SaaS architecture, design, and scalability. After more than 20 years of building products, Brian now helps customers do the same as a Solutions Architect. He has been working with AWS consistently since 2009. His areas of depths and passions are serverless architectures, microservices, and helping developers move quickly to solve day-to-day challenges.
The Availability and Beyond whitepaper discusses the concept of static stability for improving resilience. What does static stability mean with regard to a multi-Region disaster recovery (DR) plan? What if the very tools that we rely on for failover are themselves impacted by a DR event? In this post, you’ll learn how to reduce dependencies […]
Randy DeFauw
Amazon Route 53 Architecture AWS Identity and Access Management (IAM) AWS Organizations