Perform Chaos Testing on your Amazon Aurora Cluster
AWS Architecture BlogPerform Chaos Testing on your Amazon Aurora Clusterby Anthony Pasquariello, Rajeev Sakhuja, and Siddharth Pasumarthy | on19 JUL 2021| inAmazon Aurora,Architecture|Permalink|Share“Everything fails all the time”Werner Vogels, AWS CTOIn 2010, Netflix introduced a tool called“Chaos Monkey”, that was used for introducing faults in a production environment. Chaos Monkey led to the birth of Chaos engineering where teams test their live applications by purposefully injecting faults. Observations are then used to take corrective action and increase resiliency of applications.In this blog, you will learn about the fault injection capabilities available in Amazon Aurora for simulating various database faults.Chaos ExperimentsChaos experiments consist of:Understanding the application baseline:The application’s steady-state behaviorDesigning an experiment:Ask “What can go wrong?” to identify failure scenariosRun the experiment:Introduce faults in the application environmentObserve and correct:Redesign apps or infrastructure for fault toleranceChaos experiments require fault simulation across distributed components of the application.Amazon Auroraprovides a set of fault simulation capabilities that may be used by teams to exercise chaos experiments against their applications.Amazon Aurora fault injectionAmazon Aurora is a fully managed database service that is compatible with MySQL and PostgreSQL. Aurora is highly fault tolerant due to its six-way replicated storage architecture. In order to test the resiliency of an application built with Aurora, developers can leverage the native fault injection features to design chaos experiments. The outcome of the experiments gives a better understanding of the blast radius, depth of monitoring required, and the need to evaluate event response playbooks.In this section, we will describe the various fault injection scenarios that you can use for designing your own experiments. We’ll show you how to conduct the experiment and use the results. This will make your application more resilient and prepared for an actual event.Note that availability of the fault injection feature is dependent on the version ofMySQLandPostgreSQL.Figure 1. Fault injection overview1. Testing an instance crashAn Aurora cluster can have one primary and up to 15 read replicas. If the primary instance fails, one of the replicas becomes the primary. Applications must be designed to recover from these instance failures as soon as possible to have minimal impact on the end-user experience.Theinstance crash fault injectionsimulates failure of the instance/dispatcher/node in the Aurora database cluster. Fault injection may be carried out on the primary or replicas by running the API against the target instance.Example:Aurora PostgreSQL forinstance crash simulationThe query following will simulate a database instance crash:SELECT aurora_inject_crash ('instance' );Since this is a simulation, it does not lead to a failover to the replica. As an alternative to using this API, you can carry out an actual failover by using theAWS Management ConsoleorAWS CLI.The team should observe the change in the application’s behavior to understand the impact of the instance failure. Take corrective actions to reduce the impact of such failures on the application.A long recovery time on the application would require the team to reduce theDomain Name Service (DNS) time-to-live (TTL)for the DB connections. As a general best practice, the Aurora Database cluster should have at least one replica.2. Testing the replica failureAurora manages asynchronous replication between cluster nodes within a cluster. The typicalreplication lagis under 100 milliseconds. Network slowness or issues on the nodes may lead to an increase in replication lag between writer and replica nodes.Thereplica failure fault injectionallows you to simulate replication failure across one or more replicas. Note that this type of fault injection applies only to a DB cluster that has at least one read replica.Replica failure manifests itself as stale data read by the application that is connecting to the replicas. The specific functional impact on the application depends on the sensitivity to the freshness of data. Note that this fault injection mechanism does not apply to the native replication supported mechanisms in PostgreSQL and MySQL databases.Example:Aurora PostgreSQL forreplica failureThe statement following will simulate 100% failure of replica named ‘my-replica’ for 20 seconds.SELECT aurora_inject_replica_failure(100, 20, ‘my-replica’)The team must observe the behavior of the application from the data sensitivity perspective. If the observed lag is unacceptable, the team must evaluatecorrective actionssuch as vertical scaling of database instances and query optimization. As a best practice, the team should monitor thereplication lagand take proactive actions to address it.3. Testing the disk failureAurora’s storage volume consists of six copies of data across three Availability Zones (refer the diagram preceding). Aurora has an inherent ability to repair itself for failures in the storage components. This high reliability is achieved by way of aquorum model.Readsrequire only 3/6 nodes andwritesrequire 4/6 nodes to be available. However, there may still be transient impact on application depending on how widespread the issue.Thedisk failure injection capabilityallows you to simulate failures of storage nodes and partial failure of disks. The severity of failure can be set as a percentage value. The simulation continues only for the specified amount of time. There is no impact on the actual data on the storage nodes and the disk.Example:Aurora PostgreSQL fordisk failure simulationYou may get the number of disks (for index) on your cluster using the query:SELECT disks FROM aurora_show_volume_status()The query following will simulate 75% failure on disk with index 15. The simulation will end in 20 seconds.SELECT aurora_inject_disk_failure(75, 15, true, 20)Applications may experience temporary failures due to this fault injection and should be able to gracefully recover from it. If the recovery time is higher than a threshold, or the application has a complete failure, the team can redesign their application.4. Disk congestion faultDisk congestion usually happens because of heavy I/O traffic against the storage devices. The impact may range from degraded application performance, to complete application failures.Aurora provides the capability tosimulate disk congestionwithout synthetic SQL load against the database. With this fault injection mechanism, you can gain a better understanding of the performance characteristics of the application under heavy I/O spikes.Example:Aurora PostgreSQL fordisk congestion simulationYou may get the number of disks (for index) on your cluster using the query:SELECT disks FROM aurora_show_volume_status()The query following will simulate a 100% disk failure for 20 seconds. The failure will be simulated on disk with index 15. Simulated delay will be between 30 and 40 milliseconds.SELECT aurora_inject_disk_congestion(100, 15, true, 20, 30, 40)If the observed behavior is unacceptable, then the team must carefully consider the load characteristics of their application. Depending on the observations, corrective action may include query optimization, indexing, vertical scaling of the database instances, and adding more replicas.ConclusionA chaos experiment involves injecting a fault in a production environment and then observing the application behavior. The outcome of the experiment helps the team identify application weaknesses and evaluate event response processes.Amazon Auroranatively provides fault-injection capabilities that can be used by teams to conduct chaos experiments for database failure scenarios. Aurora can be used for simulating instance failure, replication failure, disk failures, and disk congestion. Try out these capabilities in Aurora to make your applications more robust and resilient from database failures.TAGS:fault injectionAnthony PasquarielloAnthony is a Senior Solutions Architect at AWS based in New York City. He specializes in modernization and security for our advanced enterprise customers. Anthony enjoys writing and speaking about all things cloud. He’s pursuing an MBA, and received his MS and BS in Electrical & Computer Engineering.Rajeev SakhujaRajeev is a Solution Architect based out of New York City. He enjoys partnering with customers to solve complex business problems using AWS services. In his free time, he likes to hike, and create video courses on emerging technologies.Siddharth PasumarthySiddharth is a Solutions Architect based out of New York City. He works with enterprise retail customers in the fashion and apparel industry, to help them migrate to cloud and adopt cutting edge technologies. He has a B.S. in Architecture from the Indian Institute of Technology and an M.S. in Information systems from Kelley School of Business. In addition to keeping up-to-date with technology, he is passionate about the arts, and creates still life acrylic paintings in his free time.
“Everything fails all the time” Werner Vogels, AWS CTO In 2010, Netflix introduced a tool called “Chaos Monkey”, that was used for introducing faults in a production environment. Chaos Monkey led to the birth of Chaos engineering where teams test their live applications by purposefully injecting faults. Observations are then used to take corrective action and […]
Anthony Pasquariello
Amazon Aurora Architecture