Seven Tips for Developing a DevOps Testing Strategy

DevOps might feel like the topic du jour in the world of software development, but it’s not just a passing fad.

Forrester declared it the year of enterprise DevOps for the second year in a row in 2018, noting that DevOps has reached Escape Velocity, with 50 percent adoption in 2017.

For testers, this shift can be worrisome. After all, DevOps represents a significant change to when and even how testing occurs by calling for testing to occur more often, frequently at the hands of developers and in a more automated fashion.

Despite these changes, testers very much have a place in a DevOps environment, it simply requires some adaptation — and a sound DevOps testing strategy. With that in mind, we sat down with Adam Satterfield, a QA leader at Anthem to put together the following tips for testing in a DevOps environment.

1) Embrace a Larger Role as a Quality Professional

Historically, testers have focused solely on the testing aspect of bringing a product to market — getting code, testing that code to validate that it works and pushing the validated code to developers. But DevOps changes that. With continuous integration, continuous delivery and the overall goal of increased efficiency, DevOps requires testers to take responsibility for quality across the entire development lifecycle, not just during the traditional testing phase.

Arguably the biggest change that results from this increased responsibility is the need to pay closer attention to what happens to software in production. In a typical waterfall or even agile scenario, production is very much a black box for testers. Once code gets sent to production, another team looks at the error rates and checks calls from customers about issues. That team then logs those issues as tickets for testers. But this system creates a long feedback cycle that simply doesn’t work in a fast-paced DevOps environment.

Meanwhile, when you adopt DevOps, one of the first things you often do is measure your cycle time and look for ways to reduce the build and test times. Given the need to resolve bugs faster, it’s essential to include the feedback cycle in that measurement as well. And the best way to shorten the feedback cycle is to involve testers in production. Specifically, instead of having your testers hand off code to production and walk away, ask them to start paying attention to what happens in production by looking for errors and working to fix them before they become issues reported by users. This approach not only shortens the feedback cycle, but it also reduces defect rates and minimizes errors found by users — an all-around win.

2) Don’t Overlook the Importance of Soft Skills

One of the biggest challenges that testers tend to face when adopting DevOps centers around soft skills. Because DevOps requires testers to be involved in more than just one siloed testing phase in the software development lifecycle, it means they must flex communication and collaboration skills that they haven’t necessarily had to use in the past.

For example, in a DevOps world, testers must be able to attend a planning session and communicate effectively with developers about what will get tested, how it will get tested, who will do the testing and when the testing will occur. This need to communicate across teams more frequently can take many testers out of their comfort zones, but it’s a necessary leap in order to make a smooth transition to DevOps. That’s because if you don’t break down silos between testers and developers by increasing communication and collaboration, you’ll never achieve the efficiency you want, regardless of how well everything runs from a technical standpoint.

3) Invest in Basic Coding Skills

Along with improving soft skills, testers also need to invest in basic coding skills in order to embrace their outsized role in the entire software development lifecycle. In DevOps, testers are not only responsible for the quality of the product, they’re also responsible for the quality of the process to bring that product to market. That responsibility means testers must act as gatekeepers for moving software from one environment to another.

As part of this effort, testers must be able to check build logs, ensure automated tests run properly and understand how the software works on a much deeper level than they’ve had to previously. Additionally, testers must get proactively involved in the delivery process to help move software through the development lifecycle faster and produce a higher quality end product. Fulfilling those imperatives requires basic coding skills.

Testing leaders can support their teams when it comes to building these skills in a variety of ways, for example by hosting lunch and learns with developers, leading a boot camp or even asking testers to automate one test per sprint that they would normally run manually.

4) Report QA’s Strategic Value, Not Test Results

Traditionally, testers have reported the results of their tests through metrics like tests passed vs. tests failed, number of defects found and requirements covered. While these test results are a great way to gauge the team’s efficiency and a product’s pre-release readiness, they don’t communicate the broader value of QA to the organization in the way that needs to happen in a DevOps environment where quality is everyone’s responsibility.

As a result, testers need to shift from reporting on these types of test results to reporting on a more strategic level that conveys how testing is progressing. For example, try reporting on how automation efforts contribute to bugs found in testing, the team’s progress toward automating more tests, how testing efficiency improves by release, how development and test teams are collaborating to reduce the turnaround time to fix issues and confirm fixes and how the testing team is increasing coverage (and thereby reducing risk).

5) Focus on Test Optimization, Not Test Automation

Test automation is absolutely necessary in a fast-moving DevOps environment, but it’s important to think about the bigger picture too. Test automation alone will not solve all of your manual process problems. Instead of focusing solely on how to achieve your desired level of test automation, try thinking about how to optimize your whole testing strategy.

Think of it this way: If you buy a Ferrari to go fast but end up taking only backroads instead of the highway, you won’t get to your destination any sooner. Focusing solely on test automation without taking steps to optimize your DevOps testing strategy creates a similar situation because you end up using an efficiency-driven tool in the least efficient way possible.

In contrast, taking on test optimization efforts should help increase efficiency by finding the right mix of unit, integration and functional automation alongside manual and exploratory testing. To find this mix, testers must identify and report on areas of risk and take the time to review test plans, including test creation, test execution and amount of testing.

6) Shift Left for Increased Automation…

Everyone knows that the longer you put off a problem, the worse it’s going to get in the long run. And in the continuous delivery-driven DevOps world, letting a problem fester means letting the release bottleneck grow. This situation happens most often when you run too many tests manually at the end of a cycle instead of automating them.

To remedy this situation, start incorporating frameworks that mandate automated tests get completed before moving software into production. The best way to do so is to layer in a combination of Test Driven Development (TDD) and Behavior Driven Development (BDD) in order to ensure testability, improve collaboration and drive greater efficiency.

7) …But Don’t Forget About Testing Right

While shifting left to automate more testing is certainly important in DevOps, a shift right needs to happen as well. Essentially, this shift right is all about moving some testing efforts into production.

Now, we’ve always said testing in production isn’t ideal. After all, finding a bug in production is extremely costly. However, testing in production has become important since the frequent releases that come with operating in a DevOps environment can cause bugs to make it through the pre-production phases of the software development lifecycle.

As a result, testers should not only be prepared to test earlier in the software development lifecycle, but they should also make plans to test in production. One way to accomplish this shift right is to implement monitoring tools that can detect where errors occur and how users are actually using the software. This setup allows end-users to inform the business about new test scenarios that require automation. From there, testers can add those tests to their library to make sure they get worked into the continuous pipeline.