VP Engineering Playbook

A practical guide for new leaders in software development

Tag: testing

Leveraging Automated Testing

Testing your application is a necessary part of the software development process. Each application has a set of behaviors that it is expected to perform consistently. As developers make code changes, established behaviors must be verified before the application is updated in the production environment. This verification of prior behaviors has traditionally been referred to as regression testing. This is contrasted with acceptance testing, which is conducted on new features. In both cases, an individual will run a set of tests and match the results against expected outcomes. Any discrepancies are logged as a bug for developers to fix.

Regression testing in particular can be very resource intensive, when performed manually by your QA team. As an application matures, the number of regression tests will grow, sometimes resulting in hundreds (or thousands) of behaviors that need to be verified. It can become impractical to run these manually before each release. Usually, the team will adapt by delaying regression testing and large releases until the end of a sprint.

Of course, the downside to this delay is that it increases the time between code completion and feature release to users. Also, bugs are reported long (relatively) after the developer has made the code changes. In order to address the bug, the developer has to refresh themselves on the code changes made at that time.

A solution to this is to reduce the time it takes to conduct regression testing through automation. On a modern, UI-driven internet application, much of this manual testing can be replaced by simulated tests run by computers. Computerized tests can be parallelized and completed faster than if a human ran them.  In this post, we will explore a couple of mechanisms available to automate regression testing and how you can measure your testing effectiveness.

Unit Testing

The first step in reducing the overhead of regression testing starts with unit tests. These are the code level tests that developers create to exercise individual functions or classes within a software component. A unit test provides a written contract that the code will satisfy. It specifies discrete inputs and an expected output in the form of an assertion.

Unit tests can be executed by each developer in their development environment before checking code into a shared repository. This catches logic errors created by that developer’s own changes. To ensure that one developer’s changes are compatible with all the other developers’ subsequent changes, we utilize continuous integration. Continuous Integration (CI) is a practice that requires developers to integrate code into a shared repository. A CI server will monitor for changes to the shared repository and then kicks off a “build”.  The server will execute the code’s build process, deploy it into a test environment and then run the set of unit tests. The output for the unit test run is published to the development team.  Popular CI servers are Jenkins and Bamboo.

As it relates to reducing the overhead of regression testing, a comprehensive set of unit tests can go a long way. These can catch simple logic errors before they get to the QA team. Developers should write unit tests as they code. These should be checked in periodically and not be postponed until after release. Also, you should measure unit test coverage, to ensure that all logical code paths are covered. A tool like Clover will examine your code base and provide a report of test coverage. How much coverage to target is debatable. More is usually better, but 100% is not realistic. Depending on the maturity of your test program and development team, somewhere between 50% to 90% should work as a target. If your team is just starting out with unit testing, increasing coverage can be a continuous improvement exercise.

Also, it is important that unit test failures surfaced by the CI builds are addressed in a timely manner. Some development shops prevent any new commits until a broken build is fixed. This is one way to ensure that unit test failures are fixed quickly. However, it blocks work for all developers. On the other hand, if unit test failures are ignored, their count will quickly increase until the continuous integration process is no longer useful. Each team can establish their own standard for fixing unit test failures, based on their culture and maturity. I think a reasonable guideline is to fix unit test failures by end of day.

Functional Testing

Functional testing generally involves testing interactions between the application’s user interface and its back-end logic. Similar to unit testing, inputs are sent to the application and expected outputs are verified. Testing of user interfaces will span all devices which the application supports – web, mobile, desktop, etc.

Because the user interface device is the test medium, this type of testing is initially performed manually by QA engineers. For acceptance testing, manual testing makes sense, but for large regression test suites, it can be unwieldy. Fortunately, functional testing can be automated by using a testing tool that simulates interactions with the particular UI device, directly making inputs and capturing responses. Test frameworks exist for each device type. For the web, a test tool can be used that simulates actions in a browser.  The most popular open source tool is Selenium, providing support for most modern browsers.  For mobile devices, ideally you would use a test framework that supports “cross platform” automated testing.  In this case, the test framework exposes a single API for interactions with iOS, Android and mobile web. A popular open source tool for this is Appium (created at Zoosk, incidentally).

A role within your QA team, called the test automation engineer, is usually responsible for maintaining the automated functional tests. Requirements for a test automation engineer are more advanced than for a manual black box tester.  An automation engineer needs the ability to code, as they will write test scripts within the automation framework.  Also, it is a good idea for an automation engineer to possess some devops skills, as they will likely own the test environment. Their role will be to create new automated tests as functionality is added to your applications and to update tests as business logic is changed.

Automating functional tests will significantly reduce the overhead in each regression test pass. Since they interact through a user interface, a nontrivial time is required to run them. A set of a few hundred Selenium web tests, for example, can take a few hours to execute. Even so, it is feasible to run these several times a day, or at least nightly. The output should be distributed to the team for investigation. Usually, a QA engineer will check test failures manually first, and then file bugs for verified code issues.

Service Interface Testing

Stand-alone services do not have a user interface. These are typically fronted by a RESTful API that facilitates interaction with a back-end application.  Services usually encapsulate a set of related application functions, involving substantial business logic and interactions with data stores. These types of interfaces are easy to test through automation. Automated testing of a service involves calling each API endpoint individually with standard inputs. The test then checks the response of the API for an expected value. A test automation engineer will script these tests, using documentation describing the service interface. Some open source tools that provide the capability to automate service interface testing are SoapUI and PyRestTest.

By moving your application functionality to stand-alone services with open interfaces, you will be able to automate more of your regression testing. This will reduce the amount of testing that must be verified through a UI interaction. Automated regression tests against an API generally run faster than through a user interface. I highlighted the advantages of a service-based architecture in a prior post.

Measuring your Test Automation Program

Creating and maintaining your test automation suites will represent a major resource investment. Like any allocation of resources, it is important to track the cost and benefit associated with that investment. You should collect a set of metrics which represent the amount of effort put into the test automation program and its relative success. Your QA manager should own this data collection. They should summarize the data periodically and present it to the team. These metrics will generate insights and provide feedback for additional changes to the automation program as it evolves.

Here is a list of sample metrics that are useful to collect:

  • Time spent on automated test creation. Track the amount of time each test automation engineer spends creating new automated tests. This is usually required when functionality is added to the application. These numbers can be aggregated on a per sprint basis.
  • Time spent updating automated tests. Track the time spent updating existing automated tests. Updates to existing tests are necessary when the business logic for an existing feature changes.
  • Bugs identified by automation. When automated tests are run, review the output. If a test failure results in filing a bug for a developer to fix, record this event. Aggregate the number of bugs caught by automation on a per sprint basis. These bugs represent the primary benefit of test automation.
  • Bugs missed by automation. After code is pushed to production, issues will be reported by users. If the issue report results in logging a bug to a developer, then make note of the bug. This represents a test failure that automation missed. Examine how testing didn’t capture the bug.
  • Time spent on manual regression testing. Hopefully, as more testing is automated, this time will go down.

These metrics should be reviewed periodically. Ideally, you will see an increasing number of bugs caught by automated testing and the amount of manual testing decrease.

Expectations for Automation Test Coverage

Over time, your metrics will give you a strong sense for the effectiveness of your test automation program. Pay particular attention to the number and types of bugs that are and are not caught by your automated functional regression testing. In my experience, automation of functional testing will not catch every bug. Some set of manual regression tests for critical functionality in your application is still advisable. You can include these for major functions like user registration, search, product pages, shopping cart, etc. Your product managers should be able to help identify the areas of the application that would have major business impact if they didn’t work. For these, your QA team can craft a short list of functional tests that can be run manually before each major release. While often redundant to the automated tests, these provide a good balance to an evolving test automation program. Once you have more confidence in your automated testing, you can cut back on these manual tests.

Also, as you collect data on your test cycles, your QA manager should share that data with other engineering leads and product managers. This data can be summarized monthly or quarterly and presented at group meetings. Sharing this type of data will generate productive conversations about the state of testing and what improvements can be made. The data should help address questions about the investment being made into the automated test program and its benefit.

Structuring your Agile Sprint Schedule – Part 2

This is a continuation of a post on how to structure your agile sprint schedule.  In Part 1, we discussed approaches to sprint length and the planning process.  In this post, we will examine testing and release.


Developers should be writing unit tests for their code.  In test-driven development, these are written first. I don’t strongly advocate for this approach, but I do insist on unit tests being included in any reasonably sized code check-in.  Unit tests should not be left to the end of the sprint, or rolled into tech debt.  Also, you should have a continuous integration system in place that runs the full set of unit tests against the code base periodically (ideally on every check-in).  If a unit test fails, the issue should be corrected by the associated developer immediately.  I have seen continuous integration runs result in increasing numbers of unit test failures, to the point where they become pointless.  Ensure you enforce the discipline within the team to prevent this.

Your QA team should be involved throughout the sprint process.  Don’t just pull in a QA team member at the end of the sprint to conduct regression testing.  A representative from QA should offer input into the planning process and should work with developers to test components of sprint deliverables as they are built.  These  front-loaded tests are very valuable, as they can flag issues early in the sprint.  Otherwise, all the issues will accumulate to the final regression test.  As the sprint end is approaching, the team should try to taper down code commits representing new features and allow the QA representative to focus on end-to-end testing.  Bugs that surface from full integration testing can be addressed in the final days of the sprint.


I define the release process as the activities that occur between the final planned code commit and the point at which the code is stable in production.  With this definition, release includes the following activities:

  • Branch Cutting.  Depending on how your team manages its code base, you will likely create a branch off of your main code line for the sprint release.  Often called the “release candidate”, this separate branch allows your team to test and apply fixes to the release candidate in isolation from the main code line.  The main code line may continue to get changes for the next sprint, without blocking developers while final regression testing occurs.  Your release engineer (often a designated developer on a small team, or a dedicated role on a larger team) will coordinate and drive the branching activities.
  • Unit Test Run.  Following the branch cutting, you will want to run your full suite of unit tests on the release candidate in your continuous integration environment.  Any unit test failures should be addressed by the developers immediately.
  • Regression Testing.  Your QA team members will install the release candidate into a dedicated test environment.  They will run through their suite of regression tests.  Some of these may be automated (ideally), but they will likely need to conduct a good percentage of tests manually.  Particularly if your application has a rich UI and complex business logic, having a set of manual “sanity” tests is important.  As the QA engineer finds bugs, they should log them for developers to fix.
  • Launch Planning.  As the QA engineers and developers are wrapping up the regression test process, the release engineer and team leads should conduct release planning.  This involves determining the sequence of steps required to move the release code to production.  This includes a plan for any database schema changes, with details of how those will be applied. Also, a list of post-launch checks should be assembled.  These will likely include business metrics and application performance monitors.  Finally, a rollback plan should be determined, in the event that there is an issue with the release.
  • Release.  Once QA has signed off on the release candidate and all bug fixes have been committed, then the team is ready to start the actual code release.  At Zoosk, we would conduct a quick stand-up with all sprint participants to review the launch plan and ensure that post-release checks were delegated out.  This stand-up also included the product manager(s), so that they were aware of the imminent release.  After code was pushed live, release checks are conducted.  Once all release checks are cleared, the release engineer calls the release stable.  If a release check indicates an issue, the team can troubleshoot it quickly and determine if a rollback is required.

Other Considerations

  • Holidays.  If a holiday occurs during your sprint, just subtract that day from the number of days available in that sprint.  I don’t recommend that you extend the sprint by a day, as this would likely break the integrity of the Monday-Friday sprint schedule.
  • Progress Checks.  While your team(s) will report on sprint status at daily stand-ups, for larger teams it makes sense to have a formal check mid-sprint on progress.  For a multi-week sprint, this usually works best on the Friday in between.  Participants should be the product manager and team leads.  At this progress check, the participants should review completed versus planned work items and gauge the likelihood of completing all items by the end of the sprint.  If it appears that all work items will not be completed in the second half of the sprint, then the product manager and team leads should determine a course of action to take. This can involve reducing scope of a work item or scheduling it into a future sprint. Planning for developer “overtime” as a means to catch up should be avoided, but does represent an option in some cases.