Software testing from the ground up

Where do you start when asked to test software? In this tip, SSQ contributor Matt Heusser talks about the tutorial he is delivering at STPCon with Pete Walen about software testing from the ground up. Learn about quick attacks, domain analysis and knowing how to get the best bang for your testing buck.

What would happen if, for just a moment, we forgot about our methodology documents, our required process, and all...

the other accidents of how we test software, and we started over? That's the question I wanted to answer with a Monday Tutorial at the Software Test Professionals Conference in Dallas, along with my speaking partner, Pete Walen. Today, we'd like to tell you about it, along with a few exercises you can use with your team back at the office; no travel or downtime required.

Getting started

We start out in classroom with tables, chairs, a projector and laptops, much like you would expect for a tutorial, except that the tables have been moved. The order and placement don't make sense. When the class comes in, Pete and I explain that the first exercise is to re-arrange the tables to meet the requirements. Here are the requirements:

1. Organize yourselves into small groups of three to five people each. You will want to sit with your group, and be able to see each other.

2. You will need to be able to see the projector and speakers in the center of the room.

3. Obey all appropriate fire codes and regulations.

When we've done this exercise in the past, people invariably form a U shape around the room, which can accomplish requirements one and two. But what about requirement three?

We've left requirement number three ambiguous by design. Without knowing what the "right answer" is, teams tend to simply ignore it and hope for the best.

It turns out that is the first lesson of the day: You've got to know what correct behavior is in order to test. We call this an oracle.

It's time to get testing!

The class, which we affectionately called "Software Testing: From Soup To Nuts (A Five Course Testing Meal)," is designed to air-drop software testers into a high-pressure, high-competition business software environment, then go through a series of real-world simulations.

We start with a real problem: Management comes to the new testing organization with some software of dubious quality. "Can you test it?"

Notice there are likely no requirements. As testers, we were not involved up front, and we do not have "executive buy-in" for any sort of process improvement program. Instead, we are asked to test some software, right now. To run this exercise, we'll start with quick attacks, a test of two websites that you can perform from your desk. (They also make a handy brown bag session.)

Once we complete the quick attacks exercise, the class has some skills to attack a graphical user interface without any background in the software -- but those attacks are very shallow. All we are doing here is overwhelming the software with common failure modes; we don't really know how to assess the business rules to see if they are correct ... at least not yet. We cover that next, in domain analysis.

Domain analysis

Consider an auto-insurance company. You've been asked to test the eQuoting interface; it accepts a birth date, some other data, and returns a price. We'll start with the tests to 'just' calculate base price. In the table below, age is the first row and base price, the second.

Take a hard look at the table; what tests should we run to determine if the software is working correctly?

Take a moment, then come back.

... time passes ...

You probably noticed that the age of twenty is confused. That's on purpose; it's a defect in the requirements. Likewise, the behavior for people who want insurance over the age of ninety is undefined. For our purposes, let's say eQuoting does not work for 90+ year olds; these require an in-person interview. So how many business-valid tests should we run? We are not talking about quick attacks here, but valid entries designed to tell us something about the software.

We could run ninety tests, one for each age. Or we could keep going past ninety-one, to make sure nonagenarians and above do not get rated. In a high-pressure environment, though, it is unlikely that we will have time to run all those.

We could 'just' test one age of each major category - perhaps ages right in the middle, like 14, 18, 25, 45, and 55. That reduces our number of test ideas from 90 to seven. We call this process of breaking down by major categories: creating equivalence classes - the idea being, for this application, the ages of 45 and 46 are pretty much equivalent.

Except they aren't. Experienced testers know that classic programming mistakes include things like "off-by-one", or, perhaps, writing "less than" when the programmer actually means "less than or equal to." So in addition to testing one for each equivalence class, we might also want to test the boundaries - 15, 16, 20, 21, and so on. Once we correct the table (it's actually 21-34), that gives us perhaps 20 or 21 tests to run -- which is still a lot less than 90.

But wait ... there's more!

Recall that the eQuote software will take birth date as an input -- not year. That means it is possible that the table lookup works, but the software has some defect where it gets the calculation of age wrong. We might need to test several different birthdays, including someone born on February 29th, to see if the software is calculating age correctly.

After we explain domain analysis, we'll provide the class with a list of parking rates for the Grand Rapids International Airport, and ask them to write down ideas to test the software, using the techniques above. We'll share and discuss the ideas, then use them to test the software, trying to answer the question "Is this software fit for use? Will it tell us what we need to know?"

The great game of testing

By the end of the day, I hope we have explained what I call the "great game of testing." That is to say, these five things:

1) What do I know about the software?

2) What questions can I ask the software ('testing') to find out more about it?

3) Of those, what are the most valuable questions -- and the quickest to run?

4) Once I've answered those questions, what do those answers tell me?

Am I done? Blocked? Do I have a problem to report?

5) And what do I do next?

In software testing, we never have enough time and attention to answer all our questions. Instead, we have to try to provide responsible answers, with integrity, under conditions of pressure and uncertainty.

It turns out that software testing shares a lot with our tutorial. We couldn't possibly cram all of testing into one day; it is unlikely we could even fully satisfy all the attendees. Planning the tutorial in advance without the attendees would be a recipe for mediocrity at best, disaster at worst.

So we decided to make a list of things we could do, for about an hour each. We might talk about repeatability in test execution, developing a test strategy, automating parts of the test process, dealing with the combinatorial explosion problem, exploratory testing, and much more. We needed a strategy to not only limit the topics, but to make sure they were the right topics for the audience.

After the talks with broad applicability in the morning, we would hold a vote in the afternoon; each person gets three votes. Once we tally the vote, we sort, putting the most-voted for subjects first. Then we present each item to the group, working through the list, until we run out of time.

That's the final lesson of the workshop: If you don't have enough time, you've got to find some way to reflect the work back to your customer, asking them what to do first.

In the end, you do the best you can with the time you have; it is the great game of testing.

We wouldn't have it any other way.

This was first published in October 2011

Dig Deeper on Software Test Design and Planning






Forgot Password?


Your password has been sent to: