State-Transition Testing

One of our goals at Expert Testers is to discuss practical topics that can help every tester do their job better. To this end, my last two articles have been about Decision Table Testing and Being an Effective Spec Reviewer. Admittedly, neither of these topics break new ground. That doesn’t mean, however, most testers have mastered these techniques. In fact, almost 50% of the respondents to our Decision Table poll said they’ve never used one.

Continuing the theme of discussing practical topics, let’s talk about State Transition Diagrams. State Transition Diagrams, or STDs as they’re affectionately called, are effective for documenting functionality and designing test cases. They should be in every testers bag of tricks, along with Decision Tables, Pair-Wise analysis, and acting annoyed at work to appear busy.

STDs show the state a system will move to, based on its current state and other inputs. These words, I understand, mean little until you’ve seen one in action, so let’s get to an example. Since I’m particularly busy (i.e., lazy) today, I’ll use a simple example I found on the web.

Below is a Hotel Reservation STD. Each rectangle, or node, represents the state of the reservation. Each arrow is a transition from one state to the next. The text above the line is the input–the event that caused the state to change. The text below the line is the output–the action the system performs in response to the event.

clip_image001

Pasted from <http://users.csc.calpoly.edu/~jdalbey/SWE/Design/STDexamples.html>

One benefit of State Transition Diagrams is that they describe the behavior of the system in a complete, yet easy-to-read and compact, way. Imagine describing this functionality in sentence format; it would take pages of text to describe it fully. STDs are much simpler to read and understand. For this reason, they can show paths that were missed by PM or Developer, or paths the Tester forgot to test.

I learned this when I was testing Microsoft Forefront Protection for Exchange Server, a product that protects email customers from malware and spam. The product logic for determining when a message would be scanned was complicated; it depended on the server role, several Forefront settings, and whether the message was previously scanned.

The feature spec described this logic in sentence format, and was nearly impossible to follow. I took it upon myself to create a State Transition Diagram to model the logic. I printed it out and stuck it on my office (i.e., cubicle) wall. Not a week went by without a Dev, Tester, or PM stopping by to figure out why their mail wasn’t being scanned as they expected.

If you read my article on Decision Tables (DTs), and I’m sure you didn’t, you may be wondering when to use an STD and when to use a DT. If you’re working on a system where the order of events matter, then use an STD; Decision Tables only work if the order of events doesn’t matter.

Another benefit of STDs is that we can use them to design our test cases. To test a system completely, you’d need to cover all possible paths in the STD. This is often either impractical or impossible.

In our simple example, there are only four paths from start of the STD to the end, but in larger systems there can be too many to cover in a reasonable amount of time. For these systems, you can use multiple STDs for sub-systems rather than trying to create a single STD for the entire system. This will make the STDs easier to read, but will not lower the total number of paths. It’s also common to find loops in an STD, resulting in an infinite number of possible paths.

When covering all paths is impractical, one alternative is to ensure each state (node) is covered by at least one test. This, however, would result in weak coverage. For our hotel booking system, we could test all seven states while leaving some transitions and events completely untested.

Often, the best strategy is to create tests that cover all transitions (the arrows) at least once. This guarantees you will test every state, event, action, and transition. It gives you good coverage in a reasonable amount of tests.

If you’re interested in learning more about STDs (it’s impossible to cover them fully in a short blog article) I highly recommend reading A Practitioner’s Guide to Software Test Design. It’s where I first learned about them.

The next time you’re having trouble describing a feature or designing your tests, give a State Transition Diagram or Decision Table a try. The DTs and STDs never felt so good!

 

Being an Effective Spec Reviewer

The first time testers can impact a feature is often during functional and design reviews. This is also when we make our first impression on our co-workers. If you want to make a great initial impact on both your product and peers, you have to be an effective reviewer.

In my seven years in testing, I’ve noticed that many testers don’t take advantage of this opportunity. Most of us fall into one of four categories:

  1. Testers who pay attention during reviews without proving feedback. This used to be me. I learned the feature, which is one of the goals of a review meeting. A more important goal, however, is to give feedback that exposes defects as early as possible.
  2. Testers who push-back (argue) at seemingly every minor point. Their goal is to increase their visibility and prove their worth as much as it is to improve the product. They learn the feature and can give valuable feedback. However, while they think they’re impressing their teammates, they’re actually frustrating them.
  3. Testers who attend reviews with their laptops open, not paying attention. If this is you, please stop; no one’s impressed with how busy you’re pretending to be.
  4. Testers who pay attention and learn the feature, while also providing constructive feedback. Not only do they understand and improve the feature, but they look good doing it. This can be you!

How do you do this, you ask? With this simple recipe that only took me four years to learn, I answer.

1. Read the Spec

Before attending any functional or design review, make sure you read the documentation. This is common sense, but most of us are so busy we don’t have time to read the specs. Instead, we use the review itself to learn the feature.

This was always a problem for me because although I learned the feature during the review, I didn’t have enough time to absorb the details and give valuable feedback during the meeting. It was only afterwards when I understood the changes we needed to make. By then it was too late–decisions had already been made and it was hard to change people’s minds. Or coding had begun, and accepting changes meant wasted development time.

A great idea Bruce Cronquist suggested is to block out the half hour before the review meeting to read the spec. Put this time on your calendar to make sure you don’t get interrupted.

2. Commit to Contributing

Come to every review with the goal of contributing at least one idea. Once I committed to this, I immediately made a bigger impact on both my product and peers. This strategy works for two reasons.

First, it forces you to pay closer attention than you normally might have. If you know you’ll be speaking during the meeting, you will pay closer attention.

Second, it forces you to speak up about ideas you might otherwise have kept to yourself. I used to keep quiet in reviews if I wasn’t 100% sure I was right. Even if I was almost positive, I would still investigate further after the meeting. The result was that someone else would often mention the idea first.

It  took four years for me to realize this is an effective tool.

3. Have an Agenda

It’s easy to say you’ll give a good idea during every review, but how can you make sure you’ll always have a good idea to give? For me, the answer was a simple checklist.

The first review checklist I made was to make sure features are testable. Not only are testers uniquely qualified to enforce testability, but if we don’t do it no one will. Bringing up testability concerns as early as possible will also make your job of testing the feature later-on much easier. My worksheet listed the key tenets of testability, had a checklist of items for each tenant, and room for notes.

At the time, I thought the concept of a review checklist was revolutionary. So much so, in fact, that I emailed Alan Page about it no less than five times. I’m now sure Alan must have thought I was some kind of stalker or mental patient. However, he was very encouraging and was kind enough to give the checklist a nice review on Toolbox–a Microsoft internal engineering website. If you work at Microsoft, you can download my testability checklist here.

I now know that not only are checklists the exact opposite of revolutionary, but there are plenty of other qualities to look for than just testability.

Test is the one discipline that knows about most (or all) of the product features.  It’s easy for us to find and identify inconsistencies between specs, such as when one PM says the product should do X, while another PM says it should do Y. It’s also our job to be a customer advocate. And we need to enforce software qualities such as performance, security, and usability. So I decided to expand my checklist.

My new checklist includes 50 attributes to look for in functional and design reviews. It’s in Excel format, so you can easily filter the items based on Review Type (Feature, Design, or Test) and Subtype (Testability, Usability, Performance, Security, etc.)

Review Checklist

Click this image to download the Review Checklist.

If there are any other items you would like added to the checklist, please list them in the comments section below. Enjoy!

Expert Testers 2012 Year in Review

When I logged into the blog today, I found that WordPress generously prepared a 2012 Annual Report for Expert Testers. If you like fireworks and useless stats, check it out.

Here’s an excerpt:

4,329 films were submitted to the 2012 Cannes Film Festival. This blog had 17,000 views in 2012. If each view were a film, this blog would power 4 Film Festivals

Click here to see the complete report. (Scroll down to see the useless stats.)

Decision Table Testing

English: The Black eyed peasDecision tables are an effective tool for describing complex functional requirements and designing test cases. Yet, although they’re far from a new concept, I’ve only seen a handful of functional specs and test plans that included one. The best way to illustrate their effectiveness is by regaling you with a tale of when one wasn’t used.

The year was 2009. Barack Obama was recently inaugurated as the 44th President of the United States, and the Black Eyed Peas hauntingly poetic Boom Boom Pow was topping the Billboard charts. I was the lead tester on a new security feature for our product.

Our software could be installed on six server types. Depending the server type and whether it was a domain controller, eleven variables would be set. The project PM attempted to describe these setting combinations in paragraph format. The result was a spec that was impossible to follow.

I tried designing my test cases from the document, but I wasn’t convinced all possible configurations were covered. You know how poor programming can have “code smell“? Well, this had “spec smell”. I decided to capture the logic in a decision table.

According to the always-reliable Wikipedia, decision tables have been around since ancient Babylon. However, my uncle, Johnny “Dumplings”, who is equally reliable, insists they’ve only been around since Thursday. I suspect the true answer lies somewhere in between.

If you’re not familiar with decision tables, let’s go through a simple example. Assume your local baseball squadron offers free tickets to kids and discounted tickets to senior citizens. One game a year, free hats are given to all fans.

To represent this logic in a decision table, create a spreadsheet and list all inputs and expected results down the left side. The inputs in this case are the fan’s age and sex. The expected results are ticket price and hat color.

Each row of a decision table should contain the different possible values for a single variable. Each column, then, represents a different combination of input values along with their expected results. In this example, the first column represents the expected results for boys under 5 — “Free Admission” and “Blue Hat”. The last column shows that female senior citizens get $10 tickets and a pink hat.

Rule 1 Rule 2 Rule 3 Rule 4 Rule 5 Rule 6
Inputs
Age < 5 Y Y
5 =< Age < 65 Y Y
Age >= 65 Y Y
Sex M F M F M F
Results
Free Admission Y Y
$10 Admission Y Y
$20 Admission Y Y
Blue Hat Giveaway Y Y Y
Pink Hat Giveaway Y Y Y

There are three major advantages to decision tables.

  1. Decision tables define expected results for all input combinations in an easy-to-read format. When included in your functional spec, decision tables help developers keep bugs out of the product from the beginning.
  2. Decision tables help us design our test cases. Every column in a decision table should be converted into at least one test case. The first column in this table defines a test for boys under 5. If an input can be a range of values, however, such as “5 =< Age < 65”, then we should create tests at the high and low ends of the range to validate our boundary conditions.
  3. Decision tables are effective for reporting test results. They can be used to clearly show management exactly what scenarios are working and not working so informed decisions can be made.

It’s important to note that decision tables only work if the order the conditions are evaluated in, and the order of the expected results, doesn’t matter. If order matters, use a state transition diagram instead. I’ll blabber about them in a future article.

Getting back to my story, I used the spec to fill in a decision table representing all possible server combinations and expected results.  The question marks in the table clearly showed that results weren’t defined for several installation configurations. I brought this to the Program Manager’s attention, and we will able to fill in the blanks to lock down the requirements.

Case 1 Case 2 Case 3 Case 4 Case 5 Case 6 Case 7 Case 8 Case 9 Case 10
Inputs
Server Type Type 1 Type 1 Type 2 Type 2 Type 3 Type 3 Type 4 Type 4 Type 5 Type 6
DC Y N Y N Y N Y N * *
Results
Setting1 Y Y Y Y Y ? N N N N
Setting2 Y ? Y Y Y ? N N N N
Setting3 Y Y Y Y Y ? N N N N
Setting4 Y Y Y Y Y ? N ? N N
Setting5 Y Y Y Y Y ? N N N N
Setting6 Y Y Y Y Y ? N N N N
Setting7 ? ? ? ? ? ? ? ? ? ?
Setting8 N N N N N ? Y Y N N
Setting9 ? N N N N ? Y N N N
Setting10 N N ? Y Y ? N N ? N
Setting11 N N Y Y Y ? N N N N

This easy-to-understand table took the place of 2 full pages of spaghetti text in the functional spec. As a result, the developers had a comprehensive set of requirements to work off of, and I had designed a thorough set of test cases to validate their work. Boom Boom Pow, indeed!

Happy Birthday to Us

Happy Birthday

It’s been one year since we launched the Expert Testers blog! I’d like to thank all of you for reading our articles, following us on Facebook and Twitter, submitting comments, and participating in our polls. In the year we’ve been online, our articles have been read over 15,000 times.  Here are a few more random facts:

  • Most popular search term sending visitors to the blog: “test case priority”
  • Most interesting search term sending visitors to the blog (tie):
    • “hundred test cases – only two testers – how to manage in two days?”
    • “how to write a mail to dev team that regression testing found new bugs?”
  • Total followers across all platforms (Twitter/Facebook/Wordpress): 206
  • Most-read article: Testers Caught Sleeping on the Job
  • Number of authors: 8
  • Articles: 27
  • Comments: 70
  • Most views by country (after the United States): India
  • Least views by country (tie): Barbados and Mauritius (1)

Thanks for reading!
Andrew

%d bloggers like this: