The key to unlock the tests is in the combination

In the last blog, Andrew Schiano discussed Equivalence Class (EQ) and Boundary Value Analysis (BVA) testing methodologies.  This blog will talk about how to extend those two ideas even further with Combinatorial Testing.

Combinatorial Testing is a form of model-based testing.  It chooses pairs or sets of inputs, out of all of the possible inputs, that will give you the best coverage with the least cost.  Fewer test cases while still finding bugs and giving high code coverage is a dream of us testers.  It is best applied when:

  • Parameters are directly interdependent
  • Parameters are semi-coupled
  • Parameter input is unordered

Let’s look at an example UI.  You have to test a character formatting dialog.  It allows you to pick between four fonts, two font styles, and three font effects.  A chart of the values looks like this:

Field Values
Font Arial, Calibri,Helvetica, BrushScript
Style Bold, Italic
Effects Strikethrough, Word Underline, Continuous Underline

For any selection of text, you can have only one Font, zero to two Styles, and zero to two effects.  Notice how for effects, you could select one pair, strikethrough and continuous underline, but you should not be able to pick another pair, word underline and continuous underline.  You could set up a matrix of tests and check each combination.  That’s old school.  You could reduce the number of cases by applying EQ.  That would be smarter. Your best bet, though, is to apply Combinatorial testing.

Combinatorial testing, also sometimes called Pairwise Testing, is just modeling the System Under Test (SUT).  You start taking unique interesting combinations of inputs, using them to test the system, and then feeding that information back into the model to help direct the next set of tests.  There is mathematics involved that take into account the coupling (interactions) of the inputs, which we won’t cover here.  Thankfully there are tools that help us choose which test cases to run.   I’ll cover the Pairwise Independent Combinatorial Testing (PICT) tool, available free from Microsoft.  Since it is a tool, it is only as good as the input you give it.

The steps to using PICT or any other combinatorial testing tool are:

  1. Analysis and feature decomposition
  2. Model parameter variables
  3. Input the parameters into the tool
  4. Run the tool
  5. Re-validate the output
  6. Modify the model
  7. Repeat steps 4-6

In our example above, the decomposition would look like this:

  • Font: Arial, Calibri, Helvetica, BrushScript
  • Bold: check, uncheck
  • Italic: check, uncheck
  • Strikethrough: check, uncheck
  • Underline: check, uncheck, word, continuous

You feed this data into the tool and it will output the number of tests you specify.  You need to validate the case before running them.  For example, the BrushScript font only allows Italic and Bold/Italic.  If the tool output the test case:

  • Font: BrushScript
  • Bold: check
  • Italic: uncheck
  • Strikethrough: check
  • Underline:  word

Being the awesome tester that you are, you would notice this is not valid.  Thankfully the PICT tool allows you to constrain invalid input combinations.  It also allows you to alias equivalent values.  So, you modify the model, not the outputted values.  In this case, you would add two line so the input file would now look something like this:

Font: Arial, Calibri, Helvetica, BrushScript

Bold: check, uncheck

Italic: check, uncheck

Strikethrough: check, uncheck

Underline: check, uncheck, word, continuous

IF [Font] = “BrushScript” AND [Italic] = “uncheck” THEN [Bold] <> “check”;

IF [Bold] = “uncheck” AND [Italic] = “uncheck” THEN NOT [Font] = “BrushScript”;

The PICT tool also allows you to weight values that are more common (who really uses BrushScript anymore?), and seed data about common historical failures.

Font: Arial(8) , Calibri(10), Helvetica(5), BrushScript (1)

Does this really work?  Here is an example from a previous Microsoft project:

Command line program with six optional arguments:

Total Number of   Blocks = 483
Default test suite
Exhaustive   coverage
Pairwise coverage
Number of test   cases
Blocks covered
Cove coverage
Functions not   covered

Now, pair this with automation so you have data-driven automated testing, and you’re REALLY off and running as a twenty-first century tester!

A few words of caution.  While this gives you the minimum set of tests, you should also test:

  1. Default combinations
  2. Boundary conditions
  3. Values known to have caused bugs in the past
  4. Any mission-critical combinations.

Lastly, don’t forget Beizer’s Pesticide Paradox.  Keep producing new test cases.  If you only run the tool once, and continually run those same cases, you’re going to miss bugs.

State-Transition Testing

One of our goals at Expert Testers is to discuss practical topics that can help every tester do their job better. To this end, my last two articles have been about Decision Table Testing and Being an Effective Spec Reviewer. Admittedly, neither of these topics break new ground. That doesn’t mean, however, most testers have mastered these techniques. In fact, almost 50% of the respondents to our Decision Table poll said they’ve never used one.

Continuing the theme of discussing practical topics, let’s talk about State Transition Diagrams. State Transition Diagrams, or STDs as they’re affectionately called, are effective for documenting functionality and designing test cases. They should be in every testers bag of tricks, along with Decision Tables, Pair-Wise analysis, and acting annoyed at work to appear busy.

STDs show the state a system will move to, based on its current state and other inputs. These words, I understand, mean little until you’ve seen one in action, so let’s get to an example. Since I’m particularly busy (i.e., lazy) today, I’ll use a simple example I found on the web.

Below is a Hotel Reservation STD. Each rectangle, or node, represents the state of the reservation. Each arrow is a transition from one state to the next. The text above the line is the input–the event that caused the state to change. The text below the line is the output–the action the system performs in response to the event.


Pasted from <>

One benefit of State Transition Diagrams is that they describe the behavior of the system in a complete, yet easy-to-read and compact, way. Imagine describing this functionality in sentence format; it would take pages of text to describe it fully. STDs are much simpler to read and understand. For this reason, they can show paths that were missed by PM or Developer, or paths the Tester forgot to test.

I learned this when I was testing Microsoft Forefront Protection for Exchange Server, a product that protects email customers from malware and spam. The product logic for determining when a message would be scanned was complicated; it depended on the server role, several Forefront settings, and whether the message was previously scanned.

The feature spec described this logic in sentence format, and was nearly impossible to follow. I took it upon myself to create a State Transition Diagram to model the logic. I printed it out and stuck it on my office (i.e., cubicle) wall. Not a week went by without a Dev, Tester, or PM stopping by to figure out why their mail wasn’t being scanned as they expected.

If you read my article on Decision Tables (DTs), and I’m sure you didn’t, you may be wondering when to use an STD and when to use a DT. If you’re working on a system where the order of events matter, then use an STD; Decision Tables only work if the order of events doesn’t matter.

Another benefit of STDs is that we can use them to design our test cases. To test a system completely, you’d need to cover all possible paths in the STD. This is often either impractical or impossible.

In our simple example, there are only four paths from start of the STD to the end, but in larger systems there can be too many to cover in a reasonable amount of time. For these systems, you can use multiple STDs for sub-systems rather than trying to create a single STD for the entire system. This will make the STDs easier to read, but will not lower the total number of paths. It’s also common to find loops in an STD, resulting in an infinite number of possible paths.

When covering all paths is impractical, one alternative is to ensure each state (node) is covered by at least one test. This, however, would result in weak coverage. For our hotel booking system, we could test all seven states while leaving some transitions and events completely untested.

Often, the best strategy is to create tests that cover all transitions (the arrows) at least once. This guarantees you will test every state, event, action, and transition. It gives you good coverage in a reasonable amount of tests.

If you’re interested in learning more about STDs (it’s impossible to cover them fully in a short blog article) I highly recommend reading A Practitioner’s Guide to Software Test Design. It’s where I first learned about them.

The next time you’re having trouble describing a feature or designing your tests, give a State Transition Diagram or Decision Table a try. The DTs and STDs never felt so good!


Expert Testers 2012 Year in Review

When I logged into the blog today, I found that WordPress generously prepared a 2012 Annual Report for Expert Testers. If you like fireworks and useless stats, check it out.

Here’s an excerpt:

4,329 films were submitted to the 2012 Cannes Film Festival. This blog had 17,000 views in 2012. If each view were a film, this blog would power 4 Film Festivals

Click here to see the complete report. (Scroll down to see the useless stats.)

Happy Birthday to Us

Happy Birthday

It’s been one year since we launched the Expert Testers blog! I’d like to thank all of you for reading our articles, following us on Facebook and Twitter, submitting comments, and participating in our polls. In the year we’ve been online, our articles have been read over 15,000 times.  Here are a few more random facts:

  • Most popular search term sending visitors to the blog: “test case priority”
  • Most interesting search term sending visitors to the blog (tie):
    • “hundred test cases – only two testers – how to manage in two days?”
    • “how to write a mail to dev team that regression testing found new bugs?”
  • Total followers across all platforms (Twitter/Facebook/Wordpress): 206
  • Most-read article: Testers Caught Sleeping on the Job
  • Number of authors: 8
  • Articles: 27
  • Comments: 70
  • Most views by country (after the United States): India
  • Least views by country (tie): Barbados and Mauritius (1)

Thanks for reading!

Writing Good Test Plans and Writing Good Tests

This post is based on one of my test projects, “Testing SQL Express Setup integration with Visual Studio 2012”.  Unlike typical testing strategies, we decided to do manual testing instead of automated testing.  We spent a lot of effort thinking about the customer scenarios and giving them different priorities.  Depending on the scenario and priority, we used pair wise combination strategies to select a small set of tests from the huge testing matrix.  Then our vendor team executed the selected test cases manually and signed-off on them.  We found a couple of UI bugs, a couple of useability issues, and functional bugs as well. In the end, we delivered a better setup experience for Visual Studio users.  The same test approach was used for Visual Studio 2012 testing as well.


  •  SSE means SQL Server Express
  •  MSI 4.5 means Windows Installer 4.5
%d bloggers like this: