Why 100% Test Pass Rates Are Bad

football goal

Are you shooting for the wrong goal?

When your test pass rate reaches 100% it’s usually cause for celebration. It shows your product has high quality. The tests you so carefully designed are passing, proving the product works as expected. It also shows your test cases have high quality; the results are repeatable and they’re successfully validating product functionality. This is the goal you’ve been striving for since you starting testing, right? If that’s the case, you’ve been shooting for the wrong goal.

A 100% pass rate implies your product works exactly as expected. The goal of testing, however, isn’t to prove the software works as expected. In fact, this objective can never even be achieved, because all complex software has defects. What sense does it make to have a goal that can’t be achieved? Your objective should be to write tests that expose these defects so an informed decision can be made on whether to fix them.

The other reason a 100% pass rate is the wrong goal is because when all your tests are consistently passing, they stop providing you new information. You see a perfect pass rate every morning, and pay no further attention to the tests. In other words, once you reach your “goal”, your tests stop helping you improve your product.

To be fair, from a management perspective a 100% pass rate is the right goal. It gives managers confidence in the quality of the product. But it can’t give them that confidence if both the managers and testers are striving for a perfect pass rate. In that case, no one is looking for bugs and the 100% pass rate means little.

Now assume you have a perfect pass rate, but the testers are striving to prove the software doesn’t work as expected. This shows management that despite the fact you’re looking for bugs, you still can’t find any. That will give management confidence in the software quality.

If you have a 100% pass rate, and still have time before Test Complete, don’t think of it as the end game. There are bugs in your product, and your tests aren’t finding them. Modify your tests. Write new ones. But design every test case with the goal of proving the software does NOT work as expected. The advice may sound obvious, but some testers don’t think this way; they’re much more interested in having their test passes go smoothly and on schedule.

Edsger Dijkstra wrote, “Testing can show the presence of errors, but not their absence.” When you change your mindset and write tests to prove the product doesn’t work, rather than to prove that it does work, you’ll be surprised how many more bugs you expose. This simple change will make you a better tester, and your products will be better for it.

%d bloggers like this: