angryweasel
Andrew Schiano
brucecrms
herrickspencer
kristianjaeger
Prakash
qingsongyao
ramonarjona
Rob T
Are you shooting for the wrong goal?
When your test pass rate reaches 100% it’s usually cause for celebration. It shows your product has high quality. The tests you so carefully designed are passing, proving the product works as expected. It also shows your test cases have high quality; the results are repeatable and they’re successfully validating product functionality. This is the goal you’ve been striving for since you starting testing, right? If that’s the case, you’ve been shooting for the wrong goal.
A 100% pass rate implies your product works exactly as expected. The goal of testing, however, isn’t to prove the software works as expected. In fact, this objective can never even be achieved, because all complex software has defects. What sense does it make to have a goal that can’t be achieved? Your objective should be to write tests that expose these defects so an informed decision can be made on whether to fix them.
The other reason a 100% pass rate is the wrong goal is because when all your tests are consistently passing, they stop providing you new information. You see a perfect pass rate every morning, and pay no further attention to the tests. In other words, once you reach your “goal”, your tests stop helping you improve your product.
To be fair, from a management perspective a 100% pass rate is the right goal. It gives managers confidence in the quality of the product. But it can’t give them that confidence if both the managers and testers are striving for a perfect pass rate. In that case, no one is looking for bugs and the 100% pass rate means little.
Now assume you have a perfect pass rate, but the testers are striving to prove the software doesn’t work as expected. This shows management that despite the fact you’re looking for bugs, you still can’t find any. That will give management confidence in the software quality.
If you have a 100% pass rate, and still have time before Test Complete, don’t think of it as the end game. There are bugs in your product, and your tests aren’t finding them. Modify your tests. Write new ones. But design every test case with the goal of proving the software does NOT work as expected. The advice may sound obvious, but some testers don’t think this way; they’re much more interested in having their test passes go smoothly and on schedule.
Edsger Dijkstra wrote, “Testing can show the presence of errors, but not their absence.” When you change your mindset and write tests to prove the product doesn’t work, rather than to prove that it does work, you’ll be surprised how many more bugs you expose. This simple change will make you a better tester, and your products will be better for it.
Filed under: Test Planning | Tagged: Edsger Dijkstra, test pass rates |
Expert Testers 
You say “To be fair, from a management perspective a 100% pass rate is the right goal. It gives managers confidence in the quality of the product.” I am not sure this is entirely true.
As a manager you ought to be worried when you consistently have a 100% pass-rate of your automated tests. If however you expect your pass-rate on the last run on the acceptance environment to be 100% successfull I am going to fully agree.
Another point is, consistenly having a 100% pass-rate will give me, the tester, the strong feeling that our tests are not effective of that no new functionality is added to the software, or even worse, no optimizations to existing functionality is done. Both adding new things and adjusting existing items should result in a failure of at least some of your autoamted tests.
If my manager would set a goal for me to achieve 100% pass-rate I think I would prove to him, by example, what kind of dramatic results that can have. Just as in code coverage measurements, with a pass-rate as a target it is easily possible to create a very elaborate test setup which will seem to be testing a lot and have a high functionality coverage, but yet will not be capable of failing. Me happy, my manager happy, but do we actually deliver a good product? I honestly would not know, cause the only thing I had in mind was achieving the 100% pass-rate 🙂
In other words, if you as a tester get told you need to achieve a 100% pass-rate, please fight it, cause it makes no sense!
100% pass rates for automated tests can be a good thing. You wrote “…when all your tests are consistently passing, they stop providing you new information.”
If the system being tested remains exactly the same, the automated tests would indeed not be providing new information. However, the ultimate goal of automated tests is to provide fast feedback about a system that is being changed. As the system being tested changes, running the automated tests gives the tester some degree of confidence that the system being tested is at least consistent with previous results. Does consistency equate to quality? Only the tester can make the call.
The automated tests are just one tool for investigating the quality of the system. It’s ultimately the tester that decides the meaning of the tool’s results. If you see a 100% pass rate for your automated tests on a new build, that provides new information about the build and new information about your tests. Ultimately, these tests are just one piece of the bigger quality investigation puzzle.
If the system under test is a “Sustain” product (i.e not being updated, just maintained) then a 100% pass rate is a good and comforting thing. However, if you are still developing on a product, is is virtually impossible to have a 100% pass rate, and quite frankly would scare me. I gain a lot from the failed tests in my automated test suites, from defects to possible performance issues.
[…] that’s not even considering that 100% testcases passed is a metric that makes no sense. Quality is not only the amount of specific attributes, as ISO/IEEE standards may lead you to […]
[…] in May my colleague wrote a post on why 100% automation pass rates are bad. In the past I also questioned whether striving for 100% pass rates in automated test passes was an […]