Last year a program manager sent me an email I still think about. The product we were working on was just released to our dogfood environment, where it would be used by employees before being released to customers.
The next day we were still completing the analysis of our automated test failures. Our initial investigation determined that these failures shouldn’t hold up the release; they either didn’t meet the bug bar, or we thought they were caused by configuration issues in our test environment. But as I continued investigating, I discovered one of the failures actually was caused by a product bug that met the bar. Later that day, I found a second product bug. Shortly after logging the second bug I received this email.
When is testing supposed to end?
My instinct was that testing should never end. At the time, our product had only been released to dogfood, not production. There was no reason to stop testing since we could still fix critical bugs before they affected customers. And even if a bug was found after shipping to customers, it’s often possible to release an update to fix the problem.
But the more I thought about it, the more I realized it’s a perfectly legitimate question. Is there ever a time when testing can stop? If you’re shipping a “shrink-wrapped” product, then the answer is “yes”. If your product is a service, the answer is more complicated.
The minimum conditions required to stop testing are:
- Development for the feature is complete (no more code churn).
- The release criteria for the feature has been met.
- All failing tests and code coverage gaps have been investigated.
The first condition is that development has stopped and the code base is stable. As long as product code is changing, there’s a possibility new bugs are being introduced. The new code changes should be tested and regression tests executed.
Once the product code is stable, the release criteria must be met. The release criteria is typically agreed upon early in the test planning stage by both the Test and Dev teams. It can include conditions such as:
- No open P1 (severe) bugs
- 100% of tests executed
- 95% test pass rate
- 90% code coverage
You might think that once development is complete and your release criteria met, that it’s safe to stop testing; this is not the case. In the example above, the release criteria included a 95% automation pass rate. If your actual pass rate, however, is anything less than 100%, it’s imperative you investigate all failures. If they were caused by a bug that meets the bar, then the bug fix will violate our first condition that the code base remains stable, so testing must continue.
The same holds true for code coverage goals. Unless you have perfect code coverage, you need to address any gaps. This can be done either by creating new test cases, or by accepting the risk that these specific areas of code remain untested.
Once our three conditions are met, you can stop testing–if your product will be shrink-wrapped and shipped on a DVD. This doesn’t mean your product is bug-free; all relatively complex software has some bugs. But that’s why you created your release criteria—to manage the risk inherent in shipping software, not to eliminate it. If your minimum test pass rate or code coverage rate is anything less than 100%, you’re choosing to accept some reasonable amount of risk.
If your product is a service then your job isn’t done yet. You don’t need to continue running existing automation in test environments because the results will always be the same. But even though your product code isn’t changing, there are other factors that can affect your service. A password can expire, or a wrong value could be entered in a configuration file. There could be hardware failures or hacking attempts. The load may be more than you planned for. So even though traditional testing may have ended, you should now shift your efforts to monitoring the production environment and testing in production.