The Case for Fewer Test Cases

Robotium Remote Testing

Testers are often encouraged to automate more and more test cases. At first glance, the case for more test cases makes sense—the more tests you have, the better your product is tested. Who can argue with that? I can.

Creating too many test cases leads to the condition known as “test case bloat”. This occurs when you have so many test cases that you spend a disproportionate amount of time executing, investigating, and maintaining these tests. This leaves little time for more important tasks, such as actually finding and resolving product issues. Test case bloat causes the following four problems:

1. Test passes take a long time to complete.

The longer it takes for your test pass to complete, the longer you have to wait before you can begin investigating the failures. I worked on one project where there were so many test cases, the daily test pass took 27 hours to finish. It’s hard to run a test pass every day when it takes more than 24 hours to complete.

2. Failure investigations take a long time to complete.

The more tests you have, the more failures you have to investigate. If your test pass takes a day to complete, and you have a mountain of failures to investigate, it could be two days or longer before a build is validated. This turn-around time may be tolerable if you’re shipping your product on a DVD. But when your software is a service, you may need to validate product changes a lot faster.

For example, the product I’m working on is an email service. If a customer is without email, it’s unacceptable for my team to take this long to validate a bug fix. Executing just the highest-priority tests to validate a hot-fix may be a valid compromise. If you have a lot of test cases, however, even this can take too long.

3. Tests take too much effort to maintain.

When your automation suffers from test case bloat, even subtle changes in product functionality can cause massive ripples in your existing test cases, drastically increasing the amount of time you spend maintaining them. This leaves little time for other, more valuable tasks, such as testing new features. It’s also a morale killer. Most testers I know— the really good ones, at least— don’t want to continually maintain the same test cases. They want to test new features and write new code.

4. After a certain threshold, more test cases no longer uncover product bugsthey mask them.

Most test cases only provide new information the first time they’re run. If the test passes, we can assume the feature works. If the test fails, we file a bug, which is eventually fixed by development, and the test case begins to pass. If it’s written well, the test will continue to pass unless a regression occurs.

Let’s assume we have 25 test cases that happily pass every time they’re run. At 3:00 a.m. an overtired developer then checks in a bug causing three tests to fail. Our pass rate would drop from 100% to an alarming 88%. The failures would be quickly investigated, and the perpetrator would be caught. Perhaps we would playfully mock him and make him wear a silly hat.

But what if we had 50 test cases? Three failures out of 50 test cases is a respectable 94% pass rate. What about a hundred or two hundred tests? With this many tests, it’s now very possible that there are some amount of failures in every pass simply due to test code problems; timing issues are a common culprit. The same three failures in two hundred tests is a 99% pass rate. But were these failures caused by expected timing issues, or a real product bug? If your team was pressed to get a hot-fix out the door to fix a live production issue, it may not investigate a 99% pass rate with as much vigor as an 88% pass rate.

Bloat Relief

If your automation suffers from test case bloat, you may be able to refactor your tests. But you can’t simply mash four or five tests with different validation points into a single test case. The more complicated a test, the more difficult it becomes to determine the cause and severity of failure.

You can, however, combine test cases when your validation points are similar, and the severity of a failure at each validation point is the same. For example, if you’re testing a UI dialog, you don’t need 50 different test cases to validate that 50 objects on the screen are all at their expected location. This can be done in one test.

You can also combine tests when you’re checking a single validation point, such as a database field, with different input combinations. Don’t create 50 different test cases that check the same field for 50 different data combinations. Create a single test case that loops through all combinations, validating the results.

When my test pass was taking 27 hours to complete, one solution we discussed was splitting the pass based on priority, feature, or some other criteria. If we had split it into three separate test passes, each would have taken only nine hours to finish. But this would have required three times as many servers. That may not be an issue if your test pass runs on a single server or virtual machines, however I’ve worked on automation that required more than twenty physical servers–tripling your server count is not always an option.

In addition to the techniques discussed above, pair-wise testing and equivalence class partitioning are tools that all testers should have in their arsenal. The ideal solution, however, is to prevent bloating before it even starts. When designing your test cases, it’s important to be aware of the number of tests you’re writing. If all else fails, I hear you can gain time by investigating your test failures while travelling at the speed of light.

Performance Test Documents

After reading Andrew’s article on test plans I started thinking about my own experiences with writing test documents. In this article I’ll describe the different types of performance test documents my team creates.

Test Strategy Documents

The first type is high-level test strategy documents. One of my team’s responsibilities is to provide guidance on performance tools, techniques, infrastructure, and goals. We document this guidance and share it with our peer feature teams. These teams use the guidance when planning out specific test cases for their features.

Strategy documents provide value in a number of ways. They help my team gain clarity on strategy, they act as documentation that feature teams can use to learn about performance testing, and they assist us in obtaining sign-off from stakeholders.

I wrote one strategy document this year and it provided all of the above. I still occasionally refer to it when I’m asked about the performance testing strategy for our organization.

Test Plan Documents

Besides creating strategy documents, we also write more traditional test plan documents. These documents define a set of tests that we intend to execute for a project milestone. They include details of the tests, features that will be covered, the expected performance goals, and the hardware and infrastructure that we will use.

Similar to strategy documents, test plans help us gain clarity on a project and act as a springboard for stakeholder review. They seem to have a shorter shelf life though — I don’t find myself reviewing old performance test plans. My approach has been to “write it, review it, and then forget about it.”

Interestingly enough, I do find myself reviewing old test plans authored by other teams. Occasionally we need to write performance tests for a feature that we’re unfamiliar with. The first thing I do is review old functional test plans to understand how the feature works and what the feature team thought were the most important test scenarios. These test plans are invaluable in getting us ramped up quickly.

Result Reports

When my team completes a milestone I like to write a report that details the results and conclusions of the performance testing. These reports contain performance results, information about bugs found, general observations, and anything else I think might be useful to document. I send the final report to stakeholders to help them understand the results of testing.

One thing I really like about these reports is that they help me figure out which types of testing provided the most value. They also help me figure out how we can improve. When I start planning a new milestone, I first go through the old reports to get ideas.

Wrapping Up

Documentation isn’t always fun but I do find that it provides value for me, my team, and the organization. I’d like to pose a question to readers — what types of test documents do you create, and how do they provide value?

Thanks for reading!

– Rob

Are Test Plans a Waste of Time?

Acceptance Test Plan Template, Test Methodology

Image by via Flickr

If you’re like many testers I know, you hate test plans. Most testers fall into three groups: Group A doesn’t like writing test plans, Group B thinks test plans are a waste of time, and Group C thinks groups A and B are both right.

Let’s start with Group A. Some testers don’t like writing test plans because they simply don’t enjoy writing. Their dislike has less to do with what they are writing, than the fact that they are writing. Perhaps writing does not come naturally to them. One tester recently told me, “I write like a third grader.” It’s no surprise he doesn’t like writing test plans–we like to do things we’re good at.

Some testers don’t enjoy writing test plans because they are not really testers at all–they are developers working in Test. They would much rather if someone else analyzed the functional specifications, wrote the test plan, and told them exactly what they needed to automate.

Like many testers, all of my previous jobs had been in development. I thought writing a test plan would be easy–until I saw my first test plan template. Performance testing? Stress testing? Long-haul testing? I quickly learned that I had no idea how to write a test plan, and I dreaded writing one. But I started reading plans from testers I respected, and I started reading books on testing. I slowly made the transformation from developer to tester. Now I enjoy writing test plans because I know it makes my testing better.

Perhaps the most interesting reason testers dislike test plans is because they don’t think the plans are useful. They point to the fact that most test plans are never updated. As the product evolves, test plans become out of sync with the current functionality. I’ve found this to be true. Of all the test plans I’ve written in my career, I know exactly how many I updated after the product changed: zero.

This is a problem for two reasons. First, it would be one thing if test plans were quick and easy to write; they are not. Depending on the feature, it can take me a week to write a solid detailed test plan. Some would argue this time could be better spent automating test cases or performing exploratory testing.

Even worse, test plans that are out of sync with product functionality give inaccurate information to the reader. Recently, I worked on a product that was being updated for its next release, and I was assigned to test a feature I was completely unfamiliar with. The first thing I did was review the original test plan to learn how the feature worked and how it was first tested. I assumed, incorrectly, that the document was up to date. As a result, I reported bugs even though the feature was working properly.

James Whittaker, Test Director at Google, recently debated the value of creating test plans on the Google Testing blog:

As to whether it is worth doing, well, that is another story entirely. Every time I look at any of the dozens of test plans my teams have written, I see dead test plans. Plans written, reviewed, referred to a few times and then cast aside as the project moves in directions not documented in the plan. This begs the question: if a plan isn’t worth bothering to update, is it worth creating in the first place?

Other times a plan is discarded because it went into too much detail or too little; still others because it provided value only in starting a test effort and not in the ongoing work. Again, if this is the case, was the plan worth the cost of creating it given its limited and diminishing value?

Some test plans document simple truths that likely didn’t really need documenting at all or provide detailed information that isn’t relevant to the day to day job of a software tester. In all these cases we are wasting effort.

I agree that there may be some wasted effort in writing a test plan. For instance, I’m guilty of spending too much time tweaking the wording of my documents. It’s a test plan, not a novel. Bullet points and sentence fragments are fine–you shouldn’t be spending time using a thesaurus to find synonyms for the word “feature”.

But that doesn’t mean it’s all wasted effort. In fact, I believe the benefit far outweighs the effort. This is true even if the test plan quickly becomes obsolete.

Consider feature specification documents: much like test plans, feature specifications often become outdated as the product evolves. This doesn’t this mean we shouldn’t write feature specifications. Potential document “staleness” is not a valid argument against writing spec documents—or test plans. Just don’t make the mistake of assuming old specifications are still accurate.

One of the most important reasons for creating a test plan is to get your thoughts about testing the feature onto paper and out of your head. This unclutters your mind and helps you think clearly. It also documents every idea you’ve had about testing the feature, so none are forgotten.

The writing process often leads you to think of both more and better ways to test the feature. Your first pass at writing a test plan may include mostly positive functional test cases, as well as a handful of negative functional tests. But the process of refining your document leads you to consider more negative tests, more boundary tests, more concerns around scalability and security. The more time you spend planning your testing, the more complete your testing will be.

Detailed test plans can also help you find bugs in the feature before the code has even been implemented, when they are much less costly to address. Two techniques that are excellent for finding design holes during the test planning phase are Decision Tables and State Transition Diagrams. I remember creating a large Decision Table as part of a test plan for a security feature which uncovered that nearly 10% of the possible input combinations didn’t have an expected result in the design specification document.

Test documents are also valuable for conducting test plan reviews. After creating your test plan, it’s important to have it reviewed by other testers–no single tester will ever think of all possible test cases. It’s also valuable to have your plan reviewed by both the developers and program managers. In my most recent test plan review, a program manager told me that one of the features I planned on testing had been cut. In the same review, a developer informed me about existing test hooks that saved me hours of development time.

When testers say they don’t want to write a test plan, I can sympathize. Most of us got into this business either because we like to program or because we like to break things, not because we like to write documents. But when testers say they don’t want to write a test plan because it’s not valuable, I have to disagree. Good test plans make your product better. So what if they become obsolete? As both Dwight D. Eisenhower and Wile E. Coyote once said, “plans are useless, but planning is indispensable.”

%d bloggers like this: