angryweasel
Andrew Schiano
brucecrms
herrickspencer
kristianjaeger
Prakash
qingsongyao
ramonarjona
Rob TThere are two schools of thought when it comes to logging bugs. One is that every product defect should be logged and tracked as its own bug. The other is that multiple similar defects should be combined into a single bug. Which process is more efficient, and is the answer always the same?
Combining Similar Bugs
Let’s start with an example. Not long after I learned about accessibility testing, I was asked to take part in a bug bash. Accessibility testing ensures customers with physical limitations can use the product. It includes verifying the UI works correctly with large fonts, all controls are accessible without a mouse, proper tab order, etc. I decided to try out my new skills and test the UI.
I logged more than 50 defects in the first day of the bash. To my surprise, this did not go over well.
Triage was not happy about evaluating 50 fairly similar bugs. They claimed it would’ve been more efficient to track some of these defects together in a single bug. My fellow testers weren’t too happy either; the bug bash organizer was awarding a prize to the tester who logged the most bugs. It looked to them as if I padded my bug count to win a $10 Panera gift card.
I was told that combining similar defects into a single bug had the following advantages:
- The triage process goes quicker. Some of the issues I logged were so similar, there was no need to evaluate them individually.
- The bug-fix process goes quicker. If I had combined these defects based on the UI screen where they were found, the developers could have fixed several bugs at once. Instead, they checked-out, modified, and checked-in the same file several times.
- The testing process goes quicker. Since developers resolve all the similar defects at once, testers can validate all the fixes at once. This is a tremendous time saver, especially if validating a defect requires deploying a new server or topology.
I was convinced. When we had our next bug bash, I combined my similar defects. I quickly found that although this saved time for the triage team, it added overhead for me.
Every time I logged a bug, not only did I have to check if the same issue was already logged, but I had to check if any similar issue was logged. And what exactly defines similar when it comes to combining bugs? If misaligned text appears on two different UI screens, should they be combined? Do the defects have to be in the same area of the application? Do they need to have the same root cause?
Once these bugs went to triage, more problems were revealed. Several of my combined bugs were closed as “Won’t Fix”. I found that if I combined four defects into one bug, and the triage team determined we shouldn’t fix one of the defects, they rejected the whole bug. It was then my responsibility to create a new bug with just the defects that would be fixed.
There were other issues. Even if the defects were similar, I had no way of knowing the same developer would be assigned to fix all of them. In some cases, the work had to be load-balanced between two developers. These bugs were assigned back to me to split up. I also had to split bugs if triage wanted to assign a different priority or severity to each defect.
Combining bugs didn’t have the expected benefits for the developer, either. When several defects were tracked in the same bug, they all had to be resolved and checked-in together. Developers were no longer able to fix one defect immediately, and save the rest for later.
This leads us back to our original question. Is it better to track multiple similar defects together, or is it better to log them separately? Some defects lend themselves to being combined, such as when the same UI screen has three hardcoded strings. Others don’t, such as when two sets of repro steps with different call stacks both result in a “can’t connect to web service” error.
I suggest you first get your team on the same page regarding the criteria for combining bugs. I’ve found, however, that in almost all cases, it’s better to log defects individually. The only time it’s more efficient to combine defects is when you know all the following are true. (Here’s a tip: you’ll never know if all the following are true.)
- Triage will assign the same status (accept, reject, postpone, etc.) to all the defects.
- Triage will assign the same priority and severity to all the defects.
- The same developer will be assigned to fix all the defects.
- The developer will be making all the fixes at the same time.
- The same tester will be assigned to validate all the fixes.
Some bug-tracking systems allow related bugs to be “linked” together, which is a great compromise. Now when I log bugs, I almost always follow the “one defect per bug” rule. But if I log similar bugs, I use this feature to link them together. The triage team might not be crazy about it, but it saves me and the company time. As a side benefit, I also have a wallet full of Panera gift cards.
Filed under: Bug Tracking | Tagged: Accessibility Testing |
Expert Testers 
Should multiple similar bugs be combined?
Sometimes. But probably not when there is a $10 Panera gift card at stake!
I wonder if the decision to combine silimar bugs is related to the decision to combine similar tests. (See https://experttesters.com/2012/02/06/the-testers-dilemma-to-pad-or-not-to-pad/ )
Although bugs may appear to be similar, they in fact, may not be similar. QA is responsible to record all defects, how that is achieved leaves some room for interpretation, and many of those points are presented in the article.When in doubt I always instruct my QA Eng to write it up, and we will sort it out later. Too bad if its hundreds of bugs !
I smiled when the author found 50 defects in his first round of testing, that meant the QA team was jaded (were used to the bugs so they just lived with it), and the developers did a lousy job and were offended that someone actually documented the bugs in the first place. I mean come on….. a disabled person who is struggling to see, or is having other difficulties doesn’t need UI bugs to impede them further !!!
Possibly another reason for the outcry is someone with a fresh look at the UI found so many issues in the first place. (as a side note I find the first time a QA eng tests an app as the most precious. That is the only time it is truly seen its its most native form, after that.. no matter how hard we try… we never see it with a fresh perspective again..,,i digress) Anyway, It is the teams responsibilities to go through each defect and determine if similar to another. Period !.
By all means If QA can determine….. with “reasonable confidence” at the time the first bug is going to be posted that multiple defects are related to a single root cause (lets say green was used when red was supposed to be used) then sure no biggie post one defect with all areas affected clearly listed (be ready for some but not all to be fixed in this case)
If multiple defects are written…. then an approach i use when I manage defects is that QA takes the responsibility to link similar defects to a master defect (i update the linked to as well as the linked defects so they are cross referenced. I then run a filter and remove all linked issues so that the developers do not become overwhelmed. Then when developer x works on a master defect and fixes it, he simply looks at the linked defects.. if the linked defects are the same root cause.. voila they get marked fixed as well.. if he finds a linked defect that he now believes is because of a different reason, that defect goes back to open (is unlinked from the master -with proper explanation) and goes into the pool of defects to be fixed. In closing QA owns the defect system, be sensitive to the dev very hard work, but always capture defects at the lowest common denominator possible
We user Jira as our bug tracking tool. One of the features we use is the creation of subtasks. That allows us to put similar bugs into a “rollup” issue, then the developer can cut out individual portions into separate issues for individually tracking and closing during test. Tool flexibility goes a long way to remediating these type of issues.
Personally I get offended when the number of bugs that I find gets tied to a bonus or a performance evaluation and I have experienced how much damage bug counting bonuses can do to the relationship between me as a tester and developers.
If I would get into a situation were it could be suspected that I am just improving statistics for a reward I might opt-out and refuse any reward. I have at times also “given away” bug reports to team members; After a while the test managers looks silly giving bug counting rewards when no one knows who really found the bug..