Ostrich a la Mode

One hallmark of a great test team is how they behave in the final days before a release. I’m not referring to working late to automate the last few tests, validate bug fixes, and run a final test pass. These long hours are an expected part of the job, and when you think about it, these tasks are really just bookkeeping. I’m talking about something that actually affects the product: does your test team still try to find bugs just before a release?

The best test teams I’ve worked on (and still do, by the way) looked for bugs right up to the release date. Testers took pride finding a release-blocking bug. The test team, manager included, congratulated the tester that found one. The developers and program managers (half) jokingly gave the tester a hard time, but everyone rallied together to resolve the issue on-schedule. Overcoming this adversity brought the team closer together. More importantly, we prevented a customer from finding the bug.

Some teams, however, don’t share this sentiment. In The Art of Software Testing, Glenford Meyers wrote that most teams are “measured on the ability to produce a program by a given date and for a certain cost.” Unfortunately, finding bugs late in the release cycle “may be viewed as decreasing the probability of meeting these schedule and cost objectives.”  These teams can go into ‘ostrich mode’–their heads buried in the sand for fear of finding a bug and missing their goal.

I remember working on a team a while back when our pass rate fell short of our release criteria. On the day of Test Complete, the team was called into a meeting where we were asked, “How can we get this pass rate above 95% today?”

The problem I had with this question was that the emphasis was on meeting the metric, not finding the bugs that caused the failures. Worse, it didn’t convey the sentiment that finding a bug would even be appreciated.

One way to quickly raise a pass rate is to fix errors in the test automation code. Another is to add new passing tests–with every new passing test case added, the pass rate increases. Both of these accomplish the stated goal, but neither improves the product quality. And what if we succeeded in reaching our pass rate goal? Shouldn’t we still investigate the failures to see if any were caused by a ship-blocking bug?

Instead of being asked how we could raise our pass rate, we should have been told, “Let’s investigate these failures right away and make sure any product bugs are triaged and resolved immediately.”

Meyers wrote that most teams “are more interested in having their system test pass proceed as ‘smoothly’ as possible and on-schedule, and are not truly motivated to demonstrate that the program does not meets its objectives.” As a tester, however, it’s your responsibility to report an accurate picture of the product quality. Think of yourself as a lawyer; the product is on trial and you’re representing the customers. If you find a ship-blocking bug, present it as evidence. A ‘jury’ will then use that evidence to decide whether to release on-schedule.

Have you ever worked on a team that went into ostrich mode? Please take the poll and share your stories in the comments section below.

%d bloggers like this: