• Twitter

    Error: Twitter did not respond. Please wait a few minutes and refresh this page.

  • Enter your email address to follow this blog and receive notifications of new posts by email.

    Join 371 other subscribers
  • Blog with Us!

    If you test at Microsoft, Apple, or Amazon, and are interested in contributing to the Expert Testers blog, please email us.
  • Legal Mumbo Jumbo

    The opinions expressed in this blog are those of the author and do not necessarily state or reflect those of Microsoft. Any reblogging without the expressed written consent of Major League Baseball is just fine with us.
  • Authors

Ostrich a la Mode

One hallmark of a great test team is how they behave in the final days before a release. I’m not referring to working late to automate the last few tests, validate bug fixes, and run a final test pass. These long hours are an expected part of the job, and when you think about it, these tasks are really just bookkeeping. I’m talking about something that actually affects the product: does your test team still try to find bugs just before a release?

The best test teams I’ve worked on (and still do, by the way) looked for bugs right up to the release date. Testers took pride finding a release-blocking bug. The test team, manager included, congratulated the tester that found one. The developers and program managers (half) jokingly gave the tester a hard time, but everyone rallied together to resolve the issue on-schedule. Overcoming this adversity brought the team closer together. More importantly, we prevented a customer from finding the bug.

Some teams, however, don’t share this sentiment. In The Art of Software Testing, Glenford Meyers wrote that most teams are “measured on the ability to produce a program by a given date and for a certain cost.” Unfortunately, finding bugs late in the release cycle “may be viewed as decreasing the probability of meeting these schedule and cost objectives.”  These teams can go into ‘ostrich mode’–their heads buried in the sand for fear of finding a bug and missing their goal.

I remember working on a team a while back when our pass rate fell short of our release criteria. On the day of Test Complete, the team was called into a meeting where we were asked, “How can we get this pass rate above 95% today?”

The problem I had with this question was that the emphasis was on meeting the metric, not finding the bugs that caused the failures. Worse, it didn’t convey the sentiment that finding a bug would even be appreciated.

One way to quickly raise a pass rate is to fix errors in the test automation code. Another is to add new passing tests–with every new passing test case added, the pass rate increases. Both of these accomplish the stated goal, but neither improves the product quality. And what if we succeeded in reaching our pass rate goal? Shouldn’t we still investigate the failures to see if any were caused by a ship-blocking bug?

Instead of being asked how we could raise our pass rate, we should have been told, “Let’s investigate these failures right away and make sure any product bugs are triaged and resolved immediately.”

Meyers wrote that most teams “are more interested in having their system test pass proceed as ‘smoothly’ as possible and on-schedule, and are not truly motivated to demonstrate that the program does not meets its objectives.” As a tester, however, it’s your responsibility to report an accurate picture of the product quality. Think of yourself as a lawyer; the product is on trial and you’re representing the customers. If you find a ship-blocking bug, present it as evidence. A ‘jury’ will then use that evidence to decide whether to release on-schedule.

Have you ever worked on a team that went into ostrich mode? Please take the poll and share your stories in the comments section below.

2 Responses

  1. It’s a good post.

  2. Andrew, what a great article !

    I was shocked and disturbed that ~80% of the 24 people who voted in the poll said that they had worked on a team that went into Ostrich Mode. Not possibly… but actually shut down. While I have been in situations where the team was exhausted and “tempted” to hide in the sand, we were able to fight through it. This is worrisome to me…. hey now that I think about it…. maybe this helps explain why many products work so poorly !

    In my 30 years experience there are many ways senior management can influence the Pass rate. They can limit resources or test duration to intentionally reduce the defect rate by limiting new defects found or fixed issues going back to open. They can also strong arm the test team as in your example, or in some case unknowingly limit the test teams effectiveness by assuming that “everything’s great so far, so what’s the risk of moving part or all of the team to work on the next great project? ” In the excitement of the current test effort going well, and wanting to start testing on the new product earlier, they inadvertently may compromise the current test effort.

    Regardless, one of QA tests most important roles is to be an advocate for the customer as well as present the most accurate assessment of how well the features work in the intended (or not intended) use of the application to protect the companies interest. You can’t do that if you put your head in the sand and cave in, during, or towards the end of the release. It is imperative that Senior management (whether they like it or not) have the most accurate picture for which features work, and those that do not work as expected, so that may be informed of the possible impact to their bottom line.

    In some cases Senior management may want “plausible deniability”, but QA should never willingly give into this type of mentality. If the QA team is being asked to compromise their tests, every effort should be made to find another way to complete the test objectives, failing that, tactically add a comment in the test report so there is a record, the test effort was compromised. It is important to understand also that Senior and/or Project Managers and/or major stake holders… will be under tremendous pressures to release a product, it is QA’s role to find a way to present the most accurate data and not be compromised during the test effort. As QA eng’s… if we feel that we are entering Ostrich mode, take it as a warning sign that your values are being compromised and stand up and fight to do your jobs correctly. Rarely will people fault you for doing your job to the best of your ability.

    In closing, I would have really liked to know how many who took the poll were junior testers <5 yrs experience. But I fear that the question as phrased “ have you ever worked on a team….” implies that even the Senior QA leads / QA Management were in Ostrich Mode.. which is un excusable.

    Again… great article
    Quality Assurance / Systems Test Manager

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: