Expert Testers

You are probably familiar with the testing approaches of black box, white box, and gray box testing.  Each “tool” in the tester’s tool belt can be used in the right circumstances, or misused in the wrong circumstances.  Exploratory Testing (ET) can be used in almost all circumstances, and whether done formally or informally, it is a tool we shouldn’t be afraid to use.

Exploratory testing (ET) is something you probably already do. It is more than just “clicking around” the product.  ET is defined as a test-execution approach where the tester uses information gained while performing tests to intuitively derive additional tests. You can think of it as that little voice in the back of your head telling you “Did I just see something that looked wrong? I better check that out more deeply.” This is subtly different from black-box (BB) testing where you apply tools like Boundary Value Analysis (BVA) and Equivalence Class (EQ) to first develop a list of tests, and second run those tests. It also differs from gray-box (GB) testing where you first use internal knowledge of the structure of the feature and code to develop a list of tests, and second run those tests. You can think of ET as BB and GB testing with a feedback loop—you do test design and test execution at the same time. You are free to explore other avenues of the product in order to track down bugs and issues.

Exploratory testing provides value to the testing effort. It is generally good at evaluating the “look and feel” of a project, but several studies raise important questions about the overall effectiveness and efficiency of behavioral testing and popular exploratory testing approaches to software testing. The details of the studies can be found in chapter six of How We Test Software at Microsoft.

ET can be explained with an analogy (from James Bach’s “Exploratory Testing Explained“):

Have you ever solved a jigsaw puzzle? If so, you have practiced exploratory testing. Consider what happens in the process. You pick up a piece and scan the jumble of unconnected pieces for one that goes with it. Each glance at a new piece is a test case (“Does this piece connect to that piece? No? How about if I turn it around? Well, it almost fits but now the picture doesn’t match…”). You may choose to perform your jigsaw testing process more rigorously, perhaps by concentrating on border pieces first, or on certain shapes, or on some attribute of the picture on the cover of the box. Still, can you imagine what it would be like to design and document all your jigsaw “test cases” before you began to assemble the puzzle, or before you knew anything about the kind of picture formed by the puzzle?

When I solve a jigsaw puzzle, I change how I work as I learn about the puzzle and see the picture form. If I notice a big blotch of color, I might decide to collect all the pieces of that approximate color into one pile. If I notice some pieces with a particularly distinctive shape, I might collect those together. If I work on one kind of testing for a while, I might switch to another kind just to keep my mind fresh. If I find I’ve got a big enough block of pieces assembled, I might move it into the frame of the puzzle to find where it connects with everything else. Sometimes I feel like I’m too disorganized, and when that happens, I can step back, analyze the situation, and adopt a more specific plan of attack. Notice how the process flows, and how it remains continuously, each moment, under the control of the practitioner. Isn’t this very much like the way you would assemble a jigsaw, too? If so, then perhaps you would agree that it would be absurd for us to carefully document these thought processes in advance. Reducing this activity to one of following explicit instructions would only slow down our work.

This is a general lesson about puzzles: the puzzle changes the puzzling. The specifics of the puzzle, as they emerge through the process of solving that puzzle, affect our tactics for solving it. This truth is at the heart of any exploratory investigation, be it for testing, development, or even scientific research or detective work.

Key advantages of ET:

• Exploratory testing is heavily influenced by the tester’s in-depth system and domain knowledge and experience. The more you know, the better you are at following the paths that are most likely to find bugs or issues.
• Less preparation is needed.
• Important bugs are quickly found.
• ET tends to be more intellectually stimulating than execution of scripted tests.
• Even if you come back and test the same area again, you are likely to perform your tests in a slightly different way (you aren’t following a script), so you are more likely find more bugs.
• ET is particularly suitable if requirements and specifications are incomplete, or if there is a lack of time.
• ET can also be used to  validate that previous testing has found the most important defects.
• ET is better than just testing.

Key drawbacks of ET:

• You must manage your time wisely. You need to know when to stop pursuing one avenue and move on to another.
• You can’t review cases in advance (and by that prevent errors in code and test cases).
• It can be hard to reproduce tests later unless you are documenting everything you do.  One idea is to use a screen recorder whenever you are doing ET.
• It can be difficult to know exactly which tests have been run. This can be partially alleviated if you are recording your test steps and creating automation, or if you tracking code coverage.
• You may end up testing paths that the user would never do. You can use customer data as an addition to your ET so that you don’t spend time testing areas that don’t need be tested.

Use ET when:

• You need to provide feedback on a new product or feature.
• You need to quickly learn a new product.
• You have already been using scripts and seek to diversify the testing.
• You want to find the single most important bug in the shortest time.
• You want to check the work of another tester by doing a brief independent investigation.
• You want to find and isolate a particular defect.
• You want to determine the status of a particular risk, in order to evaluate the need of scripted tests in that area.
• You are on a team practicing agile or Extreme Programming.

The last bullet deserves some context. Why would an agile team be interested in ET? Agile teams can suffer from groupthink. The team members spend all day working together, talking, coding, attending meetings, and so on. They tend to start thinking alike. While this helps the agile process, it can hinder testing. Why? Everyone starts to think about the product in the same way and use the product in the same way. Your scripted tests start following the same sequence as the developer’s code. ET can help break that groupthink, randomize the testing, and find issues that the customer would.

Are you a master or an amateur ET tester?

• Test design: Exploratory tester is first and foremost a test designer. Anyone can design a test accidentally. The excellent exploratory tester is able to craft tests that systematically explore the product. This requires skill such as the ability to analyze a product, evaluate risk, use tools, and think critically, among others.
• Careful observation: Excellent exploratory testers are more careful observers than novices, and for that matter, experienced scripted testers. The scripted tester will only observe what the script tells them to observe. The exploratory tester must watch for anything unusual or mysterious. Exploratory testers also must be careful to distinguish observation from inference, even under pressure, lest they allow preconceived assumptions to blind them to important tests or product behavior.
• Critical thinking: Excellent exploratory testers are able to review and explain their logic, looking for errors in their own thinking. This is especially true when reporting the status of a session of exploratory tests investigating a defect.
• Diverse ideas: Excellent exploratory testers produce more and better ideas than novices.  They may make use of heuristics to accomplish this. Heuristics are devices such as guidelines, generic checklists, mnemonics, or rules of thumb. The diversity of tester temperaments and backgrounds on a team can also be harnessed by savvy exploratory testers through the process of group brainstorming to produce better test ideas.
• Rich resources: Excellent exploratory testers build a deep inventory of tools, information sources, test data, and friends to draw upon. While testing, they stay alert for opportunities to apply those resources to the testing at hand.

Exploratory testing can be valuable in specific situations and reveal certain categories of defects more readily than other approaches. The overall effectiveness of behavioral testing approaches is heavily influenced by the tester’s in-depth system and domain knowledge and experience. Of course, the effectiveness of any test method eventually plateaus or becomes less valuable and testers must employ different approaches to further investigate and evaluate the software under test (The Pesticide Paradox).

Filed under: Test Planning, Testing Tools | Tagged: Manual testing |