Expert Testers

This is a follow on to Andrew’s article “State-Transition Testing.”  If your software can be described using states, you can use monkey automation to test your product.  While smart monkeys can take a little time to implement, even using third-party automation software, their payback can be enhanced if you release multiple versions, follow a rapid release cadence, or are short on testing resources.

Let me start by noting that Smart Monkey testing differ from Netflix’s Simian Army.  Similar name, both are code testing code.  Other than that, they are different.

In general, monkey testing is automated testing with no specific purpose in mind other than to test the product or service.  It’s also known as “stochastic testing” and is a tool in your black box tester tool belt.  Monkey testing comes in two flavors:

Dumb Monkey testing is when the automation randomly sends keystrokes or commands to the System Under Test (SUT).  The automation system or tester monitors the SUT for crashes or other incorrect reactions.  One example would be feeding random numbers into a dialog that accepts a dollar value.  Another example comes from when I was a tester in Office.  One feature I was testing is the ability to open web pages in Microsoft Word.  I wrote a simple macro in Visual Basic for Applications (VBA) to grab a random web page from the web browser, log the web address to a log file, and try to open the web page in Word.  If Word crashed, my macro would stop, and I would know that I hit a severity one (i.e. crashing) bug.  I can watch my computer test for me all day…or just run it at night.  Free bugs!

Funny Aside: I had a little code in the macro to filter out, um, inappropriate web sites.  Occasionally one got through the code, an alarm would go off in our IT department, and my manager would receive an email that I was viewing sites against HR’s policy.  It happened every few months.  I would show that it was the automation, not me, doing the offensive act.  We would laugh about it, put a letter of reprimand in the monkey’s employee file, and move on.

Smart Monkey testing is, well, smarter.  Instead of doing random things, the automation system knows something about your program:

• Where it is
• What it can do there
• Where it can go and how to get there
• Where it’s been
• If what it’s seeing is correct

This is where State Transition comes into play.

Let’s look at a State Transition diagram for a Learning Management System (LMS) and how the enrollment status of a student could change.

You would define for the smart monkey automation the different states, which states changes are possible, and how to get to those.  For the diagram above, it might look something like this:

Where it is: Registered

What it can do there #1: Learner can Cancels their registration

What it can do there #2: Learner can Attend the class

Where it is: Canceled

What it can do there #1: Learner can Register for the class

What it can do there #2: Learner can be added to the Waitlist

Where it is: Waitlisted

What it can do there #1: Learner can Cancel their registration

What it can do there #2: The learner will be automatically registered by the system if a spot opens for them

You get the general idea.  This still looks like Andrew’s State-Transition Testing.  What is different is the automation knows this information.  When you start the automation, it will start randomly traversing the states.  Sometimes it will follow an expected path:

Register | Attend

The next time, it might try a path you didn’t expect (the learner ignores their manager and attends anyways):

Request | Request Declined | Walk-in

It might also do things like request and cancel the same class fifty times.

Don’t have time to define all of the states?  Some third party software will randomly explore your software and create the state model and how to traverse it for you. You can then double-check the model it create, make any corrections or additions, and you are on your way!

How can you improve this system?

1. Have the monkey access an XML or other data file with specific cases you would like hit more than the random ones.  For example, you could use the PICT tool to create a list of the most interesting combinations of inputs to hit.
2. You can also make this system smarter by assigning probabilities to each state change.  Find out how often the user actually cancels a registration.  Now feed that data into the smart monkey.  Now your monkey will test the state changes at the same frequency as the real user.
3. The next step?  Tie your monkey into live customer data: Customer data-driven quality (CDDQ).  For example, let’s say all of a sudden your customers start canceling a lot of class registrations due to an upcoming holiday.  Your smart monkey will automatically start testing the cancel registration state change more often.

The whole idea of smart monkey testing is it will follow expected and unexpected paths.  You can run the monkey on spare machines, on your test machine overnight, pretty much anytime; and give you free testing.  If your logging is good enough and tracks the state, and which transition path it followed, you will be able to reproduce any bugs it finds.  Watch your code coverage numbers go up…maybe.  But that’s fodder for another posting.

Long live the smart monkey!

Filed under: Testing Tools | Tagged: automation software, automation system, smart monkey, state transition | 3 Comments »