TestApi… A Forgotten Soldier in the Fight Against Bugs.

Posted on May 31, 2012 by herrickspencer

Bug Fighters ala Starship Troopers

I was talking to my team today about doing globalization testing as a part of normal tests in our UI. In this conversation we were discussing ways of generating random strings using different unicode chars. I mentioned that there was a Microsoft created library for use in many different areas of testing, one of which is for string generation. However not many testers, even in Microsoft, know about or use this handy library. I’d previously used the string generation portion to do some fuzz testing in a different project, because it was much easier to use than most fuzz tools normally suggested for this purpose.

I decided it might be a good idea to disseminate this information so that it gets more use amongst our test warriors. The following is the basic information to get you started.

Here is the info on the TestApi library that Microsoft made and has many different uses:

§ Overview of TestApi

§ Part 1: Input Injection APIs

§ Part 2: Command-Line Parsing APIs

§ Part 3: Visual Verification APIs

§ Part 4: Combinatorial Variation Generation APIs

§ Part 5: Managed Code Fault Injection APIs

§ Part 6: Text String Generation APIs

§ Part 7: Memory Leak Detection APIs

§ Part 8: Object Comparison APIs

Here is an example from the String generation section that allows you to generate random strings for testing.

  //
  // Generate a Cyrillic string with a length between 10 and 30 characters.
  //

  StringProperties properties = new StringProperties();
  properties.MinNumberOfCodePoints = 10;
  properties.MaxNumberOfCodePoints = 30;
  properties.UnicodeRanges.Add(new UnicodeRange(UnicodeChart.Cyrillic));

  string s = StringFactory.GenerateRandomString(properties, 1234);

  The generated string may look as follows:
  s: Ӥёӱіӱӎ҄ҤяѪӝӱѶҾүҕГ

Enjoy!

Filed under: Coding Tips, Testing Tools | Tagged: , , , , , , , | 1 Comment »

Why 100% Test Pass Rates Are Bad

Posted on May 17, 2012 by Andrew Schiano
football goal

Are you shooting for the wrong goal?

When your test pass rate reaches 100% it’s usually cause for celebration. It shows your product has high quality. The tests you so carefully designed are passing, proving the product works as expected. It also shows your test cases have high quality; the results are repeatable and they’re successfully validating product functionality. This is the goal you’ve been striving for since you starting testing, right? If that’s the case, you’ve been shooting for the wrong goal.

A 100% pass rate implies your product works exactly as expected. The goal of testing, however, isn’t to prove the software works as expected. In fact, this objective can never even be achieved, because all complex software has defects. What sense does it make to have a goal that can’t be achieved? Your objective should be to write tests that expose these defects so an informed decision can be made on whether to fix them.

The other reason a 100% pass rate is the wrong goal is because when all your tests are consistently passing, they stop providing you new information. You see a perfect pass rate every morning, and pay no further attention to the tests. In other words, once you reach your “goal”, your tests stop helping you improve your product.

To be fair, from a management perspective a 100% pass rate is the right goal. It gives managers confidence in the quality of the product. But it can’t give them that confidence if both the managers and testers are striving for a perfect pass rate. In that case, no one is looking for bugs and the 100% pass rate means little.

Now assume you have a perfect pass rate, but the testers are striving to prove the software doesn’t work as expected. This shows management that despite the fact you’re looking for bugs, you still can’t find any. That will give management confidence in the software quality.

If you have a 100% pass rate, and still have time before Test Complete, don’t think of it as the end game. There are bugs in your product, and your tests aren’t finding them. Modify your tests. Write new ones. But design every test case with the goal of proving the software does NOT work as expected. The advice may sound obvious, but some testers don’t think this way; they’re much more interested in having their test passes go smoothly and on schedule.

Edsger Dijkstra wrote, “Testing can show the presence of errors, but not their absence.” When you change your mindset and write tests to prove the product doesn’t work, rather than to prove that it does work, you’ll be surprised how many more bugs you expose. This simple change will make you a better tester, and your products will be better for it.

Filed under: Test Planning | Tagged: , | 4 Comments »

Ten Firsts, Part 2

Posted on May 9, 2012 by brucecrms

As promised, here is part two of the ten ‘firsts’ in my career at Microsoft.  Click here to read the first five.  Now, on with the show!

First Bad Manager.  The culprit shall rename nameless.  The person was a long-time Microsoft developer.  He was a great developer, but lacked people skills.  At that time, Microsoft had limited training for managers.  Management felt, the person was a great developer/tester, they’ll make a great manager!  Once he became our manager, he would pop into our offices unannounced at a random time during the day and say “Status Me, Baby.”  You had to stop what you were doing and tell him.  We all know how important uninterrupted time is.  Well, almost everyone.

Lesson #6: Not all great {insert the discipline of your choice} make great managers.  The skillsets are different.  The skillset that got you in as an IC (Individual Contributor) won’t necessarily get you in as a manager.  Like anything, you need to learn.

First Automation System.  The project was Word for DOS 5.  Our automation system was called VCR.  You recorded keystrokes and mouse movements on one computer, and they were played back into another computer through the keyboard and mouse ports.  This was some of the first automated BVTs (Build Verification Tests) I had ever worked on.

Lesson #7: Automation matters.  It is the future of the discipline.  Hopefully the automation system is something more robust than VCR, but any extensible easy-to-maintain system is a good start.

First Black Hat.  After shipping Word for DOS 6.0, we came into the WinWord ship cycle close to RTM (Release to Manufacturing).  The six of us formed a black hat team.   We would read up some something for an hour, and then test the heck out of Word; rinse, lather, repeat.  We found twelve recall-class bugs during the RTM process.

Lesson #8: You don’t need to be an expert to test.  We are all intelligent folks.  Give the tester some information to get them started, and then let them run with it.  The innate tester will do the rest.

First Feature Cut During RTM.  I was the tester for a cutting edge technology feature in V1 of Office Web Server.  We provided the UI and relied on another team for the code that did the actual work.  The feature never gelled through the milestones: it would just crash from time-to-time. We talked with the other team and always walked away thinking “it will work fine with the next release.” The next release would have another major bug.  When we hit RC (Release Candidate), I ran some automation over the weekend in the automation lab.  We crashed one-out-of-three machines.  Needless to say, we had to cut the feature.  I spent the rest of the ship cycle verifying we removed every trace of the UI from the OS and Office apps.  Of all the products I’ve worked on over the years, this is the only one that never shipped.

Lesson #9: I learned back then what is obvious today:

  • Automation is important.
  • Cut your losses early.
  • Cross team collaboration is important.

First Addiction.  Word and web servers are fun, but one product turned out to be the “crack” in my Microsoft career.  I was part of the team for V1 of Microsoft OneNote.  All of the disciplines worked side-by-side to develop this product.  We met with customers, developed personas, ran usability studies, and so on; all in addition to our regular duties.  OneNote turned out to be one of Microsoft’s hidden gems.  We liked to joke “The first hit is free.” After we shipped, the disciplines came together again to form the product support for our customers.

Lesson #10: Jobs can be fun, but some are more fun than others. Find your ‘crack’ and enjoy your job just that much more.

Filed under: Careers in Test | Leave a Comment »

Follow

Get every new post delivered to your Inbox.

Join 290 other followers

%d bloggers like this: