TestApi… A Forgotten Soldier in the Fight Against Bugs.

Posted on May 31, 2012 by herrickspencer

Bug Fighters ala Starship Troopers

I was talking to my team today about doing globalization testing as a part of normal tests in our UI. In this conversation we were discussing ways of generating random strings using different unicode chars. I mentioned that there was a Microsoft created library for use in many different areas of testing, one of which is for string generation. However not many testers, even in Microsoft, know about or use this handy library. I’d previously used the string generation portion to do some fuzz testing in a different project, because it was much easier to use than most fuzz tools normally suggested for this purpose.

I decided it might be a good idea to disseminate this information so that it gets more use amongst our test warriors. The following is the basic information to get you started.

Here is the info on the TestApi library that Microsoft made and has many different uses:

§ Overview of TestApi

§ Part 1: Input Injection APIs

§ Part 2: Command-Line Parsing APIs

§ Part 3: Visual Verification APIs

§ Part 4: Combinatorial Variation Generation APIs

§ Part 5: Managed Code Fault Injection APIs

§ Part 6: Text String Generation APIs

§ Part 7: Memory Leak Detection APIs

§ Part 8: Object Comparison APIs

Here is an example from the String generation section that allows you to generate random strings for testing.

  //
  // Generate a Cyrillic string with a length between 10 and 30 characters.
  //

  StringProperties properties = new StringProperties();
  properties.MinNumberOfCodePoints = 10;
  properties.MaxNumberOfCodePoints = 30;
  properties.UnicodeRanges.Add(new UnicodeRange(UnicodeChart.Cyrillic));

  string s = StringFactory.GenerateRandomString(properties, 1234);

  The generated string may look as follows:
  s: Ӥёӱіӱӎ҄ҤяѪӝӱѶҾүҕГ

Enjoy!

Filed under: Coding Tips, Testing Tools | Tagged: , , , , , , , | 1 Comment »

Video Is Worth A Thousand Words

Posted on February 1, 2012 by herrickspencer

When attempting to file a bug, some people are not the best at explaining the issue (you know who you are), and time is lost by triage trying to understand the issue, as well as by the filer trying to answer any questions by developers. There is also the risk that the bug will be mismarked as ‘no repro’ or ‘by design’ if it is not well understood.

Therefore, your best friend is very good repro notes to explain how a developer can reproduce the same issue on their own environment. It is even highly advised to add a picture of the issue and attach it to the bug to give the bug readers a quick and easy way to fully understand the issue at a glance. If you simply take a screen shot (PrtScr key), paste into Paint and add that file, you are ahead of the curve… but really, the “Snipping Tool” that ships with Win7 and forward is way easier to use and allows you to annotate the image before saving it.

Yeah, smarty pants… but what about an issue that involves a set of complicated Repro Steps?

If your issue contains a series of steps, or the reactions are hard to describe, what do you do then? A picture of the one event will not suffice. This is when you go to the movies. No, not the new action flick that was a remake of a much better foreign film…. I mean MAKE a movie of the bug!

There are several options to make movies of your mouse screen actions. I’ve tried quite a few, but the one that beats all the others in my opinion is one unfortunately available only to Microsoft internal employees… so this will not help in this post…. but I’ll detail the features that are essential to me in a screen recorder, and let you evaluate some of the options available out there.

image_thumb3 Screen Recorder is a product created by a developer here at Microsoft, and meets and exceeds my expectations for a good option in bug reporting. Unfortunately, it is not yet available to the general public, but it does illustrate what a good app for bug reporting should look like.

This app does lots of things correctly:

  • very simple UI (see above)
  • allows you to configure where the output file is written easily
  • outputs to Windows media file (some apps you have to do the final encoding yourself)
  • allows for audio recording (configured from the file output menu)
  • allows for pausing and resume
  • allows for full screen or selection of one running application

This method is perfect for adding a WMV to a bug or presentation to easily portray a bug. This is also great for tutorials, how-to wiki, and PowerPoint presentations.

One publicly available option I’ve tried is “My Screen Recorder”.  It has most of the options detailed above, but is not a simple UI, however I’d say it’s the best option at present.  Another was Microsoft’s Expression Encoder, which was very versatile, but way too complex for this application, and did not encode the video in same step as the recording, which was very time consuming.

Lastly, I should mention that Windows7 ships with it’s own bug recording software for reporting bugs to Microsoft. This software, called “Problem Steps Recorder” can be used to create a detailed HTML page that includes step by step screen shots of the repro, which could be an option on a machine where you can not install additional software. A detailed view of how to use this option is shown in this TechRepublic blog post.

It would be great to see if any one of you has additional options for screen casting software that you think is superior, and why. Please feel free to leave that suggestion in the comment section, so that we all can test it out, pun intended.

Please remember that videos make excellent supplements to a bug, but they do not substitute for good, searchable text describing issues and its repro steps. The bug should include good setup, initialization, and execution steps for the issue which most teams would consider mandatory. 

For bugs to be most useful, they should be reported containing all investigatory evidence:

  • Expected result/actual result notes
  • Source data/files
  • Exception details
  • Stack trace (if possible)
  • And of course… THE MOVIE!

So let’s go to the movies people… make bug resolution more efficient in the process!

Enjoy!

Filed under: Bug Tracking | Leave a comment »

%d bloggers like this: