UI Testing Checklist

Posted on June 2, 2014 by Andrew Schiano

When testing a UI, it’s important to not only validate each input field, but to do so using interesting data. There are plenty of techniques for doing so, such as boundary value analysis, decision tables, state-transition diagrams, and combinatorial testing. Since you’re reading a testing blog, you’re probably already familiar with these. Still, it’s still nice to have a short, bulleted checklist of all the tools at your disposal. When I recently tested a new web-based UI, I took the opportunity to create one.

One of the more interesting tests I ran was a successful HTML injection attack. In an input field that accepted a string, I entered: <input type=”button” onclick=”alert(‘hi’)” value=”click me”>. When I navigated to the web page that should have displayed this string, I instead saw a button labeled “click me”. Clicking on it produced a pop-up with the message “hi”.  The web page was rendering all HTML and JavaScript I entered. Although my popup was fairly harmless, a malicious user could have used this same technique to be, well, malicious.

inject

 

 

 

 

 

 

Another interesting test was running the UI in non-English languages. Individually, each screen looked fine. But when I compared similar functionality on different screens, I noticed some dates were formatted mm/dd/yyyy and others dd/mm/yyyy. In fact, the most common bug type I found was inconsistencies between screens. The heading on some pages were name-cased, while others were lower-cased. Some headings were butted against the left size of the screen, and others had a small margin. Different fonts were used for similar purposes.

Let’s get back to boundary value analysis for a minute. Assume you’re testing an input field that accepts a value from 1 to 100. The obvious boundary tests are 0, 1, 100, and 101. However, there’s another, less obvious, boundary test. Since this value may be stored internally as an integer, a good boundary test is a number too large to be stored as an int.

My UI Testing Checklist has all these ideas, plus plenty more: accented characters, GB18030 characters, different date formats, leap days, etc.. It’s by no means complete, so please leave a comment with anything you would like added. I can (almost) guarantee it’ll lead you to uncover at least one new bug in your project.

 

Click this images to download the UI Testing Checklist

Click this image to download the UI Testing Checklist

Continue reading →

Filed under: Security, Test Planning, Testing Tools | Tagged: , , , , , , | 2 Comments »

Test Estimation Tips (But No Tricks)

Posted on July 9, 2012 by Andrew Schiano

Estimating the time it takes to test a feature is an important, yet often overlooked, skill. If your estimate is too low, it can affect your product’s quality and release date. If your estimate is too high, it can lead to an inefficient use of resources.

The topic of project estimation is complex enough topic to fill entire books. (See what I mean?). But here are a few simple tips that can quickly improve your estimation skills, not to mention your work/life balance and general happiness.

Consider non-functional testing

A common mistake among less-experienced testers is to only consider functional testing when providing an estimate. In this type of testing, software is evaluated against the functional requirements. On many projects, however, non-functional testing can take longer than functional testing. Non-functional testing includes areas such as security, performance, stress, accessibility, and long-haul.

I once badly underestimated my testing effort because I didn’t consider long-haul testing. (This was mostly because I didn’t know what long-haul testing was.) I worked some crazy hours to make sure my oversight didn’t negatively affect my deadline. My extra hours however, did negatively affect my wife’s mood, as she was left by herself many evenings to care for our newborn.

Consider the non-testing tasks

Non-testing tasks include writing your test plan, conducting a test plan review, participating in specification reviews, attending meetings, updating the automation framework, and logging your test cases and results. These tasks can easily take 25% or more of your time. You might want to track the time you spend on these tasks for a project or two to get a good idea of how long they take.

Assume bugs will be found.

All complex software contains bugs. One of the goals of testing is to find these bugs so a decision can be make on whether or not to fix them. If you’re not planning on finding bugs, then you’re essentially planning that your tests will be ineffective. Don’t plan on writing ineffective tests. Account for the time it’ll take to investigate test case failures, reproduce issues, log bugs, follow-up with dev, fight for bugs in triage, validate fixes, and re-run your tests.

Be cautious about giving an estimate before doing your research

One of the biggest mistakes you can make is to offer an early ballpark estimate before doing your due diligence. Your initial guess will greatly influence your final estimate due to a behavior known as anchoring. People have a tendency to “anchor” to their first guess and make small adjustments from there. These adjustments, however, typically under-estimate new information, leading to an inaccurate final estimate.

For example, the always-reliable Wikipedia cites a study in which people were asked to guess the percentage of African nations that are in the United Nations. Those who were asked “Was it more or less than 10%?” guessed lower values (25% on average) than those who were asked if it was more or less than 65% (45% on average).

I was once asked to estimate my test effort during the project kick-off meeting. I had just learned about the project, and had no business giving an estimate. But I was pressed to give a rough estimate, so I guessed two weeks.

When wrote my test plan, I realized my original estimate was too low. I had forgotten to consider some non-functional testing tasks and found the automation framework I planned on using wasn’t reliable. I spoke to my manager and we both agreed to adjust the estimate, but the damage had already been done. My two-week guess had become our anchor. We doubled the estimate to four weeks, but it turned out we should have raised it to six. As a result, my work/life balance took a turn for the worse, and once again, my wife was not happy.

Ask the developers for their estimate

Although it’s not ideal, the reality is that at some point you will be asked to estimate your test work before properly analyzing the project. In these situations, the amount of development work is a good gauge for the amount of test work. The more complex the feature, the longer it takes to both develop and test.

I’ve found that a reasonable rule of thumb is that the test effort is 1 to 2 times the development effort. If this ratio sounds high, consider how many times you gave a test estimate, but then wished you had more time as the project completed. I want to stress, however, that this is just a rule of thumb; it should only be used if you absolutely have to give an estimate before doing your due diligence.

If the developers haven’t yet made their estimate, then you have a stronger case for not yet giving yours. You can’t reasonably be expected to make an accurate test estimate if the developers don’t even know how complex their code will be. If management still insists on an estimate, have them speak to your wife.

Filed under: Test Planning | Tagged: , , , | Leave a comment »