The Tax Man


As I write this, Tax Day in the United States is just behind us. This is the day when individuals are required by law to settle their tax bill with the US Federal Government, and in many cases with their state governments. Filing your taxes can get complicated, and income tax preparation is a large industry in the United States, with revenue around ten billion dollars according to IBISWorld. That’s ten billion dollars spent annually, helping people cope with the hassle, the complexity, and the general all around unpleasantness of paying taxes.

In the past, software testing could feel like paying taxes and being a software tester could start to feel like working for the IRS. There are forms to fill out. There are deadlines to be met. And there’s process, process, process and at the end of the day it feels like people only notice when something goes Horribly Wrong. But there’s a way to change all that.

Do you know how to get people to appreciate and understand paying taxes? Demonstrate a clear value for their investment, be flexible, and be data-driven.

Deliver Measurable Value

How many times this week did you run a test that didn’t find a bug in the last year? Be honest.

Running tests that don’t find bugs is not, generally, a great way to deliver value. It’s tempting to confuse activity with value and it’s nice to feel busy, but can you imagine explaining how you spent your time to the CEO of your company? It would go something like this:

CEO: So, what have you done this week?

You: Well, I ran a bunch of tests.

CEO: I see. Tell me more about how that improves our product or our profitaiblity?

You: Well, most of the tests I run don’t find bugs. We just run them because we’ve always run them.

[Cricket sounds]

In a conversation with your CEO, your boss, or even your mom you should be able to clearly and distinctly describe what you did to deliver value to the organization. Imagine this conversation instead:

CEO: So, what have you done this week?

You: I prevented thousands of dollars in revenue loss.

CEO: Pray tell, how did you to this?

You: I looked at the numbers from our website and I saw that traffic was down. I talked to Larry in development and found out that we’d just changed the font on the front page, about the same time traffic dropped. I worked with him and we put together an experiment where we tested the new font with some users and the old font with some other users. Turns out users hate the new font. We reverted the change and traffic’s gone up. We’ve put a process in place to do experimentation on all our big changes now.

CEO: I see. Tell me more about your ideas.


Be Flexible

In the past, we kept our tests in long documents, like giant Word files or long Excel spreadsheets. Then we created specialized software to manage the test cases. We had checklists. Lots of checklists.

The funny thing was, it could be possible to hit every box on the checklist and still deliver mediocre software. It was also possible for the checklist to slow down product development, without having anything measurable to show for it. How many times have you had to reject a change because the test process couldn’t accommodate it in time for your product deadline? Again, be honest. I bet you can think of at least one great change that couldn’t make it into your product because your change management system or your test pass couldn’t move fast enough to get the change in before the deadline.

There are some software development activities that require very rigorous, very cautious change management either because of regulatory requirements or because of risk to life or property. It’s very likely that you do not work in one of those areas. Even if you do there are probably places in your process that could stand some scrutiny and some adjustment.

At the end of the day you are not shipping your checklist. Go through the stages of your process and ask yourself if they’re really adding value, or if they’re only there because they’ve always been there. Then cut everything out of the process that doesn’t deliver real, measureable value to your users or your organization.

Be Data-driven

There are tons of tools available for getting data back from your application. Amazon has published a framework for doing A\B testing and analytics, and Facebook has Airlock. There are frameworks available for .NET, JavaScript, and pretty much any other platform that you need.

But, you protest, your product is written in COBOL and distributed via floppy disk to a small group of users in Thule, Greenland and McMurdo Base, Antarctica. There’s no way to be data-driven without writing my own framework from scratch and I don’t have time for that!

To this I say: horsefeathers. Being data-driven isn’t about using a framework about a buzzword. It’s about going out and looking at the real world (which is where your users live) and trying to understand how they’re using your software and what could change to make life better for them. If you don’t have a framework for doing A\B testing or getting user data back directly from your application, consider doing user interviews or surveys. The important thing is that you look at real users doing their real work and don’t just rely on results you got from your manual or automated testing.

To a certain extent, being data driven can be like answering a Fermi question. Start with a question that you need an answer to, for example “Do my users like the new font?” Then figure out what you already know and what tools you have at your disposal that can help you learn more. Frameworks are nice and they can make information easier to gather, but they aren’t the only way for you to go into the real world and get data from real users.


I don’t think there’s anybody who truly enjoys paying taxes. There are, however, tons of people who enjoy testing software. If you’re one of these people, and if you want your users and your dev peers to respect your work, then deliver measurable value, be flexible and be data-driven. If you don’t want to do these things, then don’t be surprised when your collleagues greet you with about as much enthusiasm as they greet an audit from the IRS.

2 Responses

  1. It’s a common sentiment but incorrect to say “if you don’t find new bugs, you’re not adding value.”
    TIP and analytics are great, but you still have to ensure that the business logic works correctly. To do that, you need well-written checks, and you need to run them often, and lack of new bugs is itself a significant value to the project.
    “Check” is a more accurate way of describing a test that is automated, because if you take a manual test and automate it, the value changes radically from the original test. A person running a test sees many things that would be missed if the test were automated.
    If you don’t run checks, you don’t know if your business logic still works correctly with the last change set in the code, and that’s risky. To avoid the risk you need a good set of well-written checks and you need to run them often. Mostly the checks won’t find bugs, but they have very significant value.
    See MetaAutomation for more information on this (blog, and book to publish this summer)

  2. I don’t disagree. The point is that you should be able to articulate _why_ you’re running the test and _why_ the test has value. To do this, you have to be able to articulate the cost of running the test versus the cost of failure. (We covered this in an earlier entry on this blog back in February.)

    If you’re running a test that hasn’t found a bug since 1974 _but_ that test proves that the world be plunged into anew Ice Age following your latest dev checkin then you’re providing value– and you should be able to say so.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: