Expert Testers

I have been working as a Software Design Engineer in Test for a little over six years now and I wanted to share some of my experiences with security automation.  I have worked on several different on-premise and cloud-based products, the most recent being a ratings & recommendations service for the Sky Player on Xbox, Xbox.com, and the service for the Xbox Companion application.

Security automation is not all that different from other forms of test automation.  It can be used on code that is still under development as well as existing code that may have changed.

The Microsoft Security Developer center is a great resource for testers, developers, and project managers.  It has getting started guides, tools, articles, tips, and more.  Please check it out.  The Trustworthy Computing site is a good resource as well.

For security automation, feel free to start automating what you can and then iterate and improve from there.  I think that you will find that once the automation is written that it will save your team time and help find bugs.

Here are some ideas for getting started.

• CAT.NET: CAT.NET is a static analysis tool that helps find Cross-Site Scripting issues, amongst others.  It can be run from a cmd window.  So, one approach to automating this process would be to do the following: 1) create C# console project, 2) have that code accept a drop path as an argument to the tool, 3) iterate through each .dll and/or .exe file of interest, 4) save the files in a directory of your choosing.  One could probably use PowerShell or other tools as well.  You could also consider having this run as a post build step.
• FxCop: FxCop can also help pinpoint security issues, using its security rules.  It also has a command-line option called FxCopCmd.  So, using a batch file, PowerShell, or other means one could automate this as well.  This could be a good candidate for running as a post build step also.  It is also possible to have FxCop run as a pre-checkin step.
• BinScope: This is another useful security tool that analyzes binaries for SDL compliance.  It can be launched from a cmd window and the results can be piped to an XML file.  Again, a batch file or PowerShell script could be used to automate this and then also be added to a post build step set.

Using this sort of automation has saved me a considerable amount of time in the past when I was acting as a “security champion” for my team, as opposed to running the tools manually.  Currently, the Xbox.com team has CAT.NET and FxCop setup as part of the build process.  By doing this they are able to catch issues early that were caused either by code changes or additions.  It takes some time to set it up though as sometimes there are false positives or known warnings that can be suppressed if appropriate.  Also, in my current role I have developed internal security automation that is relevant and helpful to my team.

Security automation has a lot of possibilities.  I hope that you find it a helpful addition to your testing toolbox.  Cheers.

Filed under: Security | Tagged: automation, security | Leave a comment »