Home » Columns, Featured, Master's Notebook

Software Testing: A Rewarding Career

1 December 2010 6,186 views 4 Comments
This column is written for statisticians with master’s degrees and highlights areas of employment that will benefit statisticians at the master’s level. Comments and suggestions should be sent to Keith Crank, the ASA’s research and graduate education manager, at keith@amstat.org.

Contributing Editor
GoodwinGreg Goodwin earned both his bachelor’s in mathematics and statistics and his master’s in statistics from the University of Vermont. He works as an analytics software tester in the Advanced Analytics Division at SAS.

Do you use statistical software in your job? Have you ever considered software testing as a job for a professional statistician? After nearly 20 years of testing SAS/STAT software, I can assure you that software testing is a challenging and rewarding career.

The goal of a statistical software tester is to verify that the software functions correctly and produces valid results. The mantra of a successful tester is (to paraphrase President Ronald Reagan), “Don’t trust and verify!” Testers build suites of test programs and benchmark the results, including graphical displays. A SAS test suite has thousands of procedure calls to validate numeric results, verify functionality of statements and options, and check for proper error handling. These suites are run frequently during the development cycle and any benchmark changes are investigated. A thorough test suite detects unexpected side effects of updates to the code.

SAS software runs on many platforms, and test suites are run on each. Because machine precision differs with hardware architecture, slight numerical differences arise across platforms, especially in iterative computations. The tester, with assistance from the developer and platform support person, determines whether benchmark deviations result from acceptable platform differences or from real problems.

Greg Goodwin tests software with Bob Derr at SAS.

Greg Goodwin tests software with Bob Derr at SAS.

When testing a SAS procedure, we must consider an enormous number of combinations of statements, options, input data sets, and models. The standard checklist of issues to cover also includes missing value patterns, formats, number of observations, number of levels of classification variables, scale of the data, covariate specifications, and BY-group processing.

We focus testing on the most likely usage patterns of the software. However, we also test for unanticipated or pathological usages to prevent meaningless results and to exit with an appropriate error message. It is easy to get carried away testing these unusual situations. Once, after I reported a particularly unrealistic scenario, the developer threatened to change the error message to “Greg, is that you?”

Software is validated in a variety of ways. Results from a new procedure are cross-checked against equivalent results from previously tested software, when they are available. Simulations are performed when closed-form solutions don’t exist, or are used as checks when computations require much time or memory. We verify against results in text books, journal articles, and scientific websites. We also perform consistency checks within a procedure. For example, the results obtained by setting a frequency variable equal to one for every observation should match the results obtained without specifying a frequency variable.

When similar results are not available from previously tested software, we verify numeric results by writing independent validation programs, typically with the SAS/IML interactive matrix language. This is often the most creative aspect of numerical validation, and it requires an in-depth understanding of the computational techniques.

I chose statistics as my college major during my senior year in high school. The deadline for college applications was looming and I wanted a declared major. I enjoyed math, I had just completed a short section on statistics in math class, and I happened to read an article that said the job outlook for statisticians was bright. All signs pointed in one direction!

There are several reasons why I enjoy working in a software development environment. I have the opportunity to work alongside PhD statisticians who are experts in their respective fields. My group fosters a team-oriented approach, where testing is valued and my input regarding features under development is carefully considered. The need to learn about new statistical topics keeps my job fresh and demanding. And I am always thrilled when I find a problem during development.

The skills needed to succeed as a software tester include attention to detail, organizational ability, creativity, persistence, programming ability, a good statistical background, the ability to work independently, and the ability to interact and communicate effectively with developers. Advanced coursework and applied experience are critical because awareness of underlying statistical concepts and practical usage helps a tester know how to verify results. After starting at SAS, I took several graduate courses at The University of North Carolina at Chapel Hill, which expanded my knowledge in categorical data analysis, general linear models, and generalized linear models.

From the many highs of finding problems during development to the occasional low of a customer-reported problem, my job is never boring. Continuous updating of the software with new features and methods means a never-ending supply of new things to test and new areas to learn. Testing will always be an essential part of software development, and I believe it offers a challenging and fulfilling career for a statistician.

1 Star2 Stars3 Stars4 Stars5 Stars (7 votes, average: 4.71 out of 5)
Loading...

4 Comments »

  • Rick Wicklin said:

    This is an insightful article! Testers of statistical software might be interested in joining the ASA Section for Statistical Programming and Analysts (SSPA): http://www.amstat.org/sections/sspa/index.htm

    I’m glad you find SAS/IML software useful in your work. SAS/IML programmers might want to check out a blog especially for them: http://blogs.sas.com/iml

  • Laurie Jackson said:

    Nice article, Greg!

  • Betsy Enstrom said:

    What a fantastic article. It could not have come from a more perfect software tester. Greg, you have been a great instructor, coworker, and role model for so many.

  • Kelly Fellingham said:

    Greg, Super article, and hope you know this developer has appreciated all the tips you’ve shared with me for my testing challenges.