Arlington (TX) - A group of NIST researchers, in combination with University of Texas researchers, are developing an open-source tool which they believe will enable software developers to catch software errors more quickly. It’s called combinatorial testing, and promises to squash the bugs faster, and with fewer errors.
The researchers examined a wide array of software bugs and resulting crashes. Their research gave heed to the age-old adage which states, most software bugs are the result of simple errors, not complex ones. Taking this uncovered reality into consideration, the team began analyzing the number of dependent variables which could result in an error. This began with something called "pairwise testing", which looks at two variables in a program. The team then progressed until they were studying six. According to the release, "Their technique resembles combinatorial chemistry in which scientists screen multiple chemical compounds simultaneously rather than one at a time."
The example that’s given relates to word processing. Suppose there are 10 different settings, superscript, bold, italics, etc. If a program has a bug when three specific ones are set, then doing a hard-code test of all possible three-combinations would require 1,024 tests. However, by looking at the fact that you’re testing groups of three in the set of 10, only 13 tests are required.
This new software program examines the software variables and, based on the number of variables, generates sets of tests which can be run along those same lines using up to six variables. It generates fewer tests because it’s testing the combination of things which have been statistically shown to be the most common cause of software errors. Rather than testing every single possible condition exhaustively, the team believes that this interaction between a few common variables can result in a better capture rate of errors before the software is released, specifically in web-commerce software.
The team is going to open-source the project and release it next year. They are seeking beta-testers right now, though no links were given.