I got called into a conference call today because of an emerging question a client had about combination testing.
They wanted us to test their online survey, an application which consisted of several questions, most of which had an array of answers from which the respondent could pick. At the end of the survey, the respondent gets a score.
One section of the survey had 40 questions. Each question had anywhere from 2 to 28 possible answers. This made a total of 23 quintillion combinations to test.
So what to do when they only have a few weeks to release?
Enter a heuristic combination method called all-pairs. All-pairs is a mathematical technique that takes a table of options and pairs an option in one column with each option in each of the other column.
This approach is useful for flushing out the risk bugs that exist when two options are paired.
There are two FREE tools I know about that will do this kind of analysis: all-pairs tool from James Bach and PICT, a free tool from Microsoft.
Using both tools, I found that the 23 quintillion options were paired to just 673 test cases. The tools produce a nifty Excel-readable table of test cases, consisting of rows for the tester to follow — telling them which answers to select for each question.
The important point to note when using all-pairs is that it is a heuristic — a fallible mehtod for solving a problem — aka “a rule of thumb.” As such, it should be told to stakeholders that it’s not perfect, but it can help expose certain risks quickly.