A project can expect to have dozens or hundreds of files testing specific features in a stand-alone way. We may wonder if there are any interactions between features that will cause any of these tests to fail. AllCombinations explores this. It is a fixture that enumerates cases by drawing one file at a time from each of a number of lists; running each in turn, and then repeating the drawing and running for all possible combinations.

For the purpose of illustration we've created a number of files that contain small tests of the CalculatorExample. We've categorized the files as *magnitudes, signs* and *functions,* which enter values, changes signs, and compute functions, respectivly.

eg.AllCombinations | |

Documents/AllPairs/magnitude/*.html | |

Documents/AllPairs/sign/*.html | |

Documents/AllPairs/function/*.html | |

#1 | |

180+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [1] |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#2 | |

180+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [2] |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#3 | |

180+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#4 | |

180+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#5 | |

180+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#6 | |

180+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#7 | |

30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [3] |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#8 | |

30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [4] |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#9 | |

30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#10 | |

30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#11 | |

30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#12 | |

30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#13 | |

360+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [5] |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#14 | |

360+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [6] |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#15 | |

360+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#16 | |

360+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#17 | |

360+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#18 | |

360+30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

#19 | |

90-30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [7] |

cosine.html | 0 right, 1 wrong, 0 ignored, 0 exceptions [8] |

#20 | |

90-30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

change-sign.html | 0 right, 0 wrong, 0 ignored, 1 exceptions [9] |

sine.html | 0 right, 1 wrong, 0 ignored, 0 exceptions [10] |

#21 | |

90-30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 0 right, 1 wrong, 0 ignored, 0 exceptions [11] |

#22 | |

90-30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

multiply.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 0 right, 1 wrong, 0 ignored, 0 exceptions [12] |

#23 | |

90-30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

cosine.html | 0 right, 1 wrong, 0 ignored, 0 exceptions [13] |

#24 | |

90-30.html | 1 right, 0 wrong, 0 ignored, 0 exceptions |

no-change.html | 0 right, 0 wrong, 0 ignored, 0 exceptions |

sine.html | 0 right, 1 wrong, 0 ignored, 0 exceptions [14] |

The number of cases will as large as the product of the size of each list of tests. The searches in the table find four magnitues, three signs and two functions.

- 4 magnitudes X 3 signs X 2 functions = 24 cases

If the tests are intended to interact, as ours are here, then some care is required to be sure that checks in the tests are independent of this interaction. No checks at all my be suitable if we are only testing that there are no exceptions thrown. Or the simplest checks may be adaquate so long as features are tested well in isolation. It may be convenient to run a simulation, a so called *oracale*, along side the application to generate expected results.

**Implementation**

The fixture inherits path expansion from AllFiles, but captures these by overrinding *doRows(Parse, List)* and saving the list of files in a list of lists. Then, at the end of *doTable*, the fixture gets to work computing and running combinations.

See source.

We had a little trouble setting up the test data for this example so we wrote this littleeg.ArithmeticColumnFixture | ||

x | sin() | cos() |

0 | 0.0000 | 1.0000 |

30 | 0.5000 | 0.8660 |

60 | 0.8660 | 0.5000 |

90 | 1.0000 | 0.0000 |

120 | 0.8660 | -0.5000 |

150 | 0.5000 | -.8660 |

180 | 0.0000 | -1.0000 |

fit.Summary | |

counts | 72 right, 14 wrong, 0 ignored, 0 exceptions |

counts run | 42 right, 6 wrong, 0 ignored, 8 exceptions |

input file | /Users/ward/Documents/Java/fit/Release/Documents/AllCombinations.html |

input update | Mon Sep 15 16:47:23 PDT 2003 |

output file | /Users/ward/Documents/Java/fit/Release/Reports/AllCombinations.html |

run date | Mon Sep 15 16:51:05 PDT 2003 |

run elapsed time | 0:01.01 |

Last edited November 4, 2002