OOPSLA '99 Extreme Programming Workshop

Emmet Murphy, ThoughtWorks, LLC

ThoughtWorks' ATLAS project uses some Extreme Programming practices (pair programming, refactoring and others); is coached by two of the best, Ward Cunningham and Martin Fowler; but does not follow the practices rigorously enough to qualify as an "XP project". We have, however, adopted a functional testing approach that may be of interest to the XP community. A business user writes a test scenario and runs it using a Test Harness that provides immediate, detailed feedback on the status of the system. The effects of the tool on our development and testing process have been: 

  1. Test scenarios drive scheduling and progress tracking (similar to the way User Stories drive the Planning Game).
  2. Business users control functional testing.
  3. Functional tests are automated and repeatable.

The ATLAS project develops an accounts receivable system for an equipment leasing company. The system bills customers and processes their incoming payments. A large proportion of the application involves batch processing. Ward Cunningham, Martin Fowler and I created the Functional Test Harness to exercise those batch processes. A business user writes a test scenario as a set of tables in an Excel spreadsheet. The Test Harness reads the spreadsheet, runs the application and compares the computed results with the expected results. A simplified example of the Test Harness output is shown below:

Billing Test 1

Test Status

Missing rowsSurplus rowsChecked valuesIncorrect valuesErrors
1 1 2 1 0

Test Summary

TypeNumberNotes
Billing 1

Gray font = ignored value
Black font = read value

Lease

Lease # CustomerEquipmentCharge typeHow often to billWhen to startFor how longHow much
1 Brookfield Zoo Bear cage Rent Monthly 01/01/1999 12 $300

Events

DateEventNotes
02/01/1999 Run billing Create rental charges for the January and February billing periods

Expected Charges

Create DateDue DateCustomerLeaseDescriptionCharge TypeAmountBilling periodNotes
02/01/1999 01/01/1999 Brookfield Zoo 1 Bear cage Rent $300 01/01/1999 - 01/31/1999 This row is missing from the computed results
02/01/1999 02/01/1999 Brookfield Zoo 1 Bear cage Rent $300 02/01/1999 - 02/28/1999
02/01/1999 - 02/29/1999 		Green = correct
Red = incorrect

Expected value above
Computed value below
Surplus computed rows shown below:
02/01/1999 03/01/1999 Brookfield Zoo 1 Bear cage Rent $300 03/01/1999 - 03/31/1999 Surplus row

Test Environment

Date and timeTest FixtureHostDatabaseBuild
10/07/1999 10:08:00 com.thoughtworks.atlas.ar.test.BillingFixture www.thoughtworks.com ATLAS_AR 123

In the example, Brookfield Zoo rents a bear cage for a year. The test scenario checks that when the billing process is run on February 1, the first two rental periods are billed. The test fails for three reasons:

  1. The  January charge is missing.
  2. The February billing period is incorrect.
  3. The March charge is created too soon (the "surplus row").

The Test Harness (written in java) consists of two components:

  1. The Test Framework reads a test scenario into java objects, evaluates test results and reports the results in HTML. It allows the creation of Test Suites and records a daily history of test results.
  2. A Test Fixture interprets the test scenario, runs the application and instructs the Test Framework how to evaluate the results.

The Test Framework is a reusable package, while Test Fixtures are application-specific. In other words, one extends the Test Framework by writing a Test Fixture.

We use JUnit for unit testing. Developers unit test new functionality before subjecting it to functional tests. When a functional test fails due to a bug in the code, a developer adds a unit test that elicits the failure and then fixes the code.

Users respond enthusiastically to the Functional Test Harness, and it meets XP's goal of business-driven, automated and repeatable functional testing. As Kent Beck says, until a functional test proves the existence of a program feature, the feature doesn't exist.