Urban Sim

Sister

 This page describes the UrbanSim project's initial happy and positive experiences with the framework. UrbanSim is featured in this site's SampleApplications. You can start by browsing their current test results at their site.

The UrbanSim project uses the FIT framework to facilitate communication between our urban modelers (domain experts) and our developers. We wanted a scheme where the acceptance tests were easy to read, easy to write, and accessible to the modelers. We originally tried writing our acceptance tests as JUnit tests, but found the barrier to entry (downloading source code, reading Java, etc) high enough that the modelers never got around to reading or writing tests.

So we looked into the FIT framework. Specifically, we wanted a suite of tests, written as documents, in a hierarchy that matched our existing data and model documentation hierarchy. (The root of our documentation is http://www.urbansim.org/docs/urbansim2/.) We wanted our first test documents to read as:

Start with a set of good data stored in database X. Alter tables Y and Z to contain bad data. Run the data consistency checker and it should report errors A, B, and C.

We started by writing an HTML document in the desired format (using FrontPage 2000 to make it easy to manipulate the tables). We use nested HTML tables:

Notice that the table reads exactly as the desired prose. After processing, the table is annotated with green (or red) and with explanations of the errors:

Then we set out to subclass the necessary FIT classes to make this work. It was quite simple:

  1. First we extended the Parse class to parse nested tables.
  2. Then we created a DirectoryRunner to run all the tests in a single directory.
  3. Then we extended the FileRunner to use our NestedTableParse rather than the default Parse.
  4. Then we created a ConsistencyCheckerFixture as something like an ActionFixture. It recognizes certain special rows, such as "use this database" and "alter these tables". We reused our previous test code that scanned the consistency checker output log and extracted all the error messages.
  5. Our ConsistencyCheckerFixture handles expected errors either in altered tables or listed at the end of the table.
  6. We extended our ConsistencyCheckerFixture to add additional rows to the table if a run encountered unexpected error messages.
  7. We also wrote a JDBC driver to read a combination of a real database and HTML tables.
The whole installation and subclassing and integration with our build system (http://trondheim.cs.wash ... ?page=superindex) took a little less than a day. The tests and the results are available on our website http://www.urbansim.org/docs/urbansim2/ and are shipped with each release of the system http://www.urbansim.org/download/index.shtml.
I didn't document the process that first evening: I just sent an email to the developers telling them to use this new framework for writing tests. But it turns out that they didn't need any documentation because the test format was self describing and easy to use. So the developers went to town creating new tests.

Later that week, Ward demo'd the run.cgi feature of writing tests on the wiki, which I liked enough to implement a similar scheme for our pages. Our pages are CVS source controlled rather than being in a wiki, so we implemented a simple httpd server in Perl which, when given a page with the "?run.cgi" annotation, will run the test through the FIT framework and return the result.

The trickest part of setting the whole FIT framework up was extending Parse to handle nested tables. Other than that, it was quite easy to use.

See source for fixtures and the run script.

For all the rest of the details, download the UrbanSim source code (http://www.urbansim.org/download/index.shtml) and look at the urbansim.tests.fit package.

Neat! Nit: "green is good" but "Consistency checking aborted because errors have been found while checking the base data..." is green? --PaulChisholm

 
Last edited February 8, 2003
Return to WelcomeVisitors