Expected Errors


I have a comment that is actually referring to p 191 of ThePragmaticProgrammer. It shows a minimal test fixture and then some tests for a square root function. e.g.

  testValue (-4.0, 0.0);
  testValue (-0.0, 0.0);

The test fixture includes this logic in a catch block.

  if (num < 0.0)
    return;
  else
    assert(false);

I think you could see how easily this example could be converted to a test fixture as described here. I haven't looked closely through Ward's examples, but i think there is a flaw in this code that might be in others.

The problem is that the test fixture is deciding that if the argument is negative, then an exception should be expected. In my opinion, this logic/reasoning should not be in the test fixture, but rather in the data.

So i would have:

  testValue (double num, double expected, boolean exception)
  ....

if (exception) return; else assert (false);

And then

  testValue (-4.0, 0.0, true);
  testValue (0.0, 0.0, false);

(or maybe it would be better to reverse the semantics of exception, but you get my idea.) -- BretPettichord


If I understand the lesson here it is that there is a very specific division of responsibility which is as follows.

  • The system under test must be allowed to exhibit errors.
  • The fixture stimulating the system must catch these errors.
  • The specification (tables) says whether error is expected.

We currently catch errors and show them in the cell where they happened. I'm wondering if we need some notation that says don't expect a value here: pass if you get an exception and fail if you don't.

eg.Sqrt
value sqrt()
4 2.0
2 1.4
1 1.0
0 0.0
-1 error
1 error

I've put something like this into the ColumnFixture class so that it is inherited by all its subclasses. Press http:run.cgi to try it here. -- WardCunningham

I like this solution. -- BretPettichord


The reason that i was sensitive to this issue when i read ThePragmaticProgrammer was because this is actually an instance of a more general issue that i've seen really screw up the automated testing framework at one client.

Consider the tests to be SpecificationByExample. This principle i want to emphasize is that the behavioral specification should be contained in the data table, not in the fixture. The fixture should do the minimum possible to allow the behavior to be specified elsewhere. In this case, the issue was regarding specifying the error behavior. -- BretPettichord


Another alternative would be to add a method to the fixture that answers the name of the exception thrown, if any.

  public String sqrtError () {
    try { sqrt(); }
    catch (Exception e) { return e.getName(); }
    return "";
  }

This allows a check to be sure the proper exception is thrown. This may or may not be of interest to the test.


Beyond the name, we might want to be able to check the results of Exception.getMessage(), since that String sometimes turns into a message seen by the user.

getMessage() can return either null or a String. But I think you'd only care to specify a message if you expected a String. So we need a way to know that we got a null when we expected a String, but we don't need a way to specify that we expect a null. I think. -- BrianMarick

 

Last edited September 12, 2002
Return to WelcomeVisitors