Here we describe the framework in sufficient precision to be useful to framework implementors. We will use the framework to test itself. This leads to circular logic that can be unraveled more easily by the computer than a beginner learning the framework. Beginners might want to ReadMore of this site before continuing here.
Each of the following sections will become a specification document in its own right. Each bullet with a section is to become the FitAcceptanceTests.
http:run.cgi
Parse
A parser builds an in memory model of an html document. The model preserves all of the content while exposing table data to further processing. This allows input to be prepared with a variety of tools.
- word tables
- excel tables
- wiki tables
Tables are sometimes used to format html pages, as is the case with wiki. In this case a unique tag, <wiki>, is used to surround the tables that are to be parsed as tests.
The visible text within a cell is extracted and treated as a string, free of formatting.
- markup tags
- character escapes
- leading and trailing spaces
The parsed tables can be modified in memory and a revised document written.
- cell background color
- cell contents
- additional rows
- additional columns
Check
Expected values specified in cells are tested for equality with actual values extracted from the program under test.
| fat.Equals |
| type | x | y | = |
| boolean | true | TRUE | true |
| integer | 00001 | 1 | true |
| real | 1000 | 1e3 | true |
| string | abc | ABC | false |
A sequence of values can be entered and checked as a single (composite) value.
| fat.Equals |
| type | x | y | = |
| integers | 1, 3, 5 | 01,03,05 | true |
| integers | 1, 3, 5 | 01,05,03 | false |
| booleans | true, true, false | true, false | false |
| strings | a , b , c | a,b,c | true |
Domain values can be constructed from cell contents. Equality comparisons will be subject to appropriate domain rules (i.e. delegated to the domain objects.)
| fat.Equals |
| type | x | y | = |
| date | Jan 1, 1995 | January 1, 1995 | true |
| money | $10000 | $10,000.00 | true |
Floating point numbers are checked to the precision that is normal in the host language. It is possible for a domain object to infer precision from the string representation of expected values. For example, ScientificDouble checks equality to the precision implied by the number of significant digits in a value.
- non-zeros -- 123.45 is 5 digits
- zeros between non-zeros -- 100003 is 6 digits
- zeros in the fraction part -- 12.20 is 4 digits
- but not leading zeros -- 0001000 is 4 digits
- and not exponent digits -- 6.02e23 is 3 digits
- left argument (receiver) controles precision
| fat.Equals |
| type | x | y | = |
| real | 123.45 | 123.449 | false |
| scientific | 123.45 | 123.449 | true |
| scientific | 100003 | 100003.1 | true |
| scientific | 100003.1 | 100003 | false |
| scientific | 12.20 | 12.210 | false |
| scientific | 12.21 | 12.210 | true |
Improperly specified values may throw exceptions which are reported in the cell that contains the invalid number. Improper value detection is only as good as would be expected in the host language. (We are assuming that incorrectly converted values will lead to detected errors elsewhere.)
- xyz is not an integer
- 10000000000000000000000000 may or may not be an integer
| fat.Table |
| fat.Equals | |
| type | x |
| integer | 200 |
| integer | xyz |
| fat.Color |
| white | white |
| white | white |
| white | white |
| white | yellow |
Some character strings have special meaning outside of type conversion.
- blank -- omit check, report value (reported in gray letters)
- error -- expect an error
| fat.Table |
| fat.Divide | | |
| x | y | divide() |
| 100 | 2 | 50 |
| 100 | 2 | |
| 100 | 0 | error |
| 100 | 0 | |
| fat.Color |
| white | white | white |
| white | white | white |
| white | white | green |
| white | white | gray/white |
| white | white | green |
| white | white | gray/white |
The framework looks for parse support in different places.
- fixture
- domain object
- type adapter
The framework will allow fixtures to implement custom parsing.
Fixture
The first cell of a table specifies how that table will be interpreted. Normally this is the name of an objet, a Fixture, that is specifically written for this purpose.
- case Foo != foo
- package alpha.Foo != beta.Foo
By default cells are processed in order, by table, by row within tables, and by cell within rows. Distinct routines handle sequencing (plural) and interpretation (singular).
- doTables -- sequence tables within a document
- doTable -- interpret a table
- doRows -- sequence rows within a table
- doRow -- interpret a row
- doCells -- sequence cells within a row
- doCell -- interpret a row
The default interpretation of a cell is to mark it as ignored.
Cells contain givens or expected results. The fixture is responsible for distinguishing givens from expected results, checking expected results against actuals, and indicating the result with the background annotation as follows.
- white -- no check wanted (blank input)
- green -- right: expected equals actual
- red -- wrong: expected not equal actual
- yellow -- exception: trouble computing actual or comparing result
- gray -- ignore: desired check not possible
Fixtures cooperate to count the number of each result.
- counts accumulate between tables
- counts are summarized as "# right, # wrong, # exceptions, # ignores".
- runner fixtures (e.g. AllFiles) can control accumulation of counts
Fixtures cooperate to accumulate state that can be summarized at any point in the processing of a document.
- state variables
- state lifetime
- state summary
PrimitiveFixture
The primitive fixture does not depend on TypeAdapter. Type specific functions handle all checking.
- checkBoolean
- checkInteger
- checkFloat
- checkString
ColumnFixture
A column fixture distinguishes givens from expected values by the form of the column head.
- name is setter or field
- name() is getter or field
- camel case -- present value ( ) = presentValue()
- deprecate execute()
RowFixture
A row fixture expects all rows to be expected values. The fixture is assumed to have access to actual rows which it pairs with expected rows.
- name is field or getter
- name() is getter
- camel case
Rows are matched by comparing just enough values from the left most columns to get a unique match.
- some given, none actual -- missing row
- none given, some actual -- surplus rows
- one given, one actual -- check remaining columns
- many given, many actual -- match on more columns
Missing and surplus rows are marked as such in their first column
- each counts as one wrong
- surplus rows added to table in gray
ActionFixture
An action fixture distinguishes given from expected values based on keywords in the first column.
- enter -- put given in setter or field
- check -- check value with getter or field
- press -- call action (no check other than for execptions)
- start -- new actor
An actor represents some interface state such as the currently active window. The actor gives meaning to many of the words in the second column (field to be entered and buttons to be pressed).
- actor persists between tables
- initial actor is action fixture (subclass) itself
- start with no argument specifies initial actor
Start could have various meanings depending on the nature of the interface being modeled.
- create a new actor
- return to an existing actor
Other actions, like press, can change the current actor.
- replace the current actor with a new actor
- replace the current actor with another preexisting actor
- restore the previous actor
Subclasses of action fixture can add new action words.
Runner
A runner creates top-level Fixtures and provides them with parsed documents to be annotated.
- top-level fixture is instance of Fixture
A runner retrieves and disposes of documents as appropriate to its environment. The FileRunner reads and writes html files.
- input-file only -- default output
- input-file output-file -- specific output
- input-directory only -- default output in directory
- input-directory output-directory -- default outputs in specific directory
- index.html result summaries
- - is standard in/out
A runner signals the presence of errors to its environment so that subsequent processing can be triggered.
- exit status is sum of wrong and exceptions up to 255
- main is factored to simplify RunnerFixtures
A runner establishes run specific state that may influence loading and/or configuration of the program under test.
| fit.Summary |
| counts |
42 right, 0 wrong, 0 ignored, 0 exceptions |
| input file |
/Users/ward/Documents/Java/fit/Release/Documents/FitSpecification.html |
| input update |
Mon Sep 15 16:47:24 PDT 2003 |
| output file |
/Users/ward/Documents/Java/fit/Release/Reports/FitSpecification.html |
| run date |
Mon Sep 15 16:51:45 PDT 2003 |
| run elapsed time |
0:00.27 |