An Object-Oriented Testing Framework

This is some preliminary material on an object-oriented testing framework.

• Overview and purpose
• Requirements
• The object test
• The collaboration test
• How hierarchical testing works
• Running tests
• Infrequently-asked questions
• Unresolved issues

Overview and purpose

The testing framework is designed to support testing of object-oriented class hierarchies. The languages supported by the test framework are

• [incr Tcl]
• Java

• Individual object tests. In this type of test, a single object is created, called, and queried to verify correct results. This type of test is further sub-divided into:
  • Method tests. In each, a single method is called to verify correct operation.
  • Behavior tests. A series of methods is called to verify correct behavior of a series of state transitions.

• Collaboration tests. These tests construct a set of objects that collaborate with each other, and run them through various interaction scenarios.

Although it would make sense to mimic the class hierarchies of the object/s under test (OUT) in the test suite, I will assume here that the test framework and the tests suites are implemented in plain Tcl. This is purely for the purpose of gaining a wider audience. The test framework is structured using Tcl namespaces: each test "object" in the test suite is implemented as a namespace. "Inheritance" is implemented by importing namespaces. There are two key root objects: one for object testing, and one for behavioral testing. Each of these in turn is subdivided into namespaces for Java and Itcl testing.

Requirements

[I would like to take this piece of software through the new Tycho code rating system, including going so far as to have design and code reviews. To aid the design review, here is my statement of requirements.]

The purpose of the test framework is to provide a reasonably simple but powerful framework for testing object-oriented class hierarchies. It will directly support testing of individual objects and of small sets or clusters of objects.

The test framework will deal properly with the probems of testing class hierarchies. In particular, it will support testing of abstract classes and it will enable tests to be reused in subclasses, conforming to the principles of inheritance.

The test framework will encourage and reward good object-oriented design. It will be flexible enough to work around designs that are not considered "good" in recognition of pragmatic concerns.

The test framework will support Itcl and Java, with the most important of these being Java. To encourage its use in Java installations, it will be written in plain Tcl.

The test framework will generate a report that includes at least the following information for each class or cluster tested:

• A serial number indicating the test run.
• Number of tests executed, passed and failed.
• For those tests that failed, detail information on which test failed and where.
• An annotation on tests that failed that did not fail on the previous run. This is for regression testing, and must be able to ignore "intermediate" runs by test developers.

The object test

Object testing is designed to test a single class. A test object is created by the ::test::class and ::test::abstractclass commands, which take the name of the class under test as the first argument, an optional language specifier, and a series of test declarations.

::test::abstractclass classname ?-language language script

Create a new namespace for testing the class named classname. This command is the same as ::test::class, except that the class will not actually be instantiated and tested. since it is abstract. The instantiation and testing will be performed only on concrete subclasses.

::test::class classname ?-language language script

Create a new namespace for testing the class named classname. The -language option specifies the language in which the class is written, and default to Java. The script is a script that declares the object-level tests for this class.

abstract command arguments...

Declare that the following command is abstract, where command can be one of behavior, constructor, or method. This declaration prevents the test framework from running this particular test command on objects of this class. To activate the test, the subclass that can support the test must use the concrete declaration.

behavior description tests

Initialize the framework for testing a behavior of a single object, and create a single object named $this. The description is a short descriptive string that will appear in the test reports. The tests argument is a script that is executed for this behavior.

concrete command arguments...

Declare that the following command is concrete, where command can be one of behavior, constructor, or method. This declaration allows the test framework to run this particular test command on objects of this class and all subclasses.

constructor description ?script? ?results?

Initialize the framework for testing a constructor of the object. The scripts argument is a script that is executed to create a new object -- this script must set the $this variable to the newly created object. results a script that is executed to check that the object meets a minimum confidence level, and generally contains a series of query commands.

Each time constructor is executed, script becomes the default construction script to use for subsequent tests. If script is not supplied, then description is looked up in the existing constructors, and its script is used to construct objects for following tests. There is always a default constructor called "default" that construct an object with no arguments.

inherit classname ?classname... ?

Declare the class from which this class inherits. This statement allows the test framework to properly test the class hierarchy using the substitution principle.

method methodname tests

Initialize the framework for testing the method given by methodname. There can be more than one call to testmethod within each test object. The tests argument is a script that is executed for this method.

exception message

Compare the given message with an exception produced by the most recent test, and output or log the result. If a test produces an exception and the exception is not matched with this command, then the exception will be reported.

query ?description? result script

Execute script and make its result the result value of the most recently executed test. The description, if supplied, is used to identify the query in the test report (it is normally only supplied if there is more than one query on following a single test.

test description ?result? script

If inside a call to method, create a new object named $this. If inside a call to behavior, the object already exists. Then execute script. The description, if supplied, is a short descriptive string that will appear in the test reports. The result of script is the default result for this test. result, if supplied, is the result against which to compare the return value of this call.

Here is a complete example, showing tests for a directed graph class. The result against which calls on the object under test are compared are highlighted in bold type:

::test::class ::tycho::Digraph -language Itcl {
    inherit ::tycho::AbstractGraph

    constructor "Construct an empty graph" {
	set this [::tycho::Digraph #auto]
    } {
	query {} {$this vertices}
	query {} {$this edges}
    }

    constructor "Construct a small graph" {
	set this [::tycho::Digraph #auto]
	$this type configure vertex -label ""
	$this type configure edge -weight 0
	$this parse {
	    vertex a -label "First vertex"
	    vertex b -label "Second vertex"
	    vertex c
	    vertex aa
	    edge a b
	    edge b c
	    edge a c -weight 3
	}
    } {
	query {a aa b c} {lsort [$this vertices]}
	query {a b a c b c} {$this edges}
    }

    method edges {
	test "Get all edges" {a b a c b c} {
	    $this edges
	}
	test "Get edges by pattern" {a b a c} {
	    $this edges a
	}
    }

    method delete {
	test "Delete single vertex" {
	    $this delete a
	}
	query "Check vertices" {aa b c} {
	    lsort [$this vertices]
	}
	query "Make sure edges not touched" {a b a c b c} {
	    $this edges
	}
	test "Delete non-existent vertex" {
	    $this delete z
	}
	exception {Vertex "z" does not exist}
    }

    behavior "Delete vertex and undo" {
	test "Delete the vertex" {
	    $this delete a
	}
	query "Check vertices" {aa b c} {
	    lsort [$this vertices]
	}
	test "Undo the deletion" {
	    $this undo
	}
	query "Check vertices again" {a aa b c} {
	    lsort [$this vertices]
	}
    }
}

How hierarchical testing works

Hierarchical testing is best explained with a simple example. Suppose we had a CodeComment class, that contained an array of strings. One of its functions is to comment out the comment text that it contains. This operation is generic, in that each line needs to be commented, but it also needs to be specialized by the particular language that we are using.

class CodeComment {
    public variable lines[];
    public abstract String getPrefix();
    public String[] getComment() {
	return lines;
    }
    public void comment() {
	int i;
	for (i=0; i<lines.size; i++) {
	    lines[i] = (String) (getPrefix() + " " + numbers[i]);
	}
    }
}

Note: this is severely broken: we can't compute the result until we have an object, but the test creates the object and runs the test immediately! Hm.... maybe constructor should construct an object, and test won't create a new one if there exists one that hasn't already been used.

::test::abstractclass CodeComment {
    method comment {
	# Now execute the test
        test "Comment out a line" {
	    $this comment
        }
        # Now compute the result we should have got
        set result {}
        foreach line [$this getComment] {
            lappend result "[$this getPrefix] $line"
        }
	# Now compare with the actual result	
	query $result {
            $this getComment
        }
    }
}

class JavaComment extends CodeComment {
    public String getPrefix() {
        return "//";
    }
}

::test::class JavaComment {
    inherit CodeComment
    method getPrefix {
        test getPrefix "//" {
            $this getPrefix
        }
    }
}

The collaboration test

The object-level test is only the first step in testing for a reliable and robust class hierarchy. The next step is to test objects in combination. The technique recommended here is create a separate test object for each identifiable set of collaborating objects. A set of objects that implements a design pattern, for example, is an ideal (and highly visible) candidate for a collaboation test suite. Another way of identifying a suitable set of objects is by functionality: the set of objects needs to perform a certain function of the system need to be tested to see if they do perform it. (Strictly speaking, these are two different kinds of testing, but we just lump them in together.)

Because collaboration relies on the individual objects functioning properly, collaboration tests should only be written when the object-level tests are essentially complete. Because collaboration between object is often defined at an abstract level, the collaboration tests also support the notion of hierarchical testing: A test can be written that can only actually be executed when concrete subclasses become available.

Collaboration tests use a different set of commands to the object tests. The key commands are:

::test::collaboration name ?-language language script

Create a new namespace for testing the collaboration named name. The -language option specifies the language in which the classes are written, and defaults to Java. The script is a script that declares the collbaration tests.

The collaboration script can contain the following commands:

behavior description tests

Initialize the framework for testing a behavior -- that is, a sequence of interactions between the collaborators. The current constructor script is executed to create the collaborating objects. The description is a short descriptive string that will appear in the test reports. The tests argument is a script that is executed for this behavior.

collaborator classname ?option value ...?

Declare that the given class is a collaborator. A series of option-value arguments follows the class name, and can be any of:

-abstract boolean

Declare whether the given collaborator is abstract. If it is, this test suite will only be executed when concrete subclasses of the collaborator become available.

constructor description ?script? ?results?

Initialize the framework for testing a constructor of the object. The script argument is a script that is executed to create a set of collaborating objects. results a script that is executed to check that the created objects meet a minimum confidence level, and generally contains a series of query commands.

Each time constructor is executed, script becomes the default construction script to use for subsequent tests. If script is not supplied, then description is looked up in the existing constructors, and its script is used to construct objects for following tests. There is no default constructor for collaborations.

inherit name

A collaboration can extend another collaboration. This happens when a subclass of one or more of the collaborators add more complex behavior to the collaboration. If a collaboration inherits from a another, then at least one of the collaborator clauses must define a subclass of an inherited collaborator. All of the inherited test will be run as well as any defined within this collaboration.

Running tests

The test functions described in the previous sections only serve to define tests -- they do not execute any tests. The test framework defines the following commands for executing and reporting on tests:

::test::verbose flag

If flag is true, output is generated on the console while the tests are executing. Otherwise the tests are executed silently.

::test::reset

Reset the test suite. All test results accumulated so far are cleared from memory.

::test::report ?option value... ?

Generate a string that contains a report of the test result to date. By default, a useful summary of all tests run since the test framework was loaded or rest is produced. The optional arguments can be used to control the output:

-classes { class }

List one or more classes to include in the report. The special keywords all and last stand for all classes tested and the most recent, respectively. The default is all.

-format format

Select an output format. format can be one of short, default, or long. The generated report includes more or less information accordingly.