This page is primarily for Ptolemy II Developers. Some of the commands
mentioned below are not included in the Ptolemy II distribution.
Contents:
We have included regression tests for most of the Ptolemy II code. Usually,
wherever there is Java file, the tests are in the test
directory.
Running the tests
The tests themselves are written in Tcl, and use
Jacl
which is a 100% Java implementation of a subset of Tcl.
We have included the java::
man page
for use by test suite authors.
We ship Jacl as a jar file called $PTII/lib/ptjacl.jar
.
make tests
will run the tests in the current directory
and any subdirectories.
Writing your own tests
There are two ways to write tests:
- Use Vergil to create tests using the Test actor
- Write tests using Tcl
Using Vergil to write tests
The testing infrastructure will automatically run any MoML models
located in test/auto
directories. (Nowhere do the names
of these MoML files need to be listed in order for them to be run.)
However, said infrastructure has to be re-built in each new
directory containing tests.
To create the infrastructure for a new test directory, do the following:
When you have done all this, the tests in your new test/auto directory ought to run in the nightly build.
The test passes if it does not throw an exception
The Test actor
(located under "more libraries")
can be used to compare the first few results
of a simulation with a known good results. If the comparison fails,
then the test fails.
If a test is in the optional test/auto/knownFailedTests
directory, then it will be marked as a known failure if it fails.
The MoML test file should not use actors from
ptolemy.actor.lib.gui
because these gui actors
will not work during the nightly build when the models are run
without a display.
Using Vergil to write tests is quite a bit easier than writing Tcl
code, but it is much more difficult to handle corner cases and test
for erroneous conditions by writing models. Tcl tests are unit tests,
whereas tests that use models are system tests and may mask unit
test bugs.
Writing Tcl Tests
The test suite infrastructure is based on the Tcl test suite code. The
The file $PTII/util/testsuite/testDefs.tcl
defines the Tcl proc test
.
test
takes five arguments:
- The name of the test, for example:
foo-1.0
The name of the test should strictly follow the format below.
The Tcl tests that come with the Tcl distribution follow a similar
format, so unless there is a strong need to not follow the format,
please stick with what works.
- The test description, usually a single sentence.
- The contents of the test, usually Tcl code that does the action to
be tested. The last line of the contents should return a value.
- The results to be compared against.
- The last argument is optional and determines what sort of test is
being run. The default value is
NORMAL
, which means that
the test should pass under normal conditions. If the value is
KNOWN_FAILED
, then the test is expected to fail, but
eventually will be fixed. By using KNOWN_FAILED
, developers
can mark tests that they know are failing, which will save other
developers from attempting to debug known problems.
Below is a sample piece of code that sources the
testDefs.tcl
file and then runs one test. The code below
has the incorrect value return results to be compared against, so the
test suite properly indicates that the test failed.
if {[string compare test [info procs test]] == 1} then {
source [file join $PTII util testsuite testDefs.tcl]
} {}
test testExample-1.1 {This is the first test example, it does very little} {
catch {this is an error} errMsg1
set a "this is the value of a"
list $errMsg1 $a
} {{invalid command name "this"} {this is NOT the value of a}}
Parts of a test file
Test files should be located in the test
directory.
It is better to have many small test files as opposed to a few
large test files so that other developers can quickly find the tests
for the class they are working with. Usually tests for the class
Foo
are found in the file test/Foo.tcl
Each test file should have the following parts:
- The Copyright
- The code that loads the test system package
if {[string compare test [info procs test]] == 1} then {
source testDefs.tcl
}
Each directory contains a testDefs.tcl
file which
in turn sources $PTII/util/testsuite/testDefs.tcl
. The idea here is that
if the test framework changes, each test file need not be updated.
- A line that the user can uncomment if they want the test system to
produce verbose messages:
#set VERBOSE 1
- The individual tests, which should loosely follow the Ptolemy II
file format standard:
############################################################################
#### Foo
test Foo-1.1 {Test out Foo} {
} {}
There are two types of tests:
- Tests that handle all necessary setup in each individual test.
- Tests that rely on the earlier tests to do setup.
In general, each test file should be able to be run over and over again
in a binary without exiting the binary (it should be idempotent).
It is up to the author of the tests as to whether each individual
test does all the set up necessary. If each test is atomic, then it
makes it easy to highlight the text of an individual test and run it.
If lots of tests are sharing common setup, then using a separate
procedure to do setup might help. On the negative side, atomic tests
usually are longer and have more complicated return results.
Jacl is a 100% Java implementation of a subset of Tcl.
We use Jacl to test Java by writing Tcl code that exercises the Java classes.
Running the tests
To run the all the tests, do cd $PTII; make tests
If you run make
in a test directory that contains
tests written in Tcl for testing Java classes, then the 'right thing'
should just happen.
Writing Tests for Java
Below we discuss some of the details of writing tests in Tcl that test
Java classes.
Simple Example
Jacl allows us to instantiate objects in a class and call public
methods. We use Jacl and the standard Tcl test bed to create tests.
In the example below, we call java::new
to create an
instance of the Java NamedObj
class. We can then
call public methods of NamedObj
by referring to the
Java object handle $n
:
test NamedObj-2.1 {Create a NamedObj, set the name, change it} {
set n [java::new pt.kernel.NamedObj]
set result1 [$n getName]
$n setName "A Named Obj"
set result2 [$n getName]
list $result1 $result2
} {{} {A Named Obj}}
Java Tcl Test Files
It is best if each Java class has a separate Tcl file that contains tests.
The base of the name of the Tcl test file should be the same of the Java
class being tested. The Tcl test file should be located in the
test
subdirectory of the directory where the Java class
is defined.
For example, if we are testing NamedObj.java
, then
the Tcl test file should be at test/NamedObj.tcl
.
We use Sun's
JavaScope test coverage tool as part of our testing environment.
Unfortunately, on November 5, 1999, Sun decided to discontinue support
of JavaScope. In the short term, we are continuing to use JavaScope.
Here's how to review the test suite code coverage:
- Run:
cd $PTII/ptolemy/kernel
make jsall
The jsall
makefile rule does the following:
- To views the code coverage run either
jsreport
,
jssummary
or javascope
:
jssummary -HTML -PROGRESS -OUTFILE=/tmp/summary
jsreport -HTML -PROGRESS -RECURSIVE -OUTDIR=/tmp/report
- To restore the files
back to the original state, run
make jsrestore
.
make jsrestore
JavaScope Details
Flushing the JavaScope database
The JavaScope database must be flushed by hand at the end of a run, or the
code coverage data of the run will not be dumped out to disk.
There are two ways to do this.
- The
IFLUSHCLASS
option will flush out the coverage
information for the current class at the end of each method.
- The
IFLUSH
option will flush out all the coverage
information at the end of each method.
Obviously, using IFLUSH
will take more time than using
IFLUSHCLASS
.
The jsintr
options are set by consulting the following
resources in order.
- system properties - environment variables
-
$HOME/javascope.properties
-
./javascope.properties
in the current directory.
- Options can be embedded in files as comments:
/*jsoptions: ...*/
Embedded comment options can be set more than once in a file. Embedded
comment options are in effect until is the option is changed by another
embedded comment option.
- Command line arguments to
jsinstr
.
The doneTests
proc of
$PTII/util/testsuite/testDefs.tcl
includes a call that will flush the database:
catch {java::call COM.sun.suntest.javascope.database.js\$ flush}
The Ptolemy II documentation is written in HTML. There are several tools
that can be used.
weblint
Weblint tells the user about html errors. Weblint was once obtained from
ftp://ftp.cre.canon.co.uk/pub/weblint/weblint.tar.gz
but has since moved, try
using google.
To run weblint
:
cd $PTII
make weblint
htmlchek
Htmlchek is another tool that tells the user about html errors.
htmlchek
also checks for bad links. The
htmlchek
output is a little hard to read, so we tend to
use weblint
for checking individual files.
htmlchek
was once available at
ftp://ftp.cs.buffalo.edu/pub/htmlchek/
but has since moved, try using google.
The best way to run htmlchek
is to create a sample
distribution, create the files in the codeDoc
directory
and then run htmlchek
- Create the test distribution:
cd /users/ptII/adm/gen-latest; make htmlchek
- Reset
PTII
to point to the test distribution:
setenv PTII /users/ptII/adm/dists/ptII-latest
cd PTII
- Run
make install
. This will make the Itcl HTML docs
twice, which will populate the doc/codeDoc
directories.
You need to make the Itcl HTML docs twice so that the cross references are
correct.
- Run
make htmlchek
The output ends up in five files
All of the references in htmlchekout.HREF
that point
to .html
files should be checked. References to non-HTML
files appear in htmlchekout.HREF
because the non-HTML
files were not included in the list of files that
htmlchek
ran on. One quick way to search all the the *.html
files is
cd $PTII
grep mystring `find . -name "*.html" -print`
Spell checking
$PTII/util/testsuite/ptspell
is a Ptolemy II specific spelling checker.
ptspell
has the following features:
Checking the spelling in all the HTML files can be done with:
cd $PTII
ptspell `find . -name "*.html" -print`
Check the distribution for bogus files
Run the following makefile rules and commands:
-
make realclean
- This will remove the
tclIndex
files and the files in
doc/codeDoc
. The reason to remove the
codeDoc
files is so that we don't ship HTML files for any
classes that have been removed.
-
make install
- This will recreate the
tclIndex
files and the
doc/codeDoc
files.
-
make checkjunk
- Look for files in the distribution that should not be there.
-
adm/bin/chkgifs
- This file looks for gif files that are not used by HTML files
in the distribution.
The parser we use in
$PTII/com/microstar
is a non validating parser.
If you are writing MoML code, you might want to run your
file through a validating parser, below are a few references:
Below are some guidelines on proofreading documentation
- Proofreaders should write their names on the front page
of the document.
- In general, write big, and use a red pen.
- Each page that has a typo should have a mark at the top of the page
so that editors can easily find the typo.
- Proofreading symbols can be found at
http://webster.commnet.edu/writing/symbols.htm
Last Updated: $Date: 2002/05/07 17:12:56 $