Authors: Steve Neuendorffer
Christopher Brooks
Ankush Varma
Shuvra S. Bhattacharyya
The basic infrastructure is implemented in the copernicus.kernel package. The Copernicus class contains a main function suitable for invocation from the command line. The GeneratorAttribute class represents code generation parameters that can be persistently added to a model for unusual configurations of a code generator. The KernelMain class is a base class from which classes for various code generators can be derived. Instances of these subclasses are instantiated and executed in a code generation job.
The Copernicus class itself is invoked to begin code generation from a model. If invoked from the command line, it reads command line arguments to determine various code generation options and an MoML file to load a model from. If invoked via various static methods, code generation options are assumed to be passed in through a GeneratorAttribute. Default code generation options are specified in the Generator.xml file. One of the code generation parameters determines the code generator to execute. The class representing the code generator is loaded through reflection and invoked through the KernelMain base class.
The KernelMain base class provides default behavior for most code generators. It performs static analysis, such as type resolution and scheduling by invoking the Manager.preinitalizeAndResolveTypes() method and then passes control to the specific code generator.
codeGenerator: The code generator to run.
codeGeneratorClassName: The class that is instantiated to execute a particular code generator. This class is expected to be a subclass of ptolemy.copernicus.kernel.KernelMain.
compile: If true, compile the generated code. The default is true.
show: If true, then show the generated code. The default is true.
run: If true, then run the generated code. The default is true.
ptII: The location of the Ptolemy II classes. The default is the value of the ptolemy.ptII.dir Java system property
ptIIUserDirectory: The top level directory to write the code in. The default is the value of the ptII parameter. The code will appear in 'ptIIUserDirectory/targetpath'.
targetPackage: The package to generate code in. The default is the model name
targetPath: The path relative to the ptIIUserDirectory to generate code in. The default is the "cg" subdirectory of the particular code generator.
outputDirectory: The directory that code will be generated in. By default this is the targetPath parameter appended to the ptIIUserDirectory path.
modelPath: The path to the model, including the .xml extension. The modelPath parameter is converted to a URL internally before use.
compileOptions: User supplied arguments to be passed to the code generator. Defaults to the empty string.
javaClassPath: The Java class path, converted to a string.
runCommandTemplateFile: The template file that contains the command to run the generated code.
runOptions: User supplied arguments to be passed to the command that will run the generated code. Defaults to the empty string.
sootDir: The directory that contains the soot jar files. Defaults to the value of the ptII parameter + "/lib"
sootClasses: The location of sootclasses.jar, jasminclasses.jar and the Java system jar (usually rt.jar). The necessaryClassPath parameter may end up duplicating some of the elements of this parameter.
watchDogTimeout: The number of milliseconds that code generation will run for. Defaults to 720000, which is 12 minutes. The watchdog is used to prevent the code generator and the generated code from hanging the nightly build.
output: The filename to redirect the standard output stream of the code generator to. This is used, for example, in the nightly build to provide easily parseable error messages. If the value is not set, then the output will not be redirected.
The Java code generator operates in several phases, and the output of each phase is a partially specialized model. The output from the intermediate phases can be generated by setting the snapshots parameter to be true. The first snapshot consists of self-contained code specialized to the domains in the model. The second snapshot is additionally specialized to the parameter values in the model, while the third is specialized to the structure of the model. The fourth snapshot eliminates all references to Ptolemy named objects in the model, resulting in self-contained code without component interfaces. The final generated code has also been specialized for data types and contains no references to Ptolemy tokens.
One of the goals of the Java code generator was to avoid separate specifications for simulation and code generation wherever possible. The Java code generator operates by transforming Java actor specifications (actor classes) and on Java data type specifications (token classes). In most cases, new actor and token classes will be leveraged transparently by the code generator. Unfortunately, domain specifications are not as easily reused and the Java code generator contains "re-implementations" of domains for code generation. This allows for more efficient code to be generated, at the expense of duplicating aspects of existing Director and Receiver code, and making it more difficult for new domains to be implemented in code generation.
In order for existing actor code to be leveraged by the code generator, it assumes that the code is written according to the Ptolemy style for writing actors. This style assumes naming conventions for the public fields of an actor class that refer to parameters and ports of the actor. The code generator also assumes that the ports and parameters of an actor are created in the class constructor and not modified later. Some actors do not fit these constraints and cannot be used directly in the code generator. Such actor classes cannot be used directly by the code generator, although in some cases we have been able to have the code generator deal specially with such actors. In other cases, the actor class fits the constraints but cannot be effectively specialized using generic techniques. Such actors can also be dealt with specially by the code generator to more effectively generate code.
The Java code generator consists of a large number of individual transformation steps, which will not be described here. These transformation steps are implemented by classes extending the SceneTransformer class, or the BodyTransformer class. Two key points of extensibility are provided for generating domain code and for generating actor code to replace unspecializable actor classes.
Code generation for specific domains is handled by various implementations of the DomainCodeGenerator interface. An implementation of this interface is responsible for generating domain interaction code for a particular composite actor in the hierarchy, including code to invoke the methods of various actors and domain-specific communication structures. Currently, the following domains are handled:
Synchronous Dataflow (SDF): The SDF implementation transforms the SDF schedule into Java code that invokes the actors in a model. Fixed size arrays are generated for communication buffers dedicated to each relation in the model, and communication methods are replaced with circularly indexed addressing into the communication buffers.
Hybrid Systems (HS): The hybrid systems director deals with modal models. Currently, only the subset that is useful in modal SDF models is implemented.
Giotto: The Giotto implementation interfaces directly with the Java output of the Giotto compiler. It generates classes with static methods used for communication by the Giotto compiler and generates a .giotto file that describes the classes implementing the various Giotto tasks. The Giotto compiler compiles this file into a class that implements the Giotto task scheduling model.
Code Generation for actors is handled various implementations of the AtomicActorCreator interface. An implementation of this interface generates a self-contained class for a particular actor. The default implementation of this interface, the GenericAtomicActorCreator class, simply copies the existing actor specification code. Other implementations of the interface deal with generating code for specific actors. Currently the following actors are handled specially:
Expression: The standard implementation of this actor builds a parse tree for the expression and traverses the parse tree to evaluate it at run time. This actor is handled specifically by the ExpressionCreator class for two reasons. Primarily, the parse tree is convenient way of representing an arbitrary expression, but much simpler code can be generated for a specific expression. Secondarily, the parse tree complicates other transformations that specialize communication between actors and data types.
FSMActor: The standard implementation of this actor builds parse trees for every expression in a model, and suffers from the same drawbacks as the Expression actor. The FSMActor also attempts to deal with run-time modifications of the finite-state machine in an efficient manner, which is not necessary in generated code. This actor is handled specifically by the FSMCreator class.
Many actors are not handled specifically, but should be. Here is a short list:
MathFunction: This actor creates and deletes its ports based on a parameter value. In generated code will likely not happen (since the parameter value is not likely to change), but the GenericAtomicActorCreator is not smart enough to deal with ports that are not created in the constructor. It is likely easiest to handle this actor by handling it specially and checking that the parameter value does not change using reconfiguration analysis.
TypeTest: This actor tests the type system, but has no run-time behavior. It is problematic because it iterates over all of the actors in a model, which is currently not supported by the code generation mechanism. It could probably be checked statically and ignored in generated code.
RecordAssembler and RecordDisassembler: These actors iterator over their input and output ports to construct a record. They could either be dealt with specially, or the code generator could be improved to unroll iterators over ports.
ExpressionToToken and ExpressionReader: These actors operate in a similar way to the expression actor, except that the expression is received from an input port. Because of this, code generation will not work. It is not clear how to make this actor work nicely with type specialization.
The code generated from the Java code generator is a set of self-contained Java .class files with a command-line interface. A makefile is automatically generated with a large number of rules for manipulating the generated code. The makefile rules are:
runJava: Run the generated code.
compareAll: Run a series of comparisons between the simulation model, the generated code, and the obfuscated version of the generated code comparing code size, execution speed, and memory usage.
treeShake: Generate a self-contained .jar file containing only code necessary for the generated code. This rule uses reachable method information gained through static analysis in the code generator, if possible.
treeShakeByRunning: Generate a self-contained .jar file containing only code necessary for the generated code. This rule executes the generated code and extracts information from the virtual machine about which classes were loaded at runtime.
runTreeShake: Run the generated code from the self-contained .jar file.
profileTreeShake: Run the generated code from the self-contained .jar file with profiling options to report runtime memory usage. An average of several runs is reported.
treeShakeWithoutCodegen: Generate a self-contained .jar file containing only code necessary for executing the simulation model. This rule executes the generated code and extracts information from the virtual machine about which classes were loaded at runtime.
runTreeShakeWithoutCodegen: Run the original simulation model from the self-contained .jar file.
profileTreeShakeWithoutCodegen: Run the original simulation model from the self-contained .jar file with profiling options to report runtime memory usage. An average of several runs is reported.
obfuscate: Run the Jode obfuscator on the generated code, to minimize the size of the generated .jar file.
runObfuscate: Run the obfuscated version of the generated code.
profileObfuscate: Run the obfuscated version of the generated code from the self-contained .jar file with profiling options to report runtime memory usage. An average of several runs is reported.
gcj: Compile the generated code into a native executable using gcj. Note: this will likely only work for simple models, as the gcj standard Java libraries are far from complete.
Within the Ptolemy II framework, the C code generator takes the class files generated by copernicus.java as input.The C code generator can also be used as a stand-alone Java-to-C compiler to generate C code for arbitrary Java programs.
The Soot framework is used to create a Call Graph of the application. This is a graph with methods as the nodes, and calls from one method to another as directed edges.
At first glance, it seems that the transitive closure of the methods in the main class should represent all methods that can be called. However, this is not so, because the first time the field or method of a class is referenced, its class initialization method is also invoked, and this can reference other methods or fields in turn.
The method call graph also contains an edge from a method to every possible target of method calls in it. The number of such targets can be large for polymorphic method calls. A more sophisticated analysis can trim the method call graph by removing some of the edges corresponding to polymorphic invocations.
We use Soot's Variable Type Analysis (VTA) to perform this call graph trimming. This analysis computes the possible runtime types of each variable using a reaching type analysis, and uses this information to remove spurious edges.
From the analysis mentioned above, the set of all possible required classes, methods and fields (collectively grouped as entities) can be statically computed. We use a set of rules to determine which classes are required.
1. A set of compulsory entities is always required. This includes the System.initializeSystemClass() method, all methods and fields of the java.lang.Object class (since it is the global superclass) and the main method of the main class to be compiled.Interfaces are treated as classes. A worklist-based algorithm can be used to add to the set of required entities until no additional entities can be found by application of these rules. Together, rules 2, 3 and 4 encapsulate all possible dependencies between entities. This makes the set of required entities self-contained.
The further limitations of the current implementation are:
No support for threads.The dashed arrows in the UML diagram represent the coarse-grained code flow in copernicus.c. The entry class is JavaToC when used in stand-alone mode, and Main when copernicus.c is used as a ptolemy code-generation back-end.
JavaToC reads in a Java class file and implicitly converts it to the Soot Jimple format. Then it calls RequiredFileGenerator, MakeFileGenerator and MainFileGenerator.
RequiredFileGenerator uses CallGraphPruner to compute the set of required classes, methods and fields. Then it calls ClassFileGenerator, HeaderFileGenerator and StubFileGenerator on each required class.
ClassFileGenerator creates the .c file containing all the function definitions. Each of these function definitions is created by MethodCodeGenerator. MethodCodeGenerator calls CSwitch on each Jimple statement to find its C equivalent.
HeaderFileGenerator creates the .h file corresponding to the class. This consists of a class-specific C structure for the class (created by ClassStructureGenerator), an instance-specific C structure for the class (created by InstanceStructureGenerator) and various function declarations. MethodListGenerator is the class that "understands" inheritance to create the lists of constructors, inherited methods, new methods, private methods, etc.
StubFileGenerator creates a small "stub" of prototype declarations useful for breaking circular dependencies between classes.
MakeFileGenerator creates a makefile proving rules for compiling the generated C code into an executable.
MainFileGenerator creates a file that contains the C "main" method, which performs initialization functions, wraps the command-line arguments into the C equivalent of a Java string array and passes them to the "java" main method.
In addition, CNames converts Java names into unique legal C names, InterfaceLookupGenerator takes care of resolving interface method invocations, FileHandler provides file I/O utilities, Options stores configuration information, Context handles useful global information, NativeMethodGenerator handles native methods and OverriddenMethodGenerator allows user-defined code to override the compiler.
java -classpath $classpath ptolemy.copernicus.c.JavaToC $classpath <className>
(note that the classpath has to be specified twice)
Move the xml model to c/test/simple
java ptolemy.copernicus.kernel.Copernicus -codeGenerator c <model>.xml
Java requires certain native methods, which are methods implemented in platform-dependent code, typically written in another programming language such as C. The C code generator allows the user to specify C code for the body of any native method. At compile-time, this is integrated with the generated C code, allowing any C native methods to be fully supported. To do this:
1. Find the C name of that method (say f00xx_abc).It is also possible to override the C code generator and write custom C code for a given method instead. To do this:
1. Find the C name of that method (say f00xx_abc).Note that the term overridden in this context does not refer to methods that are overridden through inheritance in Java classes.
To "turn off" code generation for a method, override it without creating code for it in runtime/over_bodies. For an entire class or package, list it in OverriddenMethodGenerator.isOverriddenClass(). This will generate methods with blank bodies and trivial return statements which will return 0 or NULL.
CAVEAT: Make sure that the returned values are not used. Referencing a NULL pointer will cause the executable to throw a segmentation fault.
The Applet code generator takes a model and creates HTML files for use as a web based applet.
The applet generator reads template files that end in .in from $PTII/ptolemy/copernicus/applet substitutes keywords and writes out the files in the destination directory. Users may modify the template files to match their local setup
Making an applet available via the web is somewhat complex because the Java Plugin has two sections, one for Netscape, the other for Internet Explorer, so changes to the htm files must be replicated in both sections. The codebase and the location of the jar files also add to the problems.
If a model is named MyModel, and the user selects foo.bar as the package, then saving the model as an applet will create a directory called $PTII/foo/bar/MyModel and create the following files for that model:
make demo will run appletviewer on the HTML files
An HTML file containing the code necessary to MyModel.xml
The OrthogonalCom model generates prints output to standard out, so when this model is run as an applet, the output will appear in the Java Plugin console. Instead, we generate an applet for the Butterfly model, which will generate display a nice plot. Note that the Butterfly model uses the Expression actor so that while we cannot use deep code generation on the Butterfly actor, we can generate an applet for this model.
cd $PTII/ptolemy/domains/sdf/demo/Butterfly $PTII/bin/copernicus -codeGenerator applet Butterfly.xml
The HTML can be found in $PTII/ptolemy/copernicus/applet/cg/Butterfly.
Usually, one wants to put an applet on a website. Ptolemy applets require jar files for the runtime environment, so the applet code generator will copy the necessary jar files if the value of the ptIIUserDirectory parameter is outside the $PTII directory.
1. If you built Ptolemy II from source, generate Ptolemy II jar files by runningcd $PTII make install2. Create the target directory:
mkdir c:/tmp/ptIIapplet/Butterfly3. Invoke copernicus:
cd $PTII/ptolemy/domains/sdf/demo/Butterfly $PTII/bin/copernicus -codeGenerator applet -ptIIUserDirectory \ c:/tmp/ptIIapplet -targetPath Butterfly Butterfly.xml
Note that the copernicus command should be typed in on one line.
cd $PTII make install2. Create the target directory:
mkdir c:/tmp/ptIIapplet/Butterfly3. Open up the SDF Butterfly Model at $PTII/ptolemy/domains/sdf/demo/Butterfly/Butterfly.xml
c:/tmp/ptapplet
Note that the directory must already exist. If it does not exist, then the default directory will automatically be used.
7. Change the targetPath to the string$modelName8. Change the modelName parameter to
Butterfly9. Hit the Parameters button, which will update the parameters and display their values.
Under Web Start, you may need to add classes to the necessaryClasses parameter so that the necessaryClassPath parameter will get updated with the appropriate jar files and passed to the subprocess that invokes the applet code generator. The reason this is necessary is because Web Start is invoked using a special class loader that accesses separate jar files in the Web Start cache. The applet code generator does not have direct access to the Web Start class loader, so we tell it what classes we need so that they can be added to the class path.
It would be nice if the applet code generator would bundle up the necessary class files in a single jar file so that it was easier to install an applet.The user interface reads its parameters from a moml file, the
default is
$PTII/ptolemy/copernicus/kernel/Generator.xml
.
This file lists the parameters, the initial default value
the documentation for the parameter.
The documentation for a parameter is viewable as a tool tip. Try moving the mouse over the name of the parameter, a tool tip should pop up.
There are quite a few different parameters, and parameters
can refer to each other by name. The values of some parameters,
such as the modelName
and iterations
are derived from the model itself and not actually settable.
There are several buttons:
More Info
Parameters
Generate
compile
, show
and
run
checkboxes determine whether code is generated
shown and run.
Cancel
Clear
When the user hits View -> Code Generator, the model is queried and any necessary updates to the Parameters occurs.
When the user hits the Generate
button, the model is
again queried and any necessary updates occur.
Each of the code generators contains template files
that determine what command is actually run.
For example, the
java code generator uses
$PTII/ptolemy/copernicus/java/runCommandTemplate.txt
to determine what command to run the generated code.
The format of the files is fairly straightforward, parameters are
substituted in and the command is executed.
We use template files
so that the code generator can be run without requiring the
make
program.
In Ptolemy Classic, if you wanted to generate code for a new processor, you had to create a new domain and then populate the domain with new basic blocks that contained codeblocks of code that were generated when the basic block was used. This is a bit of a simplification, but basically it meant that for each new processor, the author had to generate a new Ramp basic block, a new Add basic block etc. This was very time consuming, and tended to have problems, since if a bug was fixed in one Ramp actor, the bug needed to be fixed in other Ramp actors in each domain.
The Adaptive Computing System (ACS) domain was an effort to work around this issue, where the interface to each actor was shared between multiple implementations. This helped make it easier to switch between different target implementations, since the ACS Ramp actor always had the same interface, whereas if there were two Ramp actors in two separate domains, then they might have different port names, which made switching between the domains difficult.
More information about the ACS domain can be found in E. K. Pauer, C. S. Myers, P. D. Fiore, J. M. Smith, C. M. Crawford, E. A. Lee, J. Lundblad and C. Hylands, "Algorithm Analysis and Mapping Environment for Adaptive Computing Systems," Presented at the Second Annual Workshop on High Performance Embedded Computing, MIT Labs, Lexington, MA, September, 1998.
The ACS domain is part of Ptolemy 0.7.2devel, see
http://ptolemy.eecs.berkeley.edu/ptolemyclassic/pt0.7.2/
The Ptolemy Classic style of code generation was used to customize
different actors to take advantage of different features of a
processor. For example, the Motorola 56x FIR filter actor would pick
a different codeblock depending on how the FIR filter was configured.
However, the downside of this approach is that the inter-actor communication tended to consume quite a bit of time, so even with really great actor implementations, we were getting performance hits when data was passed between these actors. It seems to us that looking at the whole model would yield further performance improvements.
In 2000, Jeff Tsay created an initial implementation of Deep code generation as part of his his Masters project: Jeff Tsay, "A Code Generation Framework for Ptolemy II," ERL Technical Report UCB/ERL No. M00/25, Dept. EECS, University of California, Berkeley, CA 94720, May 19, 2000.
There is a shorter summary of Jeff's work at:
Jeff Tsay, Christopher Hylands and Edward Lee, "A Code Generation Framework for Java Component-Based Designs," CASES 00, November 17-19, 2000, San Jose, CA.
Jeff's work was a prototype of how we could do code generation in a different manner, described in the above references and below. Unfortunately, his prototype code was not easy to extend to match changes in the Ptolemy type system. More specifically, the type system is Yuhong Xiong's area of research, and Jeff's work was not easy to extend to deal with ArrayTokens.
http://www.sable.mcgill.ca/soot/
Soot operates on class files by applying a series of transformations that usually do compiler things like common subexpression elimination or loop unrolling. We can add transformations that do things like flattening a model into one class for shallow code generation, or further processing the AST for deep code generation.
The advantage of Soot over Jeff Tsay's work is that by using soot, we do not have to parse java files, and the name resolution of objects is done. In Jeff's code, we spent a lot of time trying to figure out the fully dot qualified name of an object who's base name was 'String'.
Jeff Tsay's work was a proof of concept demonstration on what could be done. We hope that by using Soot, we can generate a stable system for use in a production environment.
Java can take advantage of multiple processors, so in theory, if we use the process domains in Ptolemy II without code generation on a multi processor machine, we should see that each Java thread will be run on a separate processor. However we have not done much work in this area. We'd like to see someone take the PN domain and work on running it on multiple processors and seeing what sort of improvements can be made.
In theory, once we have deep code generation, and we are creating an AST and generating .class files, we can generate code for any processor by either writing a new back end, or using a native Java compiler like gcj.
http://www.jhdl.org
)
jode
$PTII/bin/configure
looks for jode in
$PTII/vendors/jode/1.1.1
, and if it finds it, then
$PTII/bin/jode
can be used to invoke jode.
http://prdownloads.sourceforge.net/jode/jode-1.1.1.jar
and save it as $PTII/ptolemy/vendors/jode/1.1.1/jode.jar
cd $PTII rm config.* ./configure
$PTII/bin/jode
:
cd $PTII/bin make
$PTII/bin/jode ptolemy.kernel.util.NamedObj
$PTII/bin/jode ptolemy.copernicus.java.cg.OrthogonalCom.Main
javap
http://java.sun.com/j2se/1.3/docs/tooldocs/solaris/javap.html
- Shipped with Sun's JDK.
Source Again
http://www.ahpah.com/sourceagain/
- Commercial product
WingDis
http://www.wingsoft.com/wingdis.html
- Commercial product
jad
http://www.geocities.com/SiliconValley/Bridge/8617/jad.html
- Jad home page - used to have binaries only, but now broken.
http://www.geocities.com/zz_xu/jad.html#download
- Mirror site from 1999 - Jad 1.5.7
http://www.javacoffeebreak.com/directory/index(32).html
http://gcc.gnu.org/java/
: JDK1.2 compliant
- Java source code directly to native machine code,
- Java source code to Java bytecode (class files),
- and Java bytecode to native machine code.
http://www.meurrens.org/ip-Links/Java/codeEngineering/blackDown/jolt.html
: initial hack
http://compose.labri.fr/prototypes/harissa
: 1999: JDK1.0.2?
http://www.cs.arizona.edu/sumatra/toba/
: Does not support JDK1.2