Embedding graph recognition capability in the whiteboard editor introduces another level of complexity. The interesting issue is how to mix sketch and graph models in one page, because unrecognized sketches must still be displayed on the screen. Our initial approach is to create a symbol for it and add it to the sketch model when a gesture is drawn. If the gesture is recognized as a valid graph element (e.g. a node), we will create a new node and add it to the graph model. The corresponding gesture symbol will be removed from the sketch model and will be annotated as a property of the node, so that the node's render it can build the appropriate visual representation based on the symbol. We came up with several alternative designs, which are given below. In the short term, we are using the first approach, given below, because it works and is easiest to implement.
The first design is shown below. A SketchGraphPane views two models, a sketch model and a graph model. The SketchGraphProcessor processes events from the recognition engine and modifies the models accordingly. It performs the operation that has been described in the previous paragraph.
The second design (below) is more extensible than the first one. We have a canvas pane (views a page) that contains
different viewers, each of which views a specific type of data structure. This way, it is easy to extend a page
to include other types of models, so that many different diagram types can co-exist on a page.
The issue with the second design is how to "overlay" the display of the models. So the third design is to put each model on a different layer. The layers are transparent so that we can see all of the models at the same time. However the challenge here is that since a model does not know about other models, certain operations on a model do not have that information to make good judgements. For example, when we apply a graph layout algorithm to a graph data structure, the algorithm can do a better job if it knows about other sketches on the canvas that are not part of the graph, so that it can try to layout the graph in such a way that the nodes are not hidden behind other sketches.
Another approach is to keep everything that's ever drawn in the sketch model. If a symbol is recognized as a node, a node will be created, annotated with the symbol, and added to the graph model. However, the symbol will not be removed from the sketch model. This way, symbols have a place "to live" and in fact the sketch can be reinterpreted by another application viewing this model. This seems to be the most promising approach, but there are still some design issues to be worked out.