The fine print
"Copyright (c) 2003-2005 The Regents of the University of California.
All rights reserved.
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose, without fee, and without written agreement is
hereby granted, provided that the above copyright notice, the following
two paragraphs and the author appear in all copies of this software.
IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR
DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT
OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF
CALIFORNIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS
ON AN "AS IS" BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATION TO
PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS."
Copyright (c) 2003-2005 Intel Corporation
All rights reserved.
This file is distributed under the terms in the attached INTEL-LICENSE
file. If you do not find these files, copies can be found by writing to
Intel Research Berkeley, 2150 Shattuck Avenue, Suite 1300, Berkeley, CA,
94704. Attention: Intel License Inquiry.
Author: Rodrigo Fonseca
Date Last Modified: 2005/06/07
This document is a guide to a BVR implementation that runs TinyOS. It is therefore developed in the nesC language. I hope this text can help you read the BVR code, run the BVR code, and use it on your own applications. To learn more about the algorithm, and our evaluation of it, I suggest you read our NSDI paper. [put link here]
TestBVRSimple is a (simple) application that wires to BVRRouterC and sends messages back and forth between nodes. If you just want to try something with the code, you can start from this one. It is driven externally through the BVRCommand interface, and logging is done using the Logger interface.
Application (route to coords) \ BVRRouterC BVRStateC UARTLoggerM BVRCommand \ \ / / -------------------------------------- | BVRCommStack | -------------- | FilterLocalComm | LinkEstimatorComm | BVRQueuedSendComm | LinkEstimatorTaggerComm | [ UARTInterceptComm ] (PC only) | GenericCommReallyPromiscuous | BVR specific --------------------------------------- GenericCommPromiscuous TinyOS standard AMPromiscuousThe LinkEstimatorTaggerComm tags all packets with a per source unique (16 bits) sequence number. This info goes in the LEHeader structure. All messages sent through BVRCommStack should follow the BVRRawMessage format, which is to say they should have space for LEHeader as the first 4 bytes of the AM payload. LinkEstimatorComm collects statistics on the receive side, for the different neighbors. It integrates directly with the LinkEstimator module, and drives all changes to that module. LinkEstimatorComm also sends its own periodic packets for reverse link quality estimation. These two components are split because of the queue component: the reverse link estimation packets are also enqueued and retransmitted, while all packets that leave the radio (including multiple retransmissions of the same higher layer packets) receive a unique id.
BVRQueuedSendComm is basically the standard TinyOS QueuedSend, with a few additions for logging queue state and retransmission statistics. It is the module responsible for the "link level" retransmissions when we send unicast packets. It implements a queue of pointers to message buffers, so that the buffers have to belong to the different modules that originate messages. The number of retransmissions is controlled by MAX_RETRANSMIT_COUNT, and is currently set to 5. This module takes no action on incoming packets.
Lastly in the CommStack is FilterLocalComm, which reverts the effect of the promiscuous mode of the lower parts of the stack. It is here so that components that use the stack and assume that they will only receive packets destined for the node.
Eviction is handled separately from quality: after AGE_THRESHOLD periods without hearing any packets from a node, we evict it from the cache. The quality also drops, so it might be replaced before that.
When a new node is added to the cache, it enters a probation period during which it cannot be replaced. This is set by LINK_ESTIMATOR_PROBATION, and was set as the number of link estimation periods one would need to get to within 10% of the true value (assuming 0 as an initial value, and successive inputs of value 1). It thus shold vary with different values of alpha, the weight of the history in the moving average.
Replacement is done if we hear about a new neighbor. What we currently do is that every node that has quality below LINK_ESTIMATOR_REPLACE_THRESH and has past the probation period (meaning it has a sensible estimate of the quality), is up for being replaced, and we pick the one with the lowest quality. The result of this is that there are nodes with quality above the threshold, they will remain in the cache.
To do: change the link estimator replacement strategy such that we keep the k best neighbors, and have n-k slots for probation.
The CoordinateTable module also provides the very important command getNextHops, which returns a list of neighbors that are good next hops for a given packet and destination, ordered by progress weighted by bidirectional quality to the neighbor.
To do: the CoordinateTable interface passes pointers to the entries in the table. Change this to have methods that modify these entries based on the address as the key, similar to what the link estimator does.
All is done by sending and receiving BVRBeaconMsg's, which contain primarily a node's own coordinates (distances to each beacon). They also contain information to allow a node to use such information relative to each beacon in computing its own coordinates: a sequence number for each beacon, and the cumulative quality of the reverse path to each beacon.
This is how we ensure that we have reasonable trees: each node has a parent, and we make sure that when we say we are X hops from the beacon this is along a good quality path. Otherwise we might be considering very poor quality and long links to be 'hops'. It is still an open question what is the best way to count hops when we have 'links' which are not physical wires but this shared, changing wireless medium. We found though that the trees produced by this method are pretty good.
The basic idea then is that if you receive a message from a neighbor saying they are 3 hops from beacon 5, you consider yourself to be 4 hops from beacon 5. Each beacon starts this process by stating distance 0 from themselves, and update a sequence number for each such broadcast they do. A node will update its distance only once per sequence number, this avoids count-to-infinity problems, and loops. A node maintains a parent in each beacon tree, and will occasionally change parents if another parent offers a better quality path to the root. The quality of a path is measured in ETX - expected number of transmissions to get to the root, taking into account the quality of each link along that path and assuming you would do hop-by-hop retransmissions.
There are also details involving when to choose a new parent, in order to avoid instability and flapping, while still reacting to significant changes in connectivity. I encourage you to look into the code as the authoritative source on how that is done, and ask any questions you may have to me.
Lastly, how do we get to know who are the beacons? We have code that assigns beacons at compile time, by node id, and also experimental code, which is not yet released, that performs a beacon election process and does not need to be preconfigured. The static method defines beacons in the contrib/bvr/bvr/topology.h file, specifying an array with whether a node is a beacon, and the beacon id of the node.
In the BVR algorithm, each node has to know how many hops away from each beacon it is. This turns out to be not as simply done as said when we are dealing with the wireless medium of these low power radios. Nodes don't have a binary link to other nodes, in the sense of always being connected or not. Each node will hear a message from another node with a certain probability, and we try to estimate this probability for each pair using the link estimator. One of the consequences is that is not simple to define what a hop is.
One naive way of forming trees is to have the root broadcast messages, and whoever hears these is considered to be one hop away. Whoever hears messages relayed by any of these nodes is then considered to be 2 hops away, and so forth. This does not work well in practice, as you end up using long and very unreliable links because they happened to transmit a message once. The trees that result may not be representative of the topology, and may be poor for routing.
The approach we take in the implementation of BVR is to consider links that are parts of good paths to the root. To this effect we use the ETX metric accumulated along the paths. ETX is the expected numter of transmissions along a link and also along a path, and is given by 1/(p_f*p_r), where p_f is the probability of transmissions success in the forward direction and p_r in the reverse direction. A node can efficiently compute the ETX to the root through a potential parent with with only the parent's own ETX to the root and the ETX on the link to the parent. We use a 1 byte field to express the ETX, which we deemed would have enough precision for networks with diameters around 25 or so hops. When we receive an update from a beacon, through one neighbor, we get how many hops away from the beacon we are, and the ETX estimate in the quality field. When computing the quality to send, we subtract 1, since this transmission is always required for a hop. The quality field only encodes the extra transmissions you would need on the links. We also define ETX_SCALE to work as a fixed precision "shift". For details, look in BVRStateM, functions scaledEtxFromQuality and updateRootBeacon.
It implements the greedy forwarding, fallback forwarding, and scoped flood in the end, if the other modes fail. It does not implement the selective dropping of beacons described in the paper; rather, it always uses the same set of beacons to route on.
The interface to BVRRouterC assumes you know the coordinates of the node you want to talk to, and we have experimental code for a lookup database, that will be released shortly.
Forwarding takes place by calling BVRState.getNextHops, which returns two lists: one of greedy next hops, and another with fallback next hops. The latter currently always contains a single element, the node's parent in the tree of the closest beacon to the destination. The greedy nextHops list is ordered by expected progress: the progress a neighbor makes in the coordinate space, weighted by the quality of the "link" to that neighbor. A similar technique has been shown to be optimal in the case of geographic forwarding, and we experimentally verified that indeed it improves the total number of messages sent when routing. The idea is basically to avoid long, poor quality links.
The scoped flood in the end maintains a cache of messages seen, and a node stops sending messages when it is as far as the ttl of the message. There is a randomized timer to avoid synchronization. The timer hasn't been throroughly studied, and it may be improved upon. We can't do a lot of suppression at first, because we don't know which path will lead to the destination when doing the flood. One idea for making the flood more robust to different densities is to use the same scheme used in the Berkeley PEG demo: a timer that is reset by a random value every time the node hears a neighbor transmit the same message.
MAX_ROOT_BEACONS(c) | Min. TOSH_DATA_LENGTH(m) | BVR_APP_DATA_LENGTH |
<8 | 29 | m-(c+12) |
8 | 30 | 10 |
12 | 42 | 18 |
16 | 54 | 26 |
For running the experiments, we use a couple of Java tools to interact with the motes. We collect a log of the packets sent to the UART by the motes using the class net.tinyos.testbed.TestBedPacketLogger, and send commands to the motes using either a standalone tool, net.tinyos.bvr.BVRCommandInject, or a scripted driver, net.tinyos.bvr.ThroughputTestDriver. So that these tools can simultaneously talk to the motes, we connect all motes to SerialForwarders, and have the tools talk to the SerialForwarders. The tools assume that a mote of id X will be connected to a SerialForwarder that listens on port B+X. We generally use B=9100, such that mote id 85 will be connected to a SerialForwarder at port 9185. The SerialForwarder multiplexes connections to the motes, making it really convenient and flexible to run experiments.
Steps:
For the last step I wrote some perl scripts that parse the logs, but you may want to write a java class that reads the textual format of the logs and converts them to the java message classes created by mig, which would make it easy to get access to all fields from within your program. Shouldn't be hard to do. Other people have taken different approaches for d, e, and f above. For example, you can connect your motes to a database, and then process everything with some java program that understands the messages, or with MatLab.
If you don't have access to a testbed with ethernet or usb connected motes, you can implement a module that implements the Logger interface, but writes the info to the EEPROM, and then can have each mote dump its information from the EEPROM. The difficulty here would be to correlate the logs in time, since each one would be in its own timescale. Maybe if the motes are timesync'd, or you can alternatively extract causal relationships between events from one mote to the other. We don't have code to do this in BVR, though.
*Troubleshooting (some) There are several reasons motes fail to program, or respond. I generally have 15% of them not working for one reason or another. Check that the cables are connected. Try powercycling the mote and the ethernet adapter. Telnet into the EPRB and verify its settings. Reset everything. Try again. If you don't get some mote working, go back and adjust your testbed config file, and repeat, until you get a set of working motes...