10  Nil Tokens

Null or missing tokens are common in analytical systems like R and SAS where they are used to handle sparsely populated data sources. In database parlance, missing tokens are sometimes called null tokens. Since null is a Java keyword, we use the term "nil". Nil tokens are useful for analyzing real world data such as temperature where the value was not measured during every interval. In principle, an as yet unimplemented function such as average() method could properly handle nil tokens - when the average() method sees a nil token, it should be ignored. Note that this can lead to uncertainty. For example, if average() is expecting 30 values and 29 of them are nil, then the average will not be very accurate.

If an operation such as add(), divide(), modulo(), multiply(), one(), subtract(), zero() or their corresponding "reverse" operations includes a nil token, then the output is nil. If one of the arguments for isCloseTo() or isEqualTo() is nil, then the method returns false. Methods that return a nil token return a nil token with a specific type so that type safety is preserved. The following tokens have NIL values defined: ArrayToken, BooleanToken, ComplexToken, DoubleToken, FloatToken IntToken, LongToken, ShortToken, StringToken, Token, UnsignedByteToken. There is no nil token for the various matrix tokens because the underlying matrices are java native type matrices that do not support nil.

The expression language defines a constant named nil that refers to the Token.NIL field. The cast() expression language function can be used to generate references to the NIL fields of the other classes. For example, "cast(int, nil)" will return a reference to the IntToken.NIL field.