...
The field location information is represented as a table of field offsets. This table allows to determine both the offset and the length of each field.
To represent NULL values a tuple may contain a nullmap. Nullmap is skipped if there are no actual NULL values in a given tuple.To be more precise the layout of a tuple looks like this:
It contains only one byte with flags.
Bits 0-1: Size class of the variable-length area.
Bit 2: Set if the nullmap is presentFlag indicating that the size class is not optimal.
Size class encodes the number of bytes used for entries in the offset table. It is encoded like this:
...
NOTE: By the way, the last option works fine for C++ and possibly other languages but for Java it will be hard to support, so it might be excluded.
If a schema has nullable columns then the corresponding tuple may contain a nullmap. The nullmap is skipped if there are no actual NULL values in a given tuple instance.
Nullmap is a bitset that contains N bits, occupies (N + 7) / 8 bytes, where `N` is the number of columns in the schema.
In the bitset if a bit is set to `1` then the respective field is NULL.
The number of elements in the table is equal to the number of columns in the schema.
...
Each element in the table specifies where the corresponding field ends. Or in In other words, for each given field the table stores the offset where the next field starts. The last element in the table specifies the end of the last field and at the same time the end of the whole value area and consequently the end of the entire tuple.
...
A value area contains data for each tuple field one after another as described above. Potentially a field can contain any kind of data. However there are rules how to represent values of specific types as binary tuple fieldsfor representation of common data types there. These rules are specified in the following section. Data types for which there are no specified rules still may be put into a tuple as a Binary
field, but the interpretation in this case such cases is fully up to a particular application.
...
A tuple field is a sequence of bytes. This sequence is interpreted according to the associated data type provided by the tuple schema. There are different encoding rules for different values. For some data types the rules include a very simple compression mechanism. So even if the original type has fixed size it may take different number of bytes when encoded as a tuple field.
The list of supported data types for tuple fields is based on the list of types in IEP-54[1] with few additions. Namely, the additional types are:
Fixed-size values are stored in little-endian byte order. If a value is equal to NULL then it is absent in the value area. This means that in the offset table the corresponding entry is equal to the previous entry. At the same time the corresponding bit in the nullmap is set.
For some Hence for any variable-length field as we find its length from the offset table we can encounter the case when the length is equal to zero. At this point the value has to be disambiguated in this way:
This approach is naturally extended to fixed-size fields. But instead of a zero-length value we interpret it as a value of zero. More specifically:
...
types we can encounter a value with zero length. To distinguish a zero-length variable-length value from a null value we use a special magic byte that denotes an empty value. That is an empty value is encoded as single-byte sequence – 0x80. In turn if the original variable-length value starts with the magic byte 0x80 byte then this byte is doubled.
The Number and Decimal types are never empty, at least one significant byte is always present. The variable-length types that use the magic byte for encoding are as follows:
The list of supported data types is as follows:
Type | Field Size | Description |
---|---|---|
Int8 | 1 | 1-byte signed integer |
Int16 | 1, 2 | 2-byte signed integer, but may occupy less space due to compression mechanism described below |
Int32 | 1, 2, 4 | 4-byte signed integer, but may occupy less space due to compression mechanism described below |
Int64 | 1, 2, 4, 8 | 8-byte signed integer, but may occupy less space due to compression mechanism described below |
Float | 4 | 4-byte floating-point number |
Double | 4, 8 | 8-byte floating-point number, but may occupy 4 bytes if fits into float w/o loss of precision |
Number | variable | Variable-length integer |
Decimal | variable | Variable-length fixed-point number, the scale is determined by the schema |
UUID | 16 | UUID |
String | variable | An utf-8 encoded string |
Binary | variable | Variable-length arbitrary binary data |
Bitmask | variable | Variable-length binary data representing a bit-string |
Date | 3 | A timezone-free date (a year, month, day) |
Time | 4, 5, 6 | A timezone-free time (hour, minute, second, microseconds) |
DateTime | 7, 8, 9 | A timezone-free datetime encoded as (date, time) |
Timestamp | 8, 12 | Number of microseconds since Jan 1, 1970 00:00:00.000000 |
...
(with no timezone) | ||
Duration | 8, 12 | See below |
Period | 3, 6, 12 | See below |
Boolean | 1 | A boolean value (either true of false ) |
All integer values are stored in the little-endian byte order.
For
Uniform use of offset tables for all fields lets us take advantage of variable-length principle for integer fields.
Thus for integer values it is possible to keep only their significant bytes, omitting their high insignificant bytes. That is even if the original data type of a field is INTEGER and should occupy 4 bytes but the actual value is below 128
and above -129
then we can store it just as a single byte and reflect this in the offset table.
...
In SQL standard there are no unsigned integers. So their support may be is omitted for now. But should it become needed for some reason then a compressed unsigned integer must be zero-extended on decompression.
...
Unlike Date, every part is independent from others and can be in range from Integer.MIN_VALUE to Integer.MAX_VALUE.
A single byte containing 1 for the value of true
and 0 for the value of false
.
If the number of fields is N and t is an array that stores a binary tuple we can find the answers for the following:
Does the tuple contain a nullmap?
hasNullmap = t[0] & 0b100;
How many bytes are occupied by the nullmap?
...
:
...
How many bytes are occupied by one offset table entry?
...
What is offset of the value area?
valueBaseOffset = 1 + nullmapBytes + offsetTableBytes;
What is the whole tuple size?
...
In order to build a tuple using minimum possible space it is required to learn two things:
...
what is the total length of all non-null values
...
. After that we can figure out the minimum possible size of the offset table entries.
Thus, generally speaking, building a binary tuple is a two-pass procedure. Sometimes it might be possible to turn this into a single pass (almost) by over-provisioning the allocated storage for the worst case and then fixing it up at the end.
...