THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Unlike DataStream API, Table API / SQL requires converting byte arrays from and to Row/RowData. Certain mapping rules are required to ensure correctness and backward compatibility of such conversion.
TBase & Row Type Mapping
In order to support FlinkSQL workload reading from the Kafka topic, we proposed following data type mapping from Thrift Type System to Flink Row type system. We are in favor of debugging and user readability as we map enum to string type.
...
publicTType() {
}
}
TBase Field to Row Field Index Matching
Handling tbase payload and converting to Row type, Flinks needs to have deterministic mapping from each field in thrift payload to specific row index. We propose using ASC sorted thrift_id indexes (marked in brown below) when we map from tbase to row and vice versa. If a field is UNSET, in order to avoid UNSET fields causing mismatches, we use NULL as a placeholder.
...
Note, from runtime performance consideration, we propose having a type to sort field information cache.
Row Field to TBase Field index matching
We propose a reverse approach above during row-to thrift payload conversion. If row arity is smaller than tbase schema fields, we propose adding null as placeholder.
...
Xtruct3 string_thing = “boo” changed = 0 i32_thing = unset i64_thing = -1 | Row <”boo”,0,null, -1, null> |
Handling Nested Field in Row
It’s common to have thrift schemas highly nested and keep growing as more product features are added.In order to properly decode/encode thrift payload, we propose a recursive converter from nested thrift struct to Flink Row field. Based on each sub struct class schema type mapping along with field index mapping, this recursive converter could handle very sophisticated nested fields (more than 7 levels deep, 24k characters of schema string) thrift schema
...
- numFailedConvertRowDataRecord: The number of messages failed to convert to specific rowdata
- numFailedConvertThriftRecord: The number of rows failed to convert to thrift binary payload
Schema Compatibility
Our proposal leverage thrift protocol handles adding new fields without changing thrift id sequence scenarios backward and forward compatibility. One caveat is when the thrift enum adding new value not defined in the old thrift schema will break the job running with the old thrift schema.
...