...
- connector metadata has no prefix
- formats will have a prefix using the factory identifier
- key/value formats are prefixed with `key.` and `value.`
Note: We will use a relaxed data type validation (such as `LogicalTypeCasts#supportsAvoidingCast`) to make timestamp handling easier. So both TIMESTAMP and TIMESTAMP WITH LOCAL TIME ZONE will work in DDL.
Kafka
| Key | Data type | Read/Write | Notes |
|---|
topic | STRING | r | We don't allow writing to different topics for now. Maybe we will allow that in the future via a property. |
partition | INT | r | We don't allow writing to different partitions for now. Maybe we will allow that in the future via a property. |
headers | MAP<STRING, BYTES> | r/w |
|
leader-epoch | INT | r |
|
offset | BIGINT | r |
|
timestamp |
TIMESTAMP(3) [WITH LOCAL TIME ZONE]| BIGINT | r/w | Directly forward the underlying type. Do not try to abstract it into a TIMESTAMP. |
timestamp-type | STRING | r | ['NoTimestampType', 'CreateTime', 'LogAppendTime'] |
Debezium
| Key | Data type | Read/Write | Notes |
|---|
debezium-json.schema | STRING | r | Pure JSON string, can be handled with Flink's built-in JSON SQL functions |
debezium-json.timestamp |
TIMESTAMP(3) [WITH LOCAL TIME ZONE] | | BIGINT | r | Directly forward the underlying type. Do not try to abstract it into a TIMESTAMP. |
r |
debezium-json.source | MAP<STRING, STRING> | r | It seems that this is always a property list. So we can make it available as a map for easier access. |
Reading and writing from key and value
...