...
DESCRIBE DETAIL TABLE output:
name | type | description |
name | String | catalog.database.tableName |
log.kafka.bootstrap.servers | Map | Kafka brokers |
log.retention | Duration | how long changes log will be kept |
file.path | String | File path |
log.kafka.topic | String | topic of Kafka |
watermark | Timestamp | Watermark of the latest written data (If table defines watermark) |
file.format | String | format for file |
bucket | Integer | bucket number |
change-tracking | Boolean | does this table tracking changes |
...
We put necessary configurations in the global configuration to avoid the need for users to configure each individual table.
If users need to configure a table separately, users can also configure it through options.
Key | Default | Type | Description |
table-storage.log.system | kafka | String | Log system. |
table-storage.log.kafka.bootstrap.servers | (none) | Map | Kafka brokers. eg: localhost:9092 |
table-storage.log.retention | (none) | Duration | It means how long changes log will be kept. The default value is from the log system cluster. |
table-storage.log.scan | full | String | Specifies the scan startup mode for log consumer.
|
table-storage.log.pk.consistency | transactional | String | Specifies the log consistency mode for table with primary key.
|
table-storage.log.pk.changelog-mode | upsert | String | Specifies the log changelog mode for table with primary key.
|
table-storage.log.pk.key-format | json | String | Specifies the key message format of log system with primary key. |
table-storage.log.format | debezium-json | String | Specifies the message format of log system. |
table-storage.file.root-path | (none) | String | Root file path. |
table-storage.file.format | parquet | String | Format name for file. |
table-storage.bucket | 1 | Integer | Bucket number for file and Partition number for Kafka. |
table-storage.change-tracking | true | Boolean | If users do not need to consume changes from the table, they can disable Change Tracking. This can reduce resource consumption. |
...