Status
...
Page properties | |
---|---|
|
...
...
JIRA: <TODO>
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
name | type | description |
name | String | catalog.database.tableName |
log.system | String | the log system |
log.kafka.bootstrap.servers | Map | Kafka brokers |
log.retention | Duration | how long changes log will be kept |
file.path | String | File path |
log.kafka.topic | String | topic of Kafka |
file.format | String | format for file |
bucket | Integer | bucket number |
change-tracking | Boolean | does this table tracking changes |
...
Key | Default | Type | Description | ||||
table-storage.log.system | kafka | String | Log system. Now only Kafka in the MVP. | ||||
table-storage.log.kafka.bootstrap.servers | (none) | MapString | Kafka brokers. eg: localhost:9092 | ||||
table-storage.log.retention | (none) | Duration | It means how long changes log will be kept. The default value is from the log system cluster. | ||||
table-storage.log.scanconsistency | fulltransactional | String | Specifies the scan startup log consistency mode for log consumertable.
| table-storage.log.consistency | transactional | String | Specifies the log consistency mode for table.
|
table-storage.log.changelog-mode | upsertauto | String | Specifies the log changelog mode for table with primary key.
| ||||
table-storage.log.key-.format | json | String | Specifies the key message format of log system with primary key. | ||||
table-storage.log.format | debezium-json | String | Specifies the message format of log system. | ||||
table-storage.file.path | (none) | String | Root file path. | ||||
table-storage.file.format | parquet | String | Format name for file. | ||||
table-storage.bucket | 1 | Integer | Bucket number for file and Partition number for Kafka. | table-storage. | change-trackingtrueBoolean |
If users
...
need to
...
If users need to configure a table separately, users can also configure it through options without "table-storage." prefix, for example:
...
In addition to session options that can be configured individually for each table by removing the prefix, there are also some options that can be configured individually only for tables, they are the options that affect reading and writing:
Key | Default | Type | Description |
log.scan | full | String | Specifies the scan startup mode for log consumer.
|
log.scan.timestamp-mills | (none) | Long | Optional timestamp used in case of "from-timestamp" scan mode. |
change-tracking | true | Boolean | If users do not need to consume changes from the table, they can disable Change Tracking. This can reduce resource consumption. |
Bucket
The record is hashed into different buckets according to the primary key (if have) or the whole row (without primary key):
...
- Key:
- Without primary key: key is null.
- With primary key: key is json format by default. This is controlled by 'log.key-.format'.
- Value: Use debezium-json to storerecord with or without declaration primary key by default. This is controlled by 'log.format'.
...