Apache Kylin : Analytical Data Warehouse for Big Data
Page History
...
Some properties are not listed because they are not for Kylin user.
Basic configuration
| Property | Default | Description | Since |
|---|---|---|---|
kylin.snapshot.parallel-build-enabled | |||
kylin.snapshot.parallel-build-timeout-seconds | |||
kylin.snapshot.shard-size-mb | |||
kylin.storage.columnar.shard-size-mb | |||
kylin.storage.columnar.shard-rowcount | |||
kylin.storage.columnar.shard-countdistinct-rowcount | |||
kylin.storage.columnar.repartition-threshold-size-mb | |||
Advanced configuration
| Property | Default | Description | Since |
|---|---|---|---|
| kylin.engine.submit-hadoop-conf-dir | null | Please refer to Read-Write Separation Deployment for Kylin 4.0 . | 4.0.0 |
kylin.engine.spark.cache-parent-dataset-storage-level | NONE | 4.0.0 | |
kylin.engine.spark.cache-parent-dataset-count | 1 | 4.0.0 | |
kylin.engine.build-base-cuboid-enabled | true | 4.0.0 | |
kylin.engine.spark.repartition.dataset.after.encode-enabled | false | Global dictionary will be split into several buckets. To encode a column to int value more efficiently, source dataset will be repartitioned by the to-be-encoded column to the same amount of partitions as the dictionary's bucket size. It sometimes bring side effect, because repartitioning by a single column is more likely to cause serious data skew, causing one task takes the majority of time in first layer's cuboid building. When faced with this case, you can try repartitioning encoded dataset by all RowKey columns to avoid data skew. The repartition size is default to max bucket size of all dictionaries, but you can also set to other flexible value by this option: 'kylin.engine.spark |
.repartition. |
dataset.after.encode. |
num' | 4.0.0 |
kylin.engine.spark.repartition.dataset.after.encode.num | 0 |
| See above | 4.0.0 |
Spark resources automatic adjustment strategy (experimental feature)
| Property | Default | Description | Since |
|---|---|---|---|
kylin.spark-conf.auto.prior | true | For a CubeBuildJob and CubeMergeJob, it is important to allocate enough and proper resources(cpu/memory), including following config entries mainly: |
|
|
|
|
|
When `kylin.spark-conf.auto.prior` is set to true, Kylin will try to adjust above config entries according to: |
|
|
|
But user still can choose to override some config via `kylin.engine.spark-conf. |
| ` in Cube level . Check detail at How to improve cube building and query performance | 4.0.0 | |||||||||
kylin.engine.spark-conf. | null | User can choose to set spark conf of Cube/Merge Job at Cube level. | 4.0.0 | |||||||
kylin.engine.driver-memory-base | 1024 | Driver memory(spark.driver.memory) is auto adjusted by cuboid count and configuration. kylin.engine.driver-memory-strategy will decided some level. For example, "2,20,100" will transfer to four cuboid count ranges, from low to high, as following:
So, we can find a proper level for specific cuboid count. 12 will be level 2, and 230 will be level 4. Driver memory will be calculated by following formula :
| 4.0.0 | |||||||
kylin.engine.driver-memory-maximum | 4096 | See above. | 4.0.0 | |||||||
kylin.engine.driver-memory-strategy | 2,20,100 | See above. | 4.0.0 | |||||||
kylin.engine.base-executor-instance | 5 | 4.0.0 | ||||||||
kylin.engine.spark.required-cores | 1 | 4.0.0 | ||||||||
kylin.engine.executor-instance-strategy | 100,2,500,3,1000,4 | 4.0.0 | ||||||||
kylin.engine.retry-memory-gradient | 4.0.0 |
Global dictionary
| Property | Default | Description | Since |
|---|---|---|---|
kylin.dictionary.detect-data-skew-sample-enabled | |||
kylin.dictionary.detect-data-skew-sample-rate | |||
kylin.dictionary.detect-data-skew-percentage-threshold |
Overview
Content Tools
ThemeBuilder
Apps