Apache Kylin : Analytical Data Warehouse for Big Data
Page History
| Part | Link | Description |
|---|---|---|
| Build Engine | Build Engine Configuration | |
| Query Engine | Query Engine Configuration | |
| Job Engine | Job Engine Configuration | |
| Metastore | Metastore Configuration | |
| Misc Part | Rest Part Configuration | |
| Hadoop Configuration | ||
| Logging Configuration | Logger Configuration |
颜色标记：
...
Kylin原有的配置项
...
kylin.env
...
DEV
...
QA, PROD, LOCAL
...
The environment of Kylin. DEV will turn on some dev features, QA and PROD has no difference in terms of functions.
...
kylin.env.hdfs-working-dir
...
/kylin
...
Working folder in HDFS, better be qualified absolute path, make sure user has the right permission to this directory
...
kylin.env.zookeeper-base-path
...
/kylin
...
kylin zk base path
...
kylin.env.zookeeper-is-local
...
false
...
Run a TestingServer for curator locally
...
kylin.env.zookeeper-connect-string
...
sandbox.hortonworks.com
...
Connect to a remote zookeeper with the url, should set kylin.env.zookeeper-is-local to false
...
kylin.env.hadoop-conf-dir
...
/etc/hadoop/conf
...
Hadoop conf folder, will export this as "HADOOP_CONF_DIR" to run spark-submit. This must contain site xmls of core, yarn, hive, and hbase in one folder
...
kylin.env.hdfs-metastore-bigcell-dir
...
kylin.env.hdfs-metastore-bigcell-dir
...
kylin.env.zookeeper-base-sleep-time
...
3000
...
kylin.env.zookeeper.zk-auth
...
digest:ADMIN:KYLIN
...
kylin.env.zookeeper-acl-enabled
...
false
...
kylin.job.zookeeper-monitor-interval
...
30
...
kylin.job.log-dir
...
/tmp/kylin/logs
...
kylin.job.use-remote-cli
...
kylin.job.remote-cli-port
...
22
...
kylin.job.remote-cli-hostname
...
kylin.job.remote-cli-username
...
kylin.job.remote-cli-password
...
kylin.job.remote-cli-working-dir
...
kylin.job.allow-empty-segment
...
true
...
kylin.job.max-concurrent-jobs
...
10
...
kylin.job.sampling-percentage
...
100
...
kylin.job.dependency-filter-list
...
"[^,]*hive-exec[^,]*?\\.jar" + "|"
+ "[^,]*hive-metastore[^,]*?\\.jar" + "|" + "[^,]*hive-hcatalog-core[^,]*?\\.jar"
...
kylin.job.notification-enabled
...
false
...
kylin.job.notification-mail-enable-starttls
...
false
...
kylin.job.notification-mail-port
...
25
...
kylin.job.notification-mail-host
...
kylin.job.notification-mail-username
...
kylin.job.notification-mail-password
...
kylin.job.notification-mail-sender
...
kylin.job.notification-admin-emails
...
null
...
kylin.job.retry
...
0
...
kylin.job.retry-interval
...
30000
...
retry interval in milliseconds...
kylin.job.retry-exception-classes
...
kylin.job.sampling-hll-precision
...
14
...
kylin.job.scheduler.default
...
0
...
kylin.job.scheduler.priority-considered
...
false
...
kylin.job.scheduler.priority-bar-fetch-from-queue
...
20
...
kylin.job.scheduler.poll-interval-second
...
30
...
kylin.job.scheduler.safemode
...
false
...
kylin.job.scheduler.safemode.runnable-projects
...
kylin.job.error-record-threshold
...
0
...
kylin.job.use-advanced-flat-table
...
false
...
kylin.job.advanced-flat-table.class
...
kylin.job.tracking-url-pattern
...
kylin.job.metadata-persist-retry
...
5
...
kylin.job.cube-auto-ready-enabled
...
true
...
kylin.job.cube-inmem-builder-class
...
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder
...
kylin.job.execute-output.max-size
...
10485760
...
kylin.metadata.url
...
kylin_metadata@hbase
...
The metadata store in hbase
...
kylin.metadata.sync-retries
...
3
...
metadata cache sync retry times
...
kylin.metadata.sync-error-handler
...
MockupErrHandler
...
kylin.server.mode
...
all
...
query, job
...
Kylin server mode, valid value [all, query, job]
...
kylin.server.cluster-servers
...
localhost:7070
...
List of web servers in use, this enables one web server instance to sync up with other servers.
...
kylin.server.query-metrics-percentiles-intervals
...
60, 360, 3600
...
kylin query metrics percentiles intervals default=60, 300, 3600
...
kylin.server.host-address
...
localhost:7070
...
get server rest address
...
kylin.server.cluster-name
...
kylin.server.init-tasks
...
kylin.server.sequence-sql.workers-per-server
...
1
...
kylin.server.sequence-sql.expire-time
...
86400000
...
kylin.server.query-metrics-enabled
...
false
...
kylin.server.query-metrics2-enabled
...
false
...
kylin.server.auth-user-cache.expire-seconds
...
300
...
kylin.server.auth-user-cache.max-entries
...
100
...
kylin.server.external-acl-provider
...
kylin.server.cluster-servers-with-mode
...
kylin.storage.hbase.owner-tag
...
whoami@kylin.apache.org
...
Optional information for the owner of kylin platform, it can be your team's email, Currently it will be attached to each kylin's htable attribute
...
kylin.storage.url
...
hbase
...
The storage for final cube file in hbase
...
kylin.storage.hbase.cluster-fs
...
HBase Cluster FileSystem, which serving hbase, format as hdfs://hbase-cluster:8020, Leave empty if hbase running on same cluster with hive and mapreduce
...
kylin.storage.hbase.compression-codec
...
gzip
...
default compression codec for htable,snappy,lzo,gzip,lz4
...
kylin.storage.hbase.namespace
...
default
...
kylin.storage.hbase.cluster-hdfs-config-file
...
kylin.storage.hbase.region-cut-gb
...
0.1
...
The cut size for hbase region, in GB. E.g, for cube whose capacity be marked as "SMALL", split region per 10GB by default
...
kylin.storage.hbase.min-region-count
...
1
...
kylin.storage.hbase.max-region-count
...
500
...
kylin.storage.hbase.hfile-size-gb
...
2
...
The hfile size of GB, smaller hfile leading to the converting hfile MR has more reducers and be faster. set 0 to disable this optimization
...
kylin.storage.hbase.run-local-coprocessor
...
false
...
kylin.storage.hbase.coprocessor-mem-gb
...
3.0
...
kylin.storage.partition.aggr-spill-enabled
...
true
...
kylin.storage.partition.max-scan-bytes
...
3L * 1024 * 1024 * 1024
...
kylin.storage.hbase.coprocessor-timeout-seconds
...
0
...
kylin.storage.hbase.max-fuzzykey-scan
...
200
...
kylin.storage.hbase.max-fuzzykey-scan-split
...
1
...
kylin.storage.hbase.max-visit-scanrange
...
1000000
...
kylin.storage.hbase.gtstorage
...
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC
...
kylin.storage.hbase.scan-cache-rows
...
1024
...
kylin.storage.hbase.region-cut-gb
...
5.0
...
kylin.storage.hbase.max-scan-result-bytes
...
5 * 1024 * 1024
...
kylin.storage.hbase.compression-codec
...
none
...
kylin.storage.hbase.rowkey-encoding
...
FAST_DIFF
...
kylin.storage.hbase.block-size-bytes
...
1048576
...
kylin.storage.hbase.small-family-block-size-bytes
...
65536
...
kylin.storage.hbase.endpoint-compress-result
...
true
...
kylin.storage.hbase.max-hconnection-threads
...
2048
...
kylin.storage.hbase.core-hconnection-threads
...
2048
...
kylin.storage.hbase.hconnection-threads-alive-seconds
...
60
...
kylin.storage.hbase.replication-scope
...
0
...
kylin.storage.clean-after-delete-operation
...
false
...
kylin.storage.project-isolation-enable
...
true
...
kylin.storage.limit-push-down-enabled
...
true
...
kylin.storage.default
...
2
...
kylin.storage.hbase.table-name-prefix
...
KYLIN_
...
kylin.web.timezone
...
GMT+8
...
Display timezone on UI,format like[GMT+N or GMT-N]
...
kylin.web.help.length
...
4
...
kylin.web.help.0
...
start|Getting Started|http://kylin.apache.org/docs/tutorial/kylin_sample.html
...
kylin.web.help.1
...
odbc|ODBC Driver|http://kylin.apache.org/docs/tutorial/odbc.html
...
kylin.web.help.2
...
tableau|Tableau Guide|http://kylin.apache.org/docs/tutorial/tableau_91.html
...
kylin.web.help.3
...
onboard|Cube Design Tutorial|http://kylin.apache.org/docs/howto/howto_optimize_cubes.html
...
kylin.web.export-allow-admin
...
true
...
kylin.web.export-allow-other
...
true
...
kylin.web.hide-measures
...
RAW
...
kylin.web.default-time-filter
...
0
...
kylin.web.cross-domain-enabled
...
true
...
kylin.web.dashboard-enabled
...
false
...
kylin.engine.default
...
2
...
kylin.engine.mr.config-override.
...
kylin.engine.mr.mem-hungry-config-override.
...
kylin.engine.mr.uhc-config-override.
...
kylin.engine.mr.base-cuboid-config-override.
...
kylin.engine.mr.reduce-input-mb
...
500
...
kylin.engine.mr.reduce-count-ratio
...
1.0
...
kylin.engine.mr.min-reducer-number
...
1
...
kylin.engine.mr.max-reducer-number
...
500
...
kylin.engine.mr.mapper-input-rows
...
1000000
...
kylin.engine.mr.max-cuboid-stats-calculator-number
...
1
...
set 1 to disable multi-thread statistics calculation
...
kylin.engine.mr.cuboid-number-per-stats-calculator
...
100
...
kylin.engine.mr.per-reducer-hll-cuboid-number
...
100
...
kylin.engine.mr.hll-max-reducer-number
...
1
...
kylin.engine.mr.build-dict-in-reducer
...
true
...
kylin.engine.mr.yarn-check-status-url
...
null
...
kylin.engine.mr.yarn-check-interval-seconds
...
10
...
kylin.engine.mr.use-local-classpath
...
true
...
kylin.engine.mr.uhc-config-override.mapreduce.reduce.memory.mb
...
500
...
kylin.engine.mr.uhc-config-override.mapred.reduce.child.java.opts
...
-Xmx400M
...
kylin.engine.mr.yarn-check-interval-seconds
...
10
...
Time interval to check hadoop job status
...
kylin.engine.mr.max-reducer-number
...
5
...
kylin.engine.mr.uhc-reducer-count
...
3
...
for test
...
kylin.engine.mr.lib-dir
...
kylin.engine.spark-conf.
...
getSparkConfigOverride
...
kylin.engine.spark.additional-jars
...
kylin.engine.spark.rdd-partition-cut-mb
...
10.0
...
kylin.engine.spark.min-partition
...
1
...
kylin.engine.spark.max-partition
...
5000
...
kylin.engine.spark.storage-level
...
MEMORY_AND_DISK_SER
...
kylin.engine.spark.sanity-check-enabled
...
false
...
kylin.engine.spark-fact-distinct
...
false
...
kylin.engine.spark-uhc-dictionary
...
false
...
kylin.engine.spark-cardinality
...
false
...
kylin.engine.spark.output.max-size
...
10485760
...
kylin.engine.spark-dimension-dictionary
...
false
...
kylin.engine.spark-create-table-enabled
...
false
...
kylin.engine.flink-conf.
...
kylin.engine.flink.additional-jars
...
kylin.engine.flink.partition-cut-mb
...
10.0
...
kylin.engine.flink.min-partition
...
1
...
kylin.engine.flink.max-partition
...
5000
...
kylin.engine.flink.sanity-check-enabled
...
false
...
kylin.engine.livy-conf.livy-enabled
...
false
...
kylin.engine.livy.backtick.quote
...
kylin.engine.livy-conf.livy-url
...
kylin.engine.livy-conf.livy-key.
...
kylin.engine.livy-conf.livy-arr.
...
kylin.engine.livy-conf.livy-map.
...
kylin.query.enable-dict-enumerator
...
false
...
kylin.query.calcite.enumerable-rules-enabled
...
false
...
kylin.query.calcite.reduce-rules-enabled
...
true
...
kylin.query.convert-create-table-to-with
...
false
...
kylin.query.calcite.extras-props.
...
kylin.query.calcite.add-rule
...
kylin.query.calcite.remove-rule
...
kylin.query.enable-dynamic-column
...
false
...
kylin.query.skip-empty-segments
...
true
...
kylin.query.disable-cube-noagg-sql
...
false
...
kylin.query.stream-aggregate-enabled
...
true
...
kylin.query.max-limit-pushdown
...
10000
...
kylin.query.force-limit
...
-1
...
kylin.query.scan-threshold
...
10000000
...
kylin.query.lazy-query-enabled
...
false
...
kylin.query.lazy-query-waiting-timeout-milliseconds
...
60000
...
kylin.query.project-concurrent-running-threshold
...
0
...
kylin.query.max-scan-bytes
...
0
...
kylin.query.max-return-rows
...
5000000
...
kylin.query.translated-in-clause-max-size
...
1024 * 1024
...
kylin.query.derived-filter-translation-threshold
...
20
...
kylin.query.badquery-stacktrace-depth
...
10
...
kylin.query.badquery-history-number
...
50
...
kylin.query.badquery-alerting-seconds
...
90
...
kylin.query.timeout-seconds-coefficient
...
0.5
...
kylin.query.badquery-persistent-enabled
...
true
...
kylin.query.transformers
...
kylin.query.transformers
...
kylin.query.cache-threshold-duration
...
2000
...
kylin.query.cache-threshold-scan-count
...
10240
...
kylin.query.cache-threshold-scan-bytes
...
1024 * 1024
...
kylin.query.security-enabled
...
true
...
kylin.query.cache-enabled
...
true
...
kylin.query.ignore-unknown-function
...
false
...
kylin.cache.memcached.hosts
...
kylin.query.segment-cache-enabled
...
false
...
kylin.query.segment-cache-timeout
...
2000
...
kylin.query.segment-cache-max-size
...
200
...
kylin.query.access-controller
...
null
...
kylin.query.statement-cache-max-num
...
50000
...
kylin.query.statement-cache-max-num-per-key
...
50
...
kylin.query.statement-cache-enabled
...
true
...
kylin.query.max-dimension-count-distinct
...
5000000
...
kylin.query.timeout-seconds
...
0
...
kylin.query.pushdown.enabled
...
false
...
kylin.query.pushdown.update-enabled
...
false
...
kylin.query.schema-factory
...
org.apache.kylin.query.schema.OLAPSchemaFactory
...
kylin.query.pushdown.runner-class-name
...
kylin.query.pushdown.runner.ids
...
kylin.query.pushdown.converter-class-names
...
{org.apache.kylin.source.adhocquery.HivePushDownConverter}
...
kylin.query.pushdown.cache-enabled
...
false
...
kylin.query.pushdown.jdbc.url
...
kylin.query.pushdown.jdbc.driver
...
kylin.query.pushdown.jdbc.username
...
kylin.query.pushdown.jdbc.password
...
kylin.query.pushdown.jdbc.pool-max-total
...
0
...
kylin.query.pushdown.jdbc.pool-max-idle
...
8
...
kylin.query.pushdown.jdbc.pool-min-idle
...
0
...
kylin.query.security.table-acl-enabled
...
true
...
kylin.query.escape-default-keyword
...
false
...
kylin.query.realization-filter
...
null
...
kylin.query.signature-class
...
org.apache.kylin.rest.signature.FactTableRealizationSetCalculator
...
kylin.query.cache-signature-enabled
...
false
...
kylin.query.flat-filter-max-children
...
500000
Configurations new added
...
kylin.engine.spark.build-class-name
...
org.apache.kylin.engine.spark.job.CubeBuildJob
...
kylin.engine.spark.task-impact-instance-enabled
...
true
...
kylin.engine.spark.task-core-factor
...
3
...
kylin.engine.driver-memory-base
...
1024
...
kylin.engine.driver-memory-strategy
...
{"2", "20", "100" }
...
Auto adjust the memory of driver
...
kylin.engine.driver-memory-maximum
...
4096
...
kylin.engine.persist-flattable-threshold
...
1
...
kylin.engine.spark.cluster-info-fetcher-class-name
...
org.apache.kylin.cluster.YarnInfoFetcher
...
kylin.engine.spark.merge-class-name
...
org.apache.kylin.engine.spark.job.CubeMergeJob
...
kylin.engine.max-retry-time
...
3
...
kylin.engine.retry-memory-gradient
...
1.5
...
kylin.engine.retry-overheadMemory-gradient
...
0.2
...
kylin.engine.max-allocation-proportion
...
0.9
...
kylin.engine.base-executor-instance
...
5
...
kylin.engine.executor-instance-strategy
...
100,2,500,3,1000,4
...
kylin.engine.submit-hadoop-conf-dir
...
kylin.snapshot.parallel-build-enabled
...
true
...
kylin.snapshot.parallel-build-timeout-seconds
...
3600
...
kylin.snapshot.shard-size-mb
...
128
...
kylin.storage.provider
...
org.apache.kylin.common.storage.DefaultStorageProvider
...
kylin.storage.columnar.shard-size-mb
...
128
...
kylin.storage.columnar.shard-rowcount
...
2500000
...
kylin.storage.columnar.shard-countdistinct-rowcount
...
1000000
...
kylin.storage.columnar.repartition-threshold-size-mb
...
128
...
kylin.storage.columnar.shard-min
...
1
...
kylin.storage.columnar.shard-max
...
1000
...
kylin.storage.columnar.hdfs-blocksize-bytes
...
5 * shard_size
...
kylin.storage.columnar.shard-expand-factor
...
10
...
kylin.storage.columnar.dfs-replication
...
3
...
kylin.spark-conf.auto.prior
...
true
...
kylin.job.log-print-enabled
...
true
...
kylin.query.spark-engine.join-memory-fraction
...
0.3
...
driver memory that can be used by join(mostly BHJ)
...
kylin.query.spark-engine.enabled
...
true
...
kylin.query.spark-engine.partition-split-size-mb
...
64
...
kylin.query.spark-engine.expose-sharding-trait
...
true
...
kylin.query.spark-engine.spark-sql-shuffle-partitions
...
-1
...
kylin.query.spark-conf.
...
kylin.query.engine.sparder-additional-files
...
kylin.query.engine.sparder-additional-jars
...
kylin.query.pushdown.auto-set-shuffle-partitions-enabled
...
true
...
kylin.query.pushdown.base-shuffle-partition-size
...
48
...
kylin.query.intersect.separator
...
|
...
kylin.kerberos.enabled
...
false
...
kylin.kerberos.keytab
...
kylin.kerberos.zookeeper.server.principal
...
zookeeper/hadoop
...
kylin.kerberos.ticket.refresh.interval.minutes
...
720
...
kylin.kerberos.monitor.interval.minutes
...
10
...
kylin.kerberos.platform
...
kylin.platform.zk.kerberos.enable
...
kylin.kerberos.krb5.conf
...
krb5.conf
...
kylin.kerberos.jaas.conf
...
jaas.conf
...
kylin.kerberos.principal
Configurations need to be removed
Overview
Content Tools
ThemeBuilder
Apps