Apache Kylin : Analytical Data Warehouse for Big Data
Welcome to Kylin Wiki.
TRIVIAL PROCESS 4.0.0-ALPHA TRIVIAL PROCESS 4.0.0-ALPHA MINOR PROCESS 4.0.0-ALPHA TRIVIAL PROCESS The content summary objects returned by different cloud vendors are not the same, so need to provide targeted implementation. 4.0.0-ALPHA TRIVIAL PROCESS 4.0.0-ALPHA PROCESS 4.0.0-ALPHA MEDIUM PROCESS TO BE UPDATED 4.0.0-ALPHA MEDIUM PROCESS MEDIUM PROCESS 4.0.0-ALPHA MEDIUM PROCESS TO BE UPDATED 4.0.0-ALPHA MEDIUM PROCESS 4.0.0-ALPHA MAJOR PROCESS MAJOR PROCESS TO BE UPDATED MINOR PROCESS 4.0.0-ALPHA MAJOR PROCESS Set HADOOP_CONF_DIR for spark-submit. 4.0.0-ALPHA MAJOR CUBE The max size of pre-calcualted cuboid parquet file. 4.0.0-ALPHA MAJOR CUBE 2500000 4.0.0-ALPHA MAJOR CUBE 4.0.0-ALPHA MEDIUM PROCESS 4.0.0-ALPHAProperty ImportanceDatatype Configuration Level Default Description Version Reference kylin.engine.spark.build-class-nameString org.apache.kylin.engine.spark.job.CubeBuildJobFor developer only. The className use in spark-submit. kylin.engine.spark.cluster-info-fetcher-class-nameString org.apache.kylin.cluster.YarnInfoFetcher
For developer only. Fetch yarn information of spark job kylin.engine.spark-conf.XXXString Null Spark configurations want to override for build job like "spark.driver.cores". If don't set these spark properties, kylin will automaticly adjust these properties before submitting build job. Adaptively-adjust-spark-parameters kylin.storage.providerString org.apache.kylin.common.storage.DefaultStorageProvider
You can refer to this to learn more : org.apache.kylin.common.storage.IStorageProvider
kylin.engine.spark.merge-class-nameString org.apache.kylin.engine.spark.job.CubeMergeJobFor developer only. The className use in spark-submit kylin.engine.spark.task-impact-instance-enabledBoolean true Check kylin.engine.spark.task-core-factor. If kylin.engine.spark.task-impact-instance-enabled is set to true and kylin.engine.spark-conf.spark.executor.instances is not set, Kylin will calculate spark.executor.instances for Build Engine. Adaptively-adjust-spark-parameters kylin.engine.spark.task-core-factorInteger 3 kylin.engine.driver-memory-baseInteger 1024 Auto adujst spark.driver.memory for Build Engine if kylin.engine.spark-conf.spark.driver.memory is not set.
4.0.0-ALPHAAdaptively-adjust-spark-parameters kylin.engine.driver-memory-strategyArray {"2", "20", "100"}TO BE UPDATEDkylin.engine.driver-memory-maximumInteger 4096 kylin.engine.persist-flattable-thresholdInteger 1 If the number of cuboids which will be build from flat table is bigger than this threshold, the flat table will be persisted into $HDFS_WORKING_DIR/job_tmp/flat_table for saving more memory. kylin.snapshot.parallel-build-timeout-secondsInteger 3600
To improve the speed of snapshot build.
4.0.0-ALPHAkylin.snapshot.parallel-build-enabledBoolean true kylin.spark-conf.auto.priorBoolean true Enable adjust spark parameters adaptively. Adaptively-adjust-spark-parameters kylin.engine.submit-hadoop-conf-dirString /etc/hadoop/conf kylin.storage.columnar.shard-size-mbInteger 128 ShardBy kylin.storage.columnar.shard-rowcountLong The max rows of pre-calcualted cuboid parquet file. kylin.storage.columnar.shard-countdistinct-rowcountLong 1000000 The max rows of pre-calcualted cuboid parquet file when cuboid has bitmap measure. (When cuboid has BItmap, it is large.) kylin.query.spark-engine.join-memory-fractionDouble 0.3 Limit memory used by broadcast join of Sparder. (Broadcast join cause unstable.)
| File Name | Content | Comment |
|---|---|---|
| cubing_detect_items.json | ||
sampling_detect_items.json | ||
count_distinct.json | ||
resource_paths.json | ||
Overview
Content Tools
ThemeBuilder
Apps