THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Apache Kylin

Space shortcuts

Apache Kylin : Analytical Data Warehouse for Big Data

Page tree

Versions Compared

Old Version 32

changes.mady.by.user Xiaoxiang Yu

Saved on

compared with

New Version 33

changes.mady.by.user Xiaoxiang Yu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Property
Importance
DefaultDescriptionVersion
kylin.engine.spark.build-class-name
Statussubtletruetitletrivial
org.apache.kylin.engine.spark.job.CubeBuildJob
For developer only. The className use in spark-submit
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.spark.cluster-info-fetcher-class-name
 Statussubtletitletrivial
true
org.apache.kylin.cluster.YarnInfoFetcher
For developer only. Fetch yarn information of spark job
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.spark-conf.XXX
status
subtletrue
titleMinor

  1. Before Kylin submit a cubing job, some major property(cores and memory) will be automatically adjusted adaptively. (if kylin.spark-conf.auto.prior was set to true).
  2. After auto adjust, spark conf will be overwrite by this property. If you want to set spark.driver.extraJavaOptions=-Dhdp.version=current, you can add follow line in kylin.properties:
kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.version=current
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.storage.provider
 Statussubtletrue
titletrivial
org.apache.kylin.common.storage.DefaultStorageProvider
The content summary objects returned by different cloud vendors are not the same, so need to provide targeted implementation.
You can refer to this to learn more : org.apache.kylin.common.storage.IStorageProvider
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.spark.merge-class-name
 Statussubtletruetitletrivial
org.apache.kylin.engine.spark.job.CubeMergeJob
For developer only. The className use in spark-submit
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.spark.task-impact-instance-enabled
 Statussubtletruetitletrivial
true
 Status
subtletrue
colourYellow
titleUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.spark.task-core-factor
 StatussubtletruecolourYellowtitleMedium
3
 Status
subtletrue
colourYellow
titleUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.driver-memory-base
 Status
subtletrue
colourYellow
titleMedium
1024Auto adujst spark.driver.memory for Build Engine if kylin.engine.spark-conf.spark.driver.memory is not set.



 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.driver-memory-strategy
 Status
subtletrue
colourYellow
titleMedium
{"2", "20", "100"}
 Status
subtletrue
colourYellow
titleUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.driver-memory-maximum
 StatussubtletruecolourYellow
titleMedium
4096
 Status
subtletrue
colourYellow
titleUpdating
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.persist-flattable-threshold
 StatussubtletruecolourYellowtitleMedium
1If the number of cuboids which will be build from flat table is bigger than this threshold, the flat table will be persisted into $HDFS_WORKING_DIR/job_tmp/flat_table for saving more memory.
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.snapshot.parallel-build-timeout-seconds
 Statussubtletrue
colourRedtitleMAJOR
3600
To improve the speed of snapshot build.

 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.snapshot.parallel-build-enabled
 Statussubtle
true
colourRedtitleMAJORtrue
 Status
subtletrue
colourYellow
titleUpdating





kylin.spark-conf.auto.prior
 StatussubtletruecolourGreentitleMinor
true Enable adjust spark parameters adaptively.
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.engine.submit-hadoop-conf-dir
 StatussubtletruecolourRedtitleMAJOR
/etc/hadoop/conf
Set HADOOP_CONF_DIR for spark-submit.
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.storage.columnar.shard-size-mb
 StatussubtletruecolourRed
titleMAJOR
128
The max size of pre-calcualted cuboid parquet file.
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.storage.columnar.shard-rowcount
 StatussubtlecolourRedtitleMAJOR
true
2500000
The max rows of pre-calcualted cuboid parquet file.
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.storage.columnar.shard-countdistinct-rowcount
 StatussubtletruecolourRedtitleMAJOR
1000000The max rows of pre-calcualted cuboid parquet file when cuboid has bitmap measure. (When cuboid has BItmap, it is large.)
 Status
subtletrue
colourBlue
title4.0.0-alpha
kylin.query.spark-engine.join-memory-fraction
 StatussubtletruecolourYellowtitleMedium
0.3Limit memory used by broadcast join of Sparder. (Broadcast join cause unstable.)
 Status
subtletrue
colourBlue
title4.0.0-alpha