Apache Kylin : Analytical Data Warehouse for Big Data
Welcome to Kylin Wiki.
As we all know, Kylin needs to build cube data before querying, so if the build jobs and query jobs are both running in one cluster, service may be unstable because of the resource preemption.
Now, Kylin 4.0 supports to finish build and query jobs on the different Hadoop clusters which we call build cluster and query cluster. The build jobs will be sent to build cluster to build cube data, when the build jobs finished, the cube data will be sent to the HDFS on the query cluster so that we can execute query to read cube data from the query cluster.
With a read-write separation deployment, we can completely isolate build and query workloads.
Architecture
Prepare
- Make sure the hadoop version(HDP or CDH) is supported by Kylin.
- Check commands like
hdfs and
hive
are all working properly and can access cluster resources. - If the two clusters have enabled the HDFS NameNode HA, please check and make sure their HDFS nameservice names are different. If they are the same, please change one of them to avoid conflict.
- Please make sure the network latency between the two clusters is low enough, as there will be a large number of data moved back and forth during model build process.
Configuration
- Install Kylin 4.0 by the following guide on Kylin server.
- Prepare the hadoop configuration files of the two cluster and put them into Kylin server.
- Open $KYLIN_HOME/conf/kylin.properties
- Set kylin.env.hadoop-conf-dir with the path of the directories of query cluster hadoop configuration files.
- Set kylin.engine.submit-hadoop-conf-dir with the path of the directories of build cluster hadoop configuration files.
- Put the hive-site.xml of the build cluster into the directory of query cluster hadoop configuration files.
Now read/write separation deployment is configured.
Node
$KYLIN_HOME/bin/check-env.sh
and$KYLIN_HOME/bin/sample.sh
are not available in this deployment mode.In this mode,
kylin.engine.spark-conf.spark.yarn.queue
inkylin.properties
should be configured as the queue of the build cluster.