THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Apache Kylin

Space shortcuts

Apache Kylin : Analytical Data Warehouse for Big Data

Page tree

Versions Compared

Old Version 7

changes.mady.by.user Zhichao Zhang

Saved on

compared with

New Version 8

changes.mady.by.user Zhichao Zhang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Read-Write Separation Architecture

Notes:

  1. Kylin 4.0 uses the Yarn resources on build cluster to build cube data and then write the cube data back to the HDFS on query cluster directly.
  2. Build jobs read the Hive data sources which are on build cluster.
  3. When executing pushdown query, Kylin 4.0 reads the Hive data from build cluster.

Prepare

  1. Make sure the hadoop version ( HDP or CDH ) of build cluster and query cluster is supported by Kylin.
  2. The hadoop version of build cluster and query cluster must be the same.
  3. Check commands like hdfs and hive are all working properly and can access cluster resources.
  4. If the two clusters have enabled the HDFS NameNode HA, please check and make sure their HDFS nameservice names are different. If they are the same, please change one of them to avoid conflict.
  5. Please make sure the network latency between the two clusters is low enough, as there will be a large number of data moved back and forth during model build process.

...

  1. Install Kylin 4.0 by the following guide on Kylin server.
  2. Prepare Create a directory called 'build_hadoop_conf' in the $KYLIN_HOME and then copy the hadoop configuration files of the build cluster to this directory (Note: make sure copy the real configuration files not the symbolic links).
  3. Set  the value of the configuration ' two cluster and put them into Kylin server.
  4. Open $KYLIN_HOME/conf/kylin.properties
  5. Set  kylin.env.hadoop-conf-dir with the path of the directories of query cluster hadoop configuration files.
    6. Set kylin.engine.submit-hadoop-conf-dir with the path of the directories of build cluster hadoop configuration files.
     ' in the '$KYLIN_HOME/conf/kylin.properties' to the directory created in the step 2.
  6. Copy Put the hive-site.xml of the build cluster into the directory of the hive configuration on query cluster hadoop configuration files.

Now read/write separation deployment is configured.

Note

...

$KYLIN_HOME/bin/check-env.sh and $KYLIN_HOME/bin/sample.sh are not available in this deployment mode.

...

  1. , for example: /etc/hive/conf.
  2. The value of the configuration 'kylin.engine.spark-conf.spark.yarn.queue' in

...

  1. the '$KYLIN_HOME/conf/kylin.properties' should be configured as the queue of the Yarn on build cluster.

Note

  • $KYLIN_HOME/bin/sample.sh is not supported in this deployment mode.