Apache Kylin : Analytical Data Warehouse for Big Data
Page History
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
// Step 1 : Add jdbc driver(if you use mariadb) [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# find /usr/lib -name "*mariadb*" [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# mkdir ext [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/hive/lib/mariadb-connector-java.jar ext // Step 2 : replace Hadoop related jars [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# rm -rf spark$SPARK_HOME/jars/hadoop-*.jar [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/spark/jars/hadoop-*.jar spark$SPARK_HOME/jars/ [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/spark/jars/emr-spark-goodies.jar spark$SPARK_HOME/jars/ [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/spark/jars/htrace-core4-4.1.0-incubating.jar spark$SPARK_HOME/jars [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar spark$SPARK_HOME/jars [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/hadoop/lib/woodstox-core-5.0.3.jar spark$SPARK_HOME/jars/ |
Start Kylin
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
[root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# sh bin/kylin.sh start Retrieving hadoop conf dir... KYLIN_HOME is set to /home/hadoop/apache-kylin-4.0.0-SNAPSHOT-bin Retrieving hive dependency... Retrieving hadoop conf dir... Retrieving Spark dependency... Start replacing hadoop jars under /home/hadoop/apache-kylin-4.0.0-SNAPSHOT-bin/spark/jars. find: ‘/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/../hadoop/’: No such file or directory 2.10.0-amzn-0.jar Find platform specific jars: , will replace with these jars under /home/hadoop/apache-kylin-4.0.0-SNAPSHOT-bin/spark/jars. Please confirm that the corresponding hadoop jars have been replaced. The automatic replacement program cannot be executed correctly. Done hadoop jars replacement under /home/hadoop/apache-kylin-4.0.0-SNAPSHOT-bin/spark/jars. Start to check whether we need to migrate acl tables Not HBase metadata. Skip check. A new Kylin instance is started by root. To stop it, run 'kylin.sh stop' Check the log at /home/hadoop/apache-kylin-4.0.0-SNAPSHOT-bin/logs/kylin.log Web UI is at http://ip-172-31-1-253.cn-northwest-1.compute.internal:7070/kylin |
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
[root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.43.0.jar spark$SPARK_HOME/jars/ |
Screenshots
- Monitor Pages
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
// Step 0 : Add jdbc driver(if you use mariadb) [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# find /usr/lib -name "*mariadb*" [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# mkdir ext [root@ip-172-31-1-253 apache-kylin-4.0.0-SNAPSHOT-bin]# cp /usr/lib/hive/lib/mariadb-connector-java.jar ext // Step 1 : Replace hadoop related jars [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp commons-configuration-1.10.jar lib/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ rm -rf spark$SPARK_HOME/jars/hadoop-*.jar [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/spark/jars/hadoop-*.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/spark/jars/emr-spark-goodies.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/spark/jars/htrace-core4-4.1.0-incubating.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/hadoop/lib/woodstox-core-5.0.3.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/spark/jars/commons-configuration2-2.1.1.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/spark/jars/re2j-1.1.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/spark/jars/hive-exec-1.2.1-spark2-amzn-1.jar spark$SPARK_HOME/jars/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ rm spark/jars/hive-exec-1.2.1.spark2.jar // Step 2 : Change value of "hive.execution.engine" from "tez" to "mr" [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ rm hadoop_conf/hive-site.xml [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /etc/hive/conf/hive-site.xml hadoop_conf/ [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ vim hadoop_conf/hive-site.xml // Step 3 : Add lzo jars [hadoop@ip-172-31-8-116 apache-kylin-4.0.0-SNAPSHOT-bin]$ cp /usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar spark$SPARK_HOME/jars |
Overview
Content Tools
ThemeBuilder
Apps