THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Code Block | ||
|---|---|---|
| ||
npm install -g bower |
Env dependencies
Hadoop
Prerequisities Prerequisites version is 2.6.0
Hive
Prerequisities Prerequisites version is 1.2.1
Spark
Prerequisities Prerequisites version is 1.6.x
Mysql
Prerequisities Prerequisites version is 5.0
Elastic search
Prerequisities Prerequisites version is 5.x.x
Make sure you can access your elastic search instance by http protocol.
...
| Code Block | ||
|---|---|---|
| ||
#livy has one bug (https://issues.cloudera.org/browse/LIVY-94), so we need to makeput these three jars into hdfs datanucleus-api-jdo-3.2.6.jar datanucleus-core-3.2.10.jar datanucleus-rdbms-3.2.9.jar #Then you need to add some configuration in json when submit spark classpath jobs through livy. "jars": [ "hdfs:///livy/datanucleus-api-jdo-3.2.6.jar ", "hdfs:///livy/datanucleus-core-3.2.10.jar ", "hdfs:///livy/datanucleus-rdbms-3.2.9.jar" ] #We need hive accessible in spark application, but there needs some configuration if submit through livy. #livy.conf livy.repl.enableHiveContext = true #hive-site.xml should be put into hdfs, to be accessible by the spark cluster. For example, we can put it as below: hdfs:///livy/hive-site.xml #Then you need to add some configuration in json when submit spark jobs through livy. "conf": { "spark.yarn.dist.files": "hdfs:///livy/hive-site.xml" } #or like this "files": [ "hdfs:///livy/hive-site.xml" ] |
Setup Dev Env
Git clone
| Code Block | ||
|---|---|---|
| ||
git clone https://github.com/apache/incubator-griffin.git |
...
| Code Block |
|---|
#app
org.apache.griffin.measure.batch.Application |
service : web service for data assets, measure metadata, and job schedulers.
| Code Block | ||
|---|---|---|
| ||
#spring boot app org.apache.griffin.core.GriffinWebApplication |
ui : front end
Update several files to reflect your dev env
...