Page History

1. Background

System cube is a set of cubes created by kylin for better self-monitoring, which is supported from kylin-2.3.0.

...

Anchor
Manual Manual

3.2 Create system cube manually

...

Step1 Prepare configuration file

Create a configuration file SCSinkTools.json in the $KYLIN_HOME directory. For example:

[
    {
       "sink": "hive",
       "storage_type": 4,
       "cube_desc_override_properties": {
         "kylin.cube.max-building-segments": "1"
       }
    }
]

...

Step2 Generate metadata

Run the following command in $KYLIN_HOME directory to generate related metadata:

...

With this command, the related metadata will be generated and its location is <output_forder>. The details are as follows, here's system_cube is our <output_forder>:

...

Step3 Create hive tables

Run the following command to generate five hive tables in the above table:

...

By default, these tables will be created in the database named kylin in hive, and the default value kylin can be modified through the configuration item kylin.metrics.prefix.

...

Step4 Restore metadata

Then we need to restore system cube metadata to kylin metastore through the following command:

bin/metastore.sh restore <output_forder>

...

Step5 Reload metadata

Finally, Reload metadata in kylin Web UI , you can see a group of system cubes appear in the project named KYLIN_SYSTEM.

...

Step6 Build system cube regularly

After creating system cubes, as the metrics information is written into hive, these cubes need to be built regularly so that the metrics information written into hive can be quickly queried in kylin.

You can use the following methods to build the system cube on a regular basis:

1、Create 1 Create a shell script by calling org.apache.kylin.tool.job.CubeBuildingCLI to build the system cube. For example:

#!/bin/bash

dir=$(dirname ${0})
export KYLIN_HOME=${dir}/../

CUBE=$1
INTERVAL=$2
DELAY=$3
CURRENT_TIME_IN_SECOND=`date +%s`
CURRENT_TIME=$((CURRENT_TIME_IN_SECOND * 1000))
END_TIME=$((CURRENT_TIME-DELAY))
END=$((END_TIME - END_TIME%INTERVAL))

ID="$END"
echo "building for ${CUBE}_${ID}" >> ${KYLIN_HOME}/logs/build_trace.log
sh ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.job.CubeBuildingCLI --cube ${CUBE} --endTime ${END} > ${KYLIN_HOME}/logs/system_cube_${CUBE}_${END}.log 2>&1 &

2、Run 2 Run the shell script on a regular basis. This can be achieved by adding a cron job as follows:

...