Page History

Table of Contents

1. Background

System cube is a set of cubes created by kylin for better self-monitoring, which is supported from kylin-2.3.0.

...

2. Configuration

By default, system cube is disabled. To enable system cube, you need to make the following configuration:

...

The records in metrics system will be saved to HDFS after a certain period of time or a certain amount of time. Here, the default time is 10 (minutes), and the default number is 10. You can modify these two values by modifying the configuration item in configuration file $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes/kylinMetrics.xml, the configuration item of "index = 1" indicates how many pieces of data will be inserted into "hive", and the configuration item of "index = 2" indicates how long it will be inserted into "hive", in minutes:

3. Create system cube

Before using the system cube, you need to prepare the hive table and cube mentioned in the above table. You can choose to create system cube manually or create automatically using system-cube.sh.

Anchor
Manual Manual

3.2 Create system cube manually

Step1、Prepare configuration file

Create a configuration file SCSinkTools.json in the $KYLIN_HOME directory. For example:

[
    {
       "sink": "hive",
       "storage_type": 4,
       "cube_desc_override_properties": {
         "kylin.cube.max-building-segments": "1"
       }
    }
]

Step2、Generate metadata

Run the following command in $KYLIN_HOME directory to generate related metadata:

...

With this command, the related metadata will be generated and its location is <output_forder>. The details are as follows, here's system_cube is our <output_forder>:

Step3、Create hive tables

Run the following command to generate five hive tables in the above table:

...

By default, these tables will be created in the database named kylin in hive, and the default value kylin can be modified through the configuration item kylin.metrics.prefix.

Step4、Restore metadata

Then we need to restore system cube metadata to kylin metastore through the following command:

bin/metastore.sh restore <output_forder>

Step5、Reload metadata

Finally, Reload metadata in kylin Web UI , you can see a group of system cubes appear in the project named KYLIN_SYSTEM.

Step6、Build system cube regularly

After creating system cubes, as the metrics information is written into hive, these cubes need to be built regularly so that the metrics information written into hive can be quickly queried in kylin.

...

0 */2 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_QUERY_QA 3600000 1200000

20 */2 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_QUERY_CUBE_QA 3600000 1200000

40 */4 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_QUERY_RPC_QA 3600000 1200000

30 */4 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_JOB_QA 3600000 1200000

50 */12 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_JOB_EXCEPTION_QA 3600000 12000

Anchor
Automatic Automatic

3.2 Automatically create system cube

You can use ${KYLIN_HOME}/bin/system-cube.sh to help you automatically complete the above operations:

...