Page History

Versions Compared

Key

This line was added.
This line was removed.
Formatting was changed.

Kylin 的监控系统的验证的 Metrics 系统的验证, 解释和增强 (KYLIN-4371的开发日志)

...

System Cube 和 Real-time OLAP 的集成的 JIRA issue :

Jira

server	ASF JIRA
serverId	5aa69414-a9e9-3523-82ec-879b028fb15b
key	KYLIN-4371

三. 如何使用

1. 开启 Metrics 信息收集

Code Block

language	java
theme	DJangoRDark
title	kylin.properties
linenumbers	true

kylin.metrics.monitor-enabled=true
kylin.metrics.reporter-job-enabled=true
kylin.metrics.reporter-query-enabled=true
kylin.web.dashboard-enabled=true

...

Code Block

language	xml
theme	DJangoRDark
title	sink.json
linenumbers	true

[
  {
    "sink": "hive",
    "storage_type": 2,
    "cube_desc_override_properties": {
      "kylin.cube.algorithm": "INMEM",
      "kylin.cube.max-building-segments": "1"
    }
  },
  {
    "sink": "kafka",
    "storage_type": 3,
    "cube_desc_override_properties": {
      "kylin.cube.algorithm": "INMEM",
      "kylin.stream.cube.window": 28800,
      "kylin.stream.cube.duration": 3600,
      "kylin.stream.segment.retention.policy": "fullBuild",
      "kylin.cube.max-building-segments": "20"
    },
    "table_properties": {
      "bootstrap.servers": "{YOUR_SERVERS_LIST}"
    }
  }
]

...

在 create_kafka_topic_for_system_cubes.sh 中, 参数 zookeeper,

Code Block

language	bash
theme	RDark

sh create_kafka_topic_for_system_cubes.sh

...

Code Block

language	xml
theme	DJangoRDark
title	kylinMetrics.xml

<beans xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns="http://www.springframework.org/schema/beans"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
            http://www.springframework.org/schema/beans/spring-beans-3.1.xsd">

    <description>Kylin Metrics Related Configuration (System Cube)</description>

    <!-- A Reservoir which don't staged metrics message at all, emit it in no time. Maybe good for debug purpose.-->
    <bean id="instantReservoir" class="org.apache.kylin.metrics.lib.impl.InstantReservoir"/>

    <bean id="kafkaSink" class="org.apache.kylin.metrics.lib.impl.kafka.KafkaSink"/>

    <bean id="initMetricsManager" class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
        <property name="targetClass" value="org.apache.kylin.metrics.MetricsManager"/>
        <property name="targetMethod" value="initMetricsManager"/>
        <property name="arguments">
            <list>
                <!-- 
                Sink of System Cube. 
                -->
                <ref bean="kafkaSink"/>

                <!--
                Bind properties for each ActiveReservoirReporter.   
                -->    
                <map key-type="org.apache.kylin.metrics.lib.ActiveReservoir" value-type="java.util.List">

                    <!-- Each ActiveReservoir can have multi ReservoirReporter -->
                    <entry key-ref="instantReservoir">
                        <list>
                            <bean class="org.apache.kylin.common.util.Pair">
                                <!-- Implementation of ReservoirReporter-->
                                <property name="first"
                                          value="org.apache.kylin.metrics.lib.impl.kafka.KafkaReservoirReporter"/>
                                <!-- Properties for specific ReservoirReporter-->
                                <property name="second">
                                    <props>
                                        <prop key="bootstrap.servers">cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092</prop>
                                    </props>
                                </property>
                            </bean>
                        </list>
                    </entry>
                </map>
            </list>
        </property>
    </bean>
</beans>

7. 启用 System Cube

四. Metrics信息

五. 参考链接

http://kylin.apache.org/docs/tutorial/setup_systemcube.html

...

Space shortcuts

Blog

Versions Compared

Old Version 6

New Version 7

Key

三. 如何使用

四. Metrics信息

五. 参考链接