Kafka runs on JVM, but no JVM exporter in kafka-ecosystems. i wrote one on Spring boot for working and very happy to share.
Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
kafka is an excellent MQ/Data Pipeline running on JVM, but no exporters JVMly. for a better future of Kafka-Ecosystems
the Apache needs a formal exporter like https://github.com/apache/rocketmq-exporter.
i wrote one for working, and hope to give to Apache. there are a lot of metric in JMX, it can be configed in the exporter-config.
server: port: 5650 spring: application: name: kafka-exporter profiles: active: dev http: encoding: charset: UTF-8 enabled: true force: true logging: config: classpath:logback.xml task: count: 8 brokerTopicMetrics: cron: 1/15 * * * * ? lagMetrics: cron: 1/15 * * * * ? jvmMetrics: cron: 1/15 * * * * ? replicaMetrics: cron: 1/15 * * * 12 ? networkMetrics: cron: 1/15 * * * * ? logFlushMetrics: cron: 1/15 * * * * ? kafkaControllerMetrics: cron: 1/15 * * * 12 ? kafkaClusterMetrics: cron: 1/15 * * * 12 ? kafka-exporter: kafka-versions.0.10.2.0: 1 ## different kafka versions using different api versions kafka-versions.0.10.1.1: 1 ## different kafka versions using different api versions kafka-versions.1.0.0: 1 canSendToPaladin: true ## allowCollectMetrics and forbidCollectMetricNames for this yml's task config allowCollectMetrics.brokerTopicMetrics: - kafka.server:type=BrokerTopicMetrics,name=* - kafka.server:type=BrokerTopicMetrics,name=*,topic=* forbidCollectMetricNames.brokerTopicMetrics: - FetchMessageConversionsPerSec allowCollectMetrics.jvmMetrics: - java.lang:type=GarbageCollector,name=* - java.lang:type=Threading forbidCollectMetricNames.jvmMetrics: - Code Cache allowCollectMetrics.replicaMetrics: - kafka.server:type=ReplicaManager,name=* forbidCollectMetricNames.replicaMetrics: - aa allowCollectMetrics.networkMetrics: - kafka.network:type=RequestMetrics,name=*,request=* - kafka.network:type=RequestMetrics,name=*,request=*,version=* # for 2.0.0 - kafka.network:type=SocketServer,name=* - kafka.network:type=RequestChannel,name=* - kafka.server:type=KafkaRequestHandlerPool,name=* forbidCollectMetricNames.networkMetrics: - MessageConversionsTimeMs # normally , use metric name - TemporaryMemoryBytes - MessageConversionsTimeMs - ThrottleTimeMs - TotalTimeMs - LocalTimeMs - RemoteTimeMs - RequestBytes - ResponseQueueTimeMs - ResponseSendTimeMs forbidCollectMetricNames.RequestMetrics: - AlterConfigs - AlterReplicaLogDirs - ApiVersions - ControlledShutdown - CreateAcls - CreateDelegationToken - DeleteAcls - DeleteRecords - DescribeAcls - DescribeConfigs - DescribeDelegationToken - DescribeLogDirs - EndTxn - ExpireDelegationToken - InitProducerId - OffsetForLeaderEpoch - RenewDelegationToken - SaslAuthenticate - SaslHandshake - StopReplica - TxnOffsetCommit - WriteTxnMarkers - AddOffsetsToTxn allowCollectMetrics.logFlushMetrics: - kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs - kafka.log:type=LogCleanerManager,name=* forbidCollectMetricNames.logFlushMetrics: - aa allowCollectMetrics.kafkaControllerMetrics: - kafka.controller:type=KafkaController,name=* forbidCollectMetricNames.kafkaControllerMetrics: - aa allowCollectMetrics.kafkaClusterMetrics: - kafka.cluster:type=Partition,name=*,topic=*,partition=* forbidCollectMetricNames.kafkaClusterMetrics: - aa jmx-excludes-metrics.brokerTopicMetrics: - aa jmx-excludes-attrs.BrokerTopicMetrics: - aa jmx-excludes-attrs-global: - EventType - RateUnit - LatencyUnit - 50thPercentile - 75thPercentile - 98thPercentile - LastGcInfo - MemoryPoolNames - ObjectName - Valid - Name - ThreadAllocatedMemoryEnabled - ThreadAllocatedMemorySupported - ThreadContentionMonitoringEnabled - AllThreadIds - ThreadCpuTimeSupported - ThreadCpuTimeEnabled - ThreadContentionMonitoringSupported - CurrentThreadCpuTimeSupported - ObjectMonitorUsageSupported - SynchronizerUsageSupported |
kafka-exporter: zookeepers: - cluster-name: cluster-name-of-your-kafka-brokers ##cluster name zk-ip-and-port: 127.0.0.1:2181,127.0.0.2:2181 ## zookeeper addresses zk-kafka-path: /kafka ## zookeeper namespace excludes-topics.BrokerTopicMetrics: - aaa - bbb - beexiao(.*?) jmx-excludes-metrics.BrokerTopicMetrics: - aa - bb jmx-excludes-metrics.RequestMetrics: - AlterConfigs - AlterReplicaLogDirs - ApiVersions - ControlledShutdown - CreateAcls - CreateDelegationToken - DeleteAcls - DeleteRecords - DescribeAcls - DescribeConfigs - DescribeDelegationToken - DescribeLogDirs - EndTxn - ExpireDelegationToken - InitProducerId - OffsetForLeaderEpoch - RenewDelegationToken - SaslAuthenticate - SaslHandshake - StopReplica - TxnOffsetCommit - WriteTxnMarkers - AddOffsetsToTxn jmx-excludes-attrs.BrokerTopicMetrics: - EventType - RateUnit jmx-excludes-attrs.GarbageCollector: - LastGcInfo - MemoryPoolNames - ObjectName - Valid - Name jmx-excludes-attrs.ReplicaManager: - EventType - RateUnit jmx-excludes-attrs.RequestMetrics: - EventType - RateUnit - FifteenMinuteRate - FiveMinuteRate - 75thPercentile - 98thPercentile jmx-excludes-attrs.LogFlushRateAndTimeMs: - LatencyUnit - RateUnit - EventType - FifteenMinuteRate - 50thPercentile - 75thPercentile - 98thPercentile |
kafka_BrokerTopicMetrics_BytesInPerSec_Count kafka_BrokerTopicMetrics_BytesInPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_BytesInPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_BytesInPerSec_MeanRate kafka_BrokerTopicMetrics_BytesInPerSec_OneMinuteRate kafka_BrokerTopicMetrics_BytesOutPerSec_Count kafka_BrokerTopicMetrics_BytesOutPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_BytesOutPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_BytesOutPerSec_MeanRate kafka_BrokerTopicMetrics_BytesOutPerSec_OneMinuteRate kafka_BrokerTopicMetrics_BytesRejectedPerSec_Count kafka_BrokerTopicMetrics_BytesRejectedPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_BytesRejectedPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_BytesRejectedPerSec_MeanRate kafka_BrokerTopicMetrics_BytesRejectedPerSec_OneMinuteRate kafka_BrokerTopicMetrics_FailedFetchRequestsPerSec_Count kafka_BrokerTopicMetrics_FailedFetchRequestsPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_FailedFetchRequestsPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_FailedFetchRequestsPerSec_MeanRate kafka_BrokerTopicMetrics_FailedFetchRequestsPerSec_OneMinuteRate kafka_BrokerTopicMetrics_FailedProduceRequestsPerSec_Count kafka_BrokerTopicMetrics_FailedProduceRequestsPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_FailedProduceRequestsPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_FailedProduceRequestsPerSec_MeanRate kafka_BrokerTopicMetrics_FailedProduceRequestsPerSec_OneMinuteRate kafka_BrokerTopicMetrics_MessagesInPerSec_Count kafka_BrokerTopicMetrics_MessagesInPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_MessagesInPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_MessagesInPerSec_MeanRate kafka_BrokerTopicMetrics_MessagesInPerSec_OneMinuteRate kafka_BrokerTopicMetrics_ProduceMessageConversionsPerSec_Count kafka_BrokerTopicMetrics_ProduceMessageConversionsPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_ProduceMessageConversionsPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_ProduceMessageConversionsPerSec_MeanRate kafka_BrokerTopicMetrics_ProduceMessageConversionsPerSec_OneMinuteRate kafka_BrokerTopicMetrics_ReplicationBytesInPerSec_Count kafka_BrokerTopicMetrics_ReplicationBytesInPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_ReplicationBytesInPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_ReplicationBytesInPerSec_MeanRate kafka_BrokerTopicMetrics_ReplicationBytesInPerSec_OneMinuteRate kafka_BrokerTopicMetrics_ReplicationBytesOutPerSec_Count kafka_BrokerTopicMetrics_ReplicationBytesOutPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_ReplicationBytesOutPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_ReplicationBytesOutPerSec_MeanRate kafka_BrokerTopicMetrics_ReplicationBytesOutPerSec_OneMinuteRate kafka_BrokerTopicMetrics_TotalFetchRequestsPerSec_Count kafka_BrokerTopicMetrics_TotalFetchRequestsPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_TotalFetchRequestsPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_TotalFetchRequestsPerSec_MeanRate kafka_BrokerTopicMetrics_TotalFetchRequestsPerSec_OneMinuteRate kafka_BrokerTopicMetrics_TotalProduceRequestsPerSec_Count kafka_BrokerTopicMetrics_TotalProduceRequestsPerSec_FifteenMinuteRate kafka_BrokerTopicMetrics_TotalProduceRequestsPerSec_FiveMinuteRate kafka_BrokerTopicMetrics_TotalProduceRequestsPerSec_MeanRate kafka_BrokerTopicMetrics_TotalProduceRequestsPerSec_OneMinuteRate kafka_GarbageCollector_G1_Old_Generation_CollectionCount kafka_GarbageCollector_G1_Old_Generation_CollectionTime kafka_GarbageCollector_G1_Young_Generation_CollectionCount kafka_GarbageCollector_G1_Young_Generation_CollectionTime kafka_KafkaController_ActiveControllerCount_Value kafka_KafkaController_ControllerState_Value kafka_KafkaController_GlobalPartitionCount_Value kafka_KafkaController_GlobalTopicCount_Value kafka_KafkaController_OfflinePartitionsCount_Value kafka_KafkaController_PreferredReplicaImbalanceCount_Value kafka_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent_Count kafka_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent_FifteenMinuteRate kafka_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent_FiveMinuteRate kafka_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent_MeanRate kafka_KafkaRequestHandlerPool_RequestHandlerAvgIdlePercent_OneMinuteRate kafka_LogCleanerManager_max_dirty_percent_Value kafka_LogCleanerManager_time_since_last_run_ms_Value kafka_LogFlushStats_LogFlushRateAndTimeMs_95thPercentile kafka_LogFlushStats_LogFlushRateAndTimeMs_999thPercentile kafka_LogFlushStats_LogFlushRateAndTimeMs_99thPercentile kafka_LogFlushStats_LogFlushRateAndTimeMs_Count kafka_LogFlushStats_LogFlushRateAndTimeMs_FifteenMinuteRate kafka_LogFlushStats_LogFlushRateAndTimeMs_FiveMinuteRate kafka_LogFlushStats_LogFlushRateAndTimeMs_Max kafka_LogFlushStats_LogFlushRateAndTimeMs_Mean kafka_LogFlushStats_LogFlushRateAndTimeMs_MeanRate kafka_LogFlushStats_LogFlushRateAndTimeMs_Min kafka_LogFlushStats_LogFlushRateAndTimeMs_OneMinuteRate kafka_LogFlushStats_LogFlushRateAndTimeMs_StdDev kafka_Partition_InSyncReplicasCount_Value kafka_Partition_LastStableOffsetLag_Value kafka_Partition_ReplicasCount_Value kafka_Partition_UnderMinIsr_Value kafka_Partition_UnderReplicated_Value kafka_ReplicaManager_FailedIsrUpdatesPerSec_Count kafka_ReplicaManager_FailedIsrUpdatesPerSec_FifteenMinuteRate kafka_ReplicaManager_FailedIsrUpdatesPerSec_FiveMinuteRate kafka_ReplicaManager_FailedIsrUpdatesPerSec_MeanRate kafka_ReplicaManager_FailedIsrUpdatesPerSec_OneMinuteRate kafka_ReplicaManager_IsrExpandsPerSec_Count kafka_ReplicaManager_IsrExpandsPerSec_FifteenMinuteRate kafka_ReplicaManager_IsrExpandsPerSec_FiveMinuteRate kafka_ReplicaManager_IsrExpandsPerSec_MeanRate kafka_ReplicaManager_IsrExpandsPerSec_OneMinuteRate kafka_ReplicaManager_IsrShrinksPerSec_Count kafka_ReplicaManager_IsrShrinksPerSec_FifteenMinuteRate kafka_ReplicaManager_IsrShrinksPerSec_FiveMinuteRate kafka_ReplicaManager_IsrShrinksPerSec_MeanRate kafka_ReplicaManager_IsrShrinksPerSec_OneMinuteRate kafka_ReplicaManager_LeaderCount_Value kafka_ReplicaManager_OfflineReplicaCount_Value kafka_ReplicaManager_PartitionCount_Value kafka_ReplicaManager_UnderMinIsrPartitionCount_Value kafka_ReplicaManager_UnderReplicatedPartitions_Value kafka_RequestChannel_RequestQueueSize_Value kafka_RequestChannel_ResponseQueueSize_Value kafka_RequestMetrics_RequestQueueTimeMs_95thPercentile kafka_RequestMetrics_RequestQueueTimeMs_999thPercentile kafka_RequestMetrics_RequestQueueTimeMs_99thPercentile kafka_RequestMetrics_RequestQueueTimeMs_Count kafka_RequestMetrics_RequestQueueTimeMs_Max kafka_RequestMetrics_RequestQueueTimeMs_Mean kafka_RequestMetrics_RequestQueueTimeMs_Min kafka_RequestMetrics_RequestQueueTimeMs_StdDev kafka_RequestMetrics_RequestsPerSec_Count kafka_RequestMetrics_RequestsPerSec_FifteenMinuteRate kafka_RequestMetrics_RequestsPerSec_FiveMinuteRate kafka_RequestMetrics_RequestsPerSec_MeanRate kafka_RequestMetrics_RequestsPerSec_OneMinuteRate kafka_SocketServer_MemoryPoolAvailable_Value kafka_SocketServer_MemoryPoolUsed_Value kafka_SocketServer_NetworkProcessorAvgIdlePercent_Value kafka_Threading_CurrentThreadCpuTime kafka_Threading_CurrentThreadUserTime kafka_Threading_DaemonThreadCount kafka_Threading_PeakThreadCount kafka_Threading_ThreadCount kafka_Threading_TotalStartedThreadCount kafka_consumer_lag kafka_topic_partitions |
build a whole new kafka-exporter RUN-ON-JVM for kafka
let's do this!