DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Draft
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Metrics regarding broker startup and shutdown times are helpful to have. Long startups/shutdown times can hang rolls, and may not be visible from the admin’s perspective if the broker itself is not yet at the point (or past the point) of reporting metrics. These metrics can be reported on the controller side using the BrokerHeartbeatManager to monitor broker startup/shutdown.
Public Interfaces
Monitoring
This KIP will add the following metrics:
| Name | Type | Description |
|---|---|---|
| kafka.controller:type=KafkaController,name=LongestPendingStartupTimeMs | Long | The duration, in milliseconds, of the longest pending broker startup. |
| kafka.controller:type=KafkaController,name=LongestPendingStartupBroker | String | The broker ID of the longest pending broker startup. |
| kafka.controller:type=KafkaController,name=LongestPendingControlledShudownTimeMs | Long | The duration, in milliseconds, of the longest pending broker controlled shutdown. |
| kafka.controller:type=KafkaController,name=LongestPendingControlledShutdownBroker | String | The broker ID of the longest pending broker controlled shutdown. |
Pending broker startup duration: The time between when the controller receives a BrokerRegistration for a brokerID and when the controller processes a BrokerHeartbeat for that broker that causes the broker to be considered "caught up" and unfenced.
Pending broker controlled shutdown duration: The time between when the controller updates the controlled shutdown offset for a broker and when the broker leaves controlled shutdown to the SHUTDOWN_NOW state.
Rationale
From the admin's perspective, it seems that the most important thing to monitor is the longest pending startup/shutdown duration, and the broker associated with it. Alerts fired on these metrics tell the admin if a roll is hanging or if the cluster is in a degraded state because of a broker startup/shutdown taking longer than reasonably expected. Tracking these metrics specifically, rather than a duration for every broker ensures that metric cardinality is independent of the number of brokers in a cluster while still capturing the desired information on the controller-side about brokers that may not be producing metrics yet.
Additional Metrics/Alternatives
Whether or not the cluster is in a degraded state (i.e. >= 1 broker fenced from the controller's POV) can also be reported. This metric would not be specific to broker startup or controlled shutdown, and is less descriptive than the ones above, but nevertheless still offers important monitoring information.
Proposed Changes
These metrics can be implemented using two monotonically increasing LinkedHashMaps mapping from brokerID → registrationTimeMs/controlledShutdownBeginTimeMs.
Entries are put in at the beginning of the durations defined above and removed at the end.
These metric values are updated whenever the controller processes a BrokerHeartbeat. During this processing, updating the longest duration takes constant time, since getting the oldest entry and removing entries are both O(1).
Modifications on these maps are thread-safe since they only occur on the main controller thread.
Compatibility, Deprecation, and Migration Plan
These will be newly exposed metrics and there will be no impact on existing kafka versions.
Test Plan
We will add junit tests to verify the new metrics.
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.