DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Status
Current state: Under discussion
Discussion thread: https://lists.apache.org/thread/z8cwnksl6op4jfg7j0nwsg9xxsf8mwhh
JIRA:
KAFKA-18666
-
Getting issue details...
STATUS
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Metrics regarding broker startup and shutdown times are helpful to have. Long startups/shutdown times can stretch out upgrades, and may not be visible from the admin’s perspective if the broker itself is not yet at the point (or past the point) of reporting metrics. These metrics can be reported on the controller side using the BrokerHeartbeatManager to monitor broker startup/shutdown.
Public Interfaces
Monitoring
This KIP will add the following metrics:
| Name | Type | Description |
|---|---|---|
| kafka.controller:type=KafkaController,name=LongestPendingStartupTimeMs | Long | The duration, in milliseconds, of the longest pending broker startup. |
| kafka.controller:type=KafkaController,name=LongestPendingStartupBroker | String | The broker ID of the longest pending broker startup. |
| kafka.controller:type=KafkaController,name=NumberOfBrokersInStartup | Integer | The number of brokers currently starting up. |
| kafka.controller:type=KafkaController,name=LongestPendingControlledShudownTimeMs | Long | The duration, in milliseconds, of the longest pending broker controlled shutdown. |
| kafka.controller:type=KafkaController,name=LongestPendingControlledShutdownBroker | String | The broker ID of the longest pending broker controlled shutdown. |
| kafka.controller:type=KafkaController,name=NumberOfBrokersInControlledShutdown | Integer | The number of brokers currently in controlled shutdown. |
Pending broker startup duration: The time between when the controller receives a BrokerRegistration for a brokerID and when the controller processes a BrokerHeartbeat for that broker that causes the broker to be considered "caught up" and unfenced.
Pending broker controlled shutdown duration: The time between when the controller updates the controlled shutdown offset for a broker and when the broker leaves controlled shutdown to the SHUTDOWN_NOW state.
Rationale
From the admin's perspective, it seems that the most important thing to monitor is the longest pending startup/shutdown duration, and the broker associated with it. Alerts fired on these metrics tell the admin if a cluster upgrade is hanging or if the cluster is in a degraded state because of a broker startup/shutdown taking longer than reasonably expected. Tracking these metrics specifically, rather than a duration for every broker makes metric cardinality independent of the number of brokers in a cluster while still capturing the desired information on the controller-side about brokers that may not be producing metrics yet.
Monitoring the NumberOfBrokersInStartup and NumberOfBrokersInControlledShutdown metrics can be used to alert the admin in cases where multiple brokers are restarted at the same time or unexpected failures happen during upgrades.
Proposed Changes
These metrics can be implemented using two monotonically increasing LinkedHashMaps mapping from brokerID → registrationTimeMs/controlledShutdownBeginTimeMs.
Entries are put in at the beginning of the durations defined above and removed at the end.
These metric values are updated whenever the controller processes a BrokerHeartbeat and entries exist in the respective maps. During this processing, updating the longest duration takes constant time, since getting the oldest entry and removing entries are both O(1).
Modifications on these maps are thread-safe since they only occur on the main controller thread.
Compatibility, Deprecation, and Migration Plan
These will be newly exposed metrics and there will be no impact on existing kafka versions.
Test Plan
We will add junit tests to verify the new metrics.