DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Metrics regarding broker startup and shutdown times are helpful to have. Long startups/shutdown times can hang rollsstretch out upgrades, and may not be visible from the admin’s perspective if the broker itself is not yet at the point (or past the point) of reporting metrics. These metrics can be reported on the controller side using the BrokerHeartbeatManager to monitor broker startup/shutdown.
...
| Name | Type | Description |
|---|---|---|
| kafka.controller:type=KafkaController,name=LongestPendingStartupTimeMs | Long | The duration, in milliseconds, of the longest pending broker startup. |
| kafka.controller:type=KafkaController,name=LongestPendingStartupBroker | String | The broker ID of the longest pending broker startup. |
| kafka.controller:type=KafkaController,name=NumberOfBrokersInStartup | Integer | The number of brokers currently starting up. |
| kafka.controller:type=KafkaController,name=LongestPendingControlledShudownTimeMs | Long | The duration, in milliseconds, of the longest pending broker controlled shutdown. |
| kafka.controller:type=KafkaController,name=LongestPendingControlledShutdownBroker | String | The broker ID of the longest pending broker controlled shutdown. |
| kafka.controller:type=KafkaController,name=NumberOfBrokersInControlledShutdown | Integer | The number of brokers currently in controlled shutdown. |
Pending broker startup duration: The time between when the controller receives a BrokerRegistration for a brokerID and when the controller processes a BrokerHeartbeat for that broker that causes the broker to be considered "caught up" and unfenced.
...
From the admin's perspective, it seems that the most important thing to monitor is the longest pending startup/shutdown duration, and the broker associated with it. Alerts fired on these metrics tell the admin if a roll cluster upgrade is hanging or if the cluster is in a degraded state because of a broker startup/shutdown taking longer than reasonably expected. Tracking these metrics specifically, rather than a duration for every broker ensures that makes metric cardinality is independent of the number of brokers in a cluster while still capturing the desired information on the controller-side about brokers that may not be producing metrics yet.
Additional Metrics/Alternatives
Whether or not the cluster is in a degraded state (i.e. >= 1 broker fenced from the controller's POV) can also be reported. This metric would not be specific to broker startup or controlled shutdown, and is less descriptive than the ones above, but nevertheless still offers important monitoring informationMonitoring the NumberOfBrokersInStartup and NumberOfBrokersInControlledShutdown metrics can be used to alert the admin in cases where multiple brokers are restarted at the same time or unexpected failures happen during upgrades.
Proposed Changes
These metrics can be implemented using two monotonically increasing LinkedHashMaps mapping from brokerID → registrationTimeMs/controlledShutdownBeginTimeMs.
...
These metric values are updated whenever the controller processes a BrokerHeartbeat and entries exist in the respective maps. During this processing, updating the longest duration takes constant time, since getting the oldest entry and removing entries are both O(1).
...