Status

Current state: Under discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Metrics regarding broker startup and shutdown times are helpful to have. Long startups/shutdown times can stretch out upgrades, and may not be visible from the admin’s perspective if the broker itself is not yet at the point (or past the point) of reporting metrics. These metrics can be reported on the controller side using the BrokerHeartbeatManager to monitor broker startup/shutdown.

Public Interfaces

Monitoring

The controller currently exposes metrics for the active and fenced broker counts. However, these metrics are not descriptive enough, since they do not tell us from the controller-side which broker is in which state. Instead, upon alerting on these metrics, the operator will have to look through the metadata log to find which broker(s) are in which states. This KIP proposes adding controller-side metrics to monitor the states of brokers.

Public Interfaces

Monitoring

Rationale

These metrics would be useful for monitoring the following use cases:

• When multiple brokers are fenced – In addition to how many brokers are fenced, the operator will now know which brokers are fenced without having to look at the logs. Additionally, information provided by the heartbeat metrics can help the operator narrow down the cause of the fencing.
• Expanding and shrinking the cluster – When expanding, the brokers being added are initially fenced until they catch up on metadata, so the benefits are similar to above. When shrinking, the operator can monitor that controlled shutdown of the to-be-removed brokers is not taking longer than expected.

Proposed Changes

The metrics as they are described above can be implemented with what currently exists in the codebase, so no change to the broker registration records needs to be made.

However, if we want to add another value to BrokerRegistrationState  that maps to starting up brokers (i.e. never unfenced), this would require adding a boolean to the broker's registration record. Additionally, if we want to track is a broker has been uncleanly shutdown, we would likely need to store the broker epoch in its registration record.

Compatibility, Deprecation, and Migration Plan

These will be newly exposed metrics and there will be no impact on existing kafka versions.

Test Plan

We will add junit tests to verify the new metrics.

Rejected Alternatives

...

Name	Type	Description
kafka.controller:type=KafkaController,name=LongestPendingStartupTimeMs	Long	The duration, in milliseconds, of the longest pending broker startup.
kafka.controller:type=KafkaController,name=LongestPendingStartupBroker	String	The broker ID of the longest pending broker startup.
kafka.controller:type=KafkaController,name=NumberOfBrokersInStartup	Integer	The number of brokers currently starting up.
kafka.controller:type=KafkaController,name=LongestPendingControlledShudownTimeMs	Long	The duration, in milliseconds, of the longest pending broker controlled shutdown.
kafka.controller:type=KafkaController,name=LongestPendingControlledShutdownBroker	String	The broker ID of the longest pending broker controlled shutdown.
kafka.controller:type=KafkaController,name=NumberOfBrokersInControlledShutdown	Integer	The number of brokers currently in controlled shutdown.

Pending broker startup duration: The time between when the controller receives a BrokerRegistration for a brokerID and when the controller processes a BrokerHeartbeat for that broker that causes the broker to be considered "caught up" and unfenced.

Pending broker controlled shutdown duration: The time between when the controller updates the controlled shutdown offset for a broker and when the broker leaves controlled shutdown to the SHUTDOWN_NOW state.

Rationale

From the admin's perspective, it seems that the most important thing to monitor is the longest pending startup/shutdown duration, and the broker associated with it. Alerts fired on these metrics tell the admin if a cluster upgrade is hanging or if the cluster is in a degraded state because of a broker startup/shutdown taking longer than reasonably expected. Tracking these metrics specifically, rather than a duration for every broker makes metric cardinality independent of the number of brokers in a cluster while still capturing the desired information on the controller-side about brokers that may not be producing metrics yet.

Monitoring the  NumberOfBrokersInStartup  and  NumberOfBrokersInControlledShutdown  metrics can be used to alert the admin in cases where multiple brokers are restarted at the same time or unexpected failures happen during upgrades.

Proposed Changes

These metrics can be implemented using two monotonically increasing LinkedHashMaps mapping from brokerID → registrationTimeMs/controlledShutdownBeginTimeMs.

Entries are put in at the beginning of the durations defined above and removed at the end.

These metric values are updated whenever the controller processes a BrokerHeartbeat and entries exist in the respective maps. During this processing, updating the longest duration takes constant time, since getting the oldest entry and removing entries are both O(1).

Modifications on these maps are thread-safe since they only occur on the main controller thread.

Compatibility, Deprecation, and Migration Plan

These will be newly exposed metrics and there will be no impact on existing kafka versions.

Test Plan

We will add junit tests to verify the new metrics.

...

These duration metrics would rely on soft state, so they would be reset when controller failover occurs. This is a bit confusing, and the scope of these metrics is limited to startup and controlled shutdown scenarios. Instead, more general purpose metrics would be applicable across more use cases.