Status

Current state: DraftAccepted

Vote thread: https://lists.apache.org/thread/rysj5t3rfpz5pp1rjk0s7w9v7lq7dfpb

Discussion thread: https://lists.apache.org/thread/z8cwnksl6op4jfg7j0nwsg9xxsf8mwhh

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Metrics regarding broker startup and shutdown times are helpful to have. Long startups/shutdown times can hang rolls, and may not be visible from the admin’s perspective if the broker itself is not yet at the point (or past the point) of reporting metrics. These metrics can be reported on the controller side using the BrokerHeartbeatManager to monitor broker startup/shutdown.

Public Interfaces

Monitoring

The controller currently exposes metrics for the active and fenced broker counts. However, these metrics are not descriptive enough, since they do not tell us from the controller-side which broker is in which state. Instead, upon alerting on these metrics, the operator will have to look through the metadata log to find which broker(s) are in which states. This KIP proposes adding controller-side metrics to monitor the states of brokers.

Public Interfaces

Monitoring

We will use tagging for the per-broker metrics where X represents the ID of the broker.

Rationale

These metrics would be useful for monitoring the following use cases:

• When multiple brokers are fenced – In addition to how many brokers are fenced, the operator will now know which brokers are fenced without having to look at the logs. Additionally, information provided by the heartbeat metrics can help the operator narrow down the cause of the fencing.
• Expanding and shrinking the cluster – When expanding, the brokers being added are initially fenced until they catch up on metadata, so the benefits are similar to above. When shrinking, the operator can monitor that controlled shutdown of the to-be-removed brokers is not taking longer than expected.

Compatibility, Deprecation, and Migration Plan

These will be newly exposed metrics and there will be no impact on existing kafka versions.

Test Plan

We will add junit tests to verify the new metrics.

Rejected Alternatives

...

Name	Type	Description
kafka.controller:type=KafkaController,name=LongestPendingStartupTimeMs	Long	The duration, in milliseconds, of the longest pending broker startup.
kafka.controller:type=KafkaController,name=LongestPendingStartupBroker	String	The broker ID of the longest pending broker startup.
kafka.controller:type=KafkaController,name=NumberOfBrokersInStartup	Integer	The number of brokers currently starting up.
kafka.controller:type=KafkaController,name=LongestPendingControlledShudownTimeMs	Long	The duration, in milliseconds, of the longest pending broker controlled shutdown.
kafka.controller:type=KafkaController,name=LongestPendingControlledShutdownBroker	String	The broker ID of the longest pending broker controlled shutdown.

Pending broker startup duration: The time between when the controller receives a BrokerRegistration for a brokerID and when the controller processes a BrokerHeartbeat for that broker that causes the broker to be considered "caught up" and unfenced.

Pending broker controlled shutdown duration: The time between when the controller updates the controlled shutdown offset for a broker and when the broker leaves controlled shutdown to the SHUTDOWN_NOW state.

Rationale

From the admin's perspective, it seems that the most important thing to monitor is the longest pending startup/shutdown duration, and the broker associated with it. Alerts fired on these metrics tell the admin if a roll is hanging or if the cluster is in a degraded state because of a broker startup/shutdown taking longer than reasonably expected. Tracking these metrics specifically, rather than a duration for every broker ensures that metric cardinality is independent of the number of brokers in a cluster while still capturing the desired information on the controller-side about brokers that may not be producing metrics yet.

Additional Metrics/Alternatives

Whether or not the cluster is in a degraded state (i.e. >= 1 broker fenced from the controller's POV) can also be reported. This metric would not be specific to broker startup or controlled shutdown, and is less descriptive than the ones above, but nevertheless still offers important monitoring information.

Proposed Changes

These metrics can be implemented using two monotonically increasing LinkedHashMaps mapping from brokerID → registrationTimeMs/controlledShutdownBeginTimeMs.

Entries are put in at the beginning of the durations defined above and removed at the end.

These metric values are updated whenever the controller processes a BrokerHeartbeat. During this processing, updating the longest duration takes constant time, since getting the oldest entry and removing entries are both O(1).

Modifications on these maps are thread-safe since they only occur on the main controller thread.

Compatibility, Deprecation, and Migration Plan

These will be newly exposed metrics and there will be no impact on existing kafka versions.

Test Plan

We will add junit tests to verify the new metrics.

These duration metrics would rely on soft state, so they would be reset when controller failover occurs. This is a bit confusing, and the scope of these metrics is limited to startup and controlled shutdown scenarios. Instead, more general purpose metrics would be applicable across more use cases.

However, if we want to add another value to BrokerRegistrationState  that maps to starting up brokers (i.e. never unfenced), this would require adding a boolean to the broker's registration record. Additionally, if we want to track is a broker has been uncleanly shutdown, we would likely need to store the broker epoch in its registration record.

...