Status

Current state: Under Discussion

Discussion thread: [link]

JIRA: [link]

Released: 0.5.0

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

By default, DL uses zookeeper as the subscription store. However, zookeeper is not a write-scalable service. So the subscription offset update can not be too frequent, otherwise it would overwhelm the zookeeper service. Besides that, in production environment, it is encouraged to not use zookeeper in a write heavy workload.

Instead, DL/BK is designed for write heavy workload - The log/log segment is naturally good for keeping the history of subscription updates. 

Public Interfaces

No changes to existing public interface.

Proposed Changes

Introduce a log segment based implementation of SubscriptionsStore.

For each subscription in the SubscriptionsStore, it will use a log segment to store the updates for this subscription.

  • Metadata Change
    • This change will still use a znode to represent a subscription for backward compatibility. However, instead of storing the subscription updates directly to the znode, the znode will be changed to store the log segment id for the active log segment used by this subscription to store the subscription updates.
    • This change will change to use either protobuf or thrift to store the subscription metadata for extendability in future.

  • The proposed format for subscription metadata will be:

    message Subscription {
	string name	= 1;
	long segment_id = 2;
	bytes last_commit_dlsn = 3;
}
 
message AdvanceLastCommitDLSN {
	bytes commit_dlsn = 1;
}
  • The DLSN for the subscription is either stored in the last record in the log segment or stored as `last_commit_dlsn` in the subscription metadata.
    • last_commit_dlsn will be updated and segment_id will be set to empty when rolling into a new log segment or gracefully shutting down the subscription store.

Configuration Change


There will be a new configuration setting "logSegmentBasedSubscriptionsStoreEnabled". By default it will be disabled. When it is set to true, it would enable log segment based subscriptions store.

Compatibility, Deprecation, and Migration Plan

  • The code change will be backward compatible.
  • The rollout sequence will be
    • Update all the readers that use subscription store with the latest code and with logSegmentBasedSubscriptionsStoreEnabled disabled.
    • Update all the readers with new configuration with logSegmentBasedSubscriptionsStoreEnabled enabled

Test Plan

N/A

Rejected Alternatives

N/A

 

  • No labels