Motivation

Currently, when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal-breaker for microservice use cases. Basically, as soon as a task is assigned to the new node, it initializes the store and opens it. Post it the task is moved to Restoring state where it could take time to restore all the data from the changelog topic depending upon data in that partition and then it will transition to the Running state and is available to serve the queries. As of now, the tasks serve data only when they are in the Running state.  

I am proposing to serve data in restoring state as well so that queries which hit this partition to fetch the data via api while it is restoring can serve stale data while it is restoring and not fail the query which currently happens because the store is not available for that partition. 

Public Interfaces

• None

Proposed Changes

Add another loop in StreamThreadStateStoreProvider.java and add the restoring tasks in the returned stores list which currently only adds running tasks. This will add even the restoring tasks to list in which which the key will be further checked in CompositeReadOnlyKeyValueStore.java

Compatibility, Deprecation, and Migration Plan

Since this doesn't affect any current public class, no migration or version upgrade is needed. But we can, however, leave it to api users to choose if they want to be served from restoring state or not(basically they are ok with stale reads or not). For this, we might have to pass on another argument to QueryableStoreProvider and to StreamThreadStateStoreProvider.java in the stores() based on which we can choose whether to add restoring stores in the list or not.

Rejected Alternatives

The suggested changes are fairly simple. No other potential alternatives have been considered.