...

// TODO – main background/motivation. Can start with what's in the ticket description and build off of that 

We would like to address a gap in the API right now when it comes to stream-stream joins. For a quick refresher, these are a special case in the DSL right now, and actually cannot be configured via the default.dsl.store config

Public Interfaces

The new feature will be accessed via a new config, defined below. This will take precedence over the old default.dsl.store config, which will only be used if the new config is not set. We will also deprecate the default.dsl.store config to make it clear which of the two should be preferred. Further discussion on this point can be found under "Rejected Alternatives"

...

1. Don't deprecate the old default.dsl.store config and instead maintain it alongside the new DSLStoreProvider config. The advantage here is obviously that we minimize disruption to anyone using this config already. However I believe the existing userbase is likely to be quite small, as in its current form this feature is only useful to those who (a) have many state stores to the point overriding them separately is cumbersome, and (b) also wishes to override all (or most) of their stores to be in-memory specifically. This seems to be a relatively uncommon pattern already, not least of which being that it's a recipe for OOM disaster unless their applications are extremely small and/or low traffic. In the absence of any evidence that this config is already in use despite such a niche application, it seems best to take this opportunity to deprecate it now, before it gains any more traction, and have everyone converge on the same, single configuration going forward. Whenever there are two configs affecting the same underlying feature, it's inevitable there will be confusion over their interaction/hierarchy which can often lead to misconfiguration. Deprecating the old config is a clear sign to users which of the two should be preferred and will take precedence.
2. Instead of positioning the new config as a full replacement for default.dsl.store, we could instead introduce it as a complementary config for custom stores specifically. For example, add a 3rd StoreType to the enum definition, such as CUSTOM. If (and only if) the CUSTOM StoreType is used, the value of the new default.dsl.store.provider config will be looked at and used to obtain the actual StoreSupplier instances of this custom type. This option has the same advantages as the above, though I would qualify it in the same way. In general, having two configs that do the same thing or are tied to the same feature will be confusing to new (or old!) users. However, between these two alternatives I would personally advocate for this one as the better option, as it at least solves the concern over potential misconfiguration due to unclear interaction between the two configs.
3. On a different front, we should also consider the alternatives presented by for the Materialized API. First, it may make sense to also introduce a new Materialized.as(DSLStoreProvider) API. Doing this would bring the interface of Materialized into closer alignment with the public interface presented by the change in default DSL store configs. The bigger argument is of course that it would bring custom states into full feature parity with the OOTB flavors, with the benefits here being all the same ones offered by the Materialized.as(StoreType) API introduced alongside Arguably, we could deprecate the default.dsl.store config (1. that it's a much more ergonomic API compared with the actual StoreSuppliers, and 2. being that it saves the user from havingLastly, we could also deprecate the Materialized.as(StoreType) API and correspondingly, the StoreType enum itself. However, but not the two Materialized APIs related to the StoreType enum. I would be more open to leaving these two APIs as-is than to leaving the config, as I don't think (all of) the same arguments as in from Alternative #1 apply to this case:. First, the userbase here is likely to be much wider, as it may include any applications which wish to override only a handful of stores to be in-memory (a considerably more practical and realistic setup). This could even include users who set the default.dsl.store to IN_MEMORY but wanted to keep some subset of stores on rocksdb. AlsoSecond, even if the size of the userbase was exactly the same, the magnitude of the disruption would not be: deprecating default.dsl.store means each user/application has to change at most a single line of code to upgrade onto the new config. But deprecating Materialized.as(StoreType) would mean having to reconfigure each and every materialized operation that was using this, requiring a theoretically unbounded number of code changes per app.
   Of course, it would still be a very simple migration with a predictable pattern that one could easily achieve with a find-and-replace tool or or sed command. It would be great if some users can weigh in during the KIP discussion for some basic data, even anecdotal, on how heavily this is/isn't used. Without reason to believe this would be a major disruption, it seems best to deprecate it now, before the usage becomes any more widespread, and position ourselves with a nice, clean API going forward.