Page History

...

By Default, we use the key and value in IncomingMessageEnvelop to store an AD data item in a store, user provides serde as done today. Configuration options will be provided to to allow user to

• construct user defined keys
• construct user defined values

User defined objects will be treated as POJO's, and we may provide a default serde for POJO.

Configuration

Key classes

AdjunctDataStoreManagerFactory

...

/**
 * An AdjunctDataStoreManager instance is responsible for
 * 1. maintaining the mapping between system streams and adjunct data stores
 * 2. extracting the key and value from an IncomingMessageEnvelop
 * 3. populating adjunct data stores
 */
public interface AdjunctDataStoreManager {
   /**
    * Invoked before a message is passed to a task
    * @returns true if the message is saved in a store, false otherwise
    */
    AdjunctDataStoreManager boolean process(IncomingMessageEnvelop message);
}

/**
 * A factory to instantiate converterskey/value extractors
 */
public interface ConverterFactoryKeyValueExtractorFactory {
    ConverterKeyValueExtractor getConvertgetKeyValueExtractor(Config config);
}

/**
 * A converterkey/value convertsextractor anthat inputextracts objectkey toand anothervalue type
 */
public interface Converter {from in incoming
 * message /**envelop
    * Converts an input object to another type
    **/
public interface Converter {
    *Object @param input the object to be converted
    * @returns the converted object
    */
getKey(IncomingMessageEnvelop input);
     Object convertgetValue(ObjectIncomingMessageEnvelop input);
}

No changes to public interface

...

For bounded datasets delivered (such as files) as streams, the adjunct data store may need to maintain multiple version of underlying stores in order to guarantee serving of a consistent version. This would require the abstraction of adjunct data store, which manages internally versions of underlying stores.

Key extractor and value converter

Another option considered is to embed key extraction and value conversion logic in serde layer. It would work, but would make serde asymmetric and less portable.