You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Next »

Status

Current state: Under discussion

Discussion thread: https://sematext.com/opensee/m/Kafka/uyzND1VU1Ou1y0Lbh?subj=+DISCUSS+KIP+466+Add+support+for+List+lt+T+gt+serialization+and+deserialization

JIRA:  Unable to render Jira issues macro, execution error.

Motivation

I believe there are many use cases where List Serde could be used.

Ex. https://stackoverflow.com/questions/41427174/aggregate-java-objects-in-a-list-with-kafka-streams-dsl-windows, https://stackoverflow.com/questions/46365884/issue-with-arraylist-serde-in-kafka-streams-api For instance, aggregate grouped (by key) values together in a list to do other subsequent operations on the collection.

Public Interfaces

  • New class org.apache.kafka.common.serialization.ListSerializer which implements the Serializer<List<T>> interface
  • New class org.apache.kafka.common.serialization.ListDeserializer which implements Deserializer<List<T>> interface
  • New subclass ListSerde<T> in org.apache.kafka.common.serialization.Serdes which creates new serde based on ListSerializer and ListDeserializer classes
  • New method public static <T> Serde<List<T>> ListSerde(Serde<T> innerSerde) in org.apache.kafka.common.serialization.Serdes class

Proposed Changes

This KIP proposes adding new ListSerializer and ListDeserializer classes as well as support for the new classes into the Serdes class. This will allow using List<T> Serde directly from Consumers, Producers and Streams.

List<T> serialization and deserialization will be done through repeatedly calling a serializer/deserializer for each entry provided by passed generic T's Serde. For example, if you want to create List of Strings serde, then serializer/deserializer of StringSerde will be used to serialize/deserialize each entry in `List<String>`.

Proposed Configurations

List serde is an unusual type of serde because we need to consider two things here: the implementation of List (i.e. ArrayList, LinkedList, etc) and enclosed elements' type.

First, as usually we need to specify our list serde using:

default.key/value.serde = org.apache.kafka.common.serialization.Serdes$ListSerde

Then, we need to cover those two configurations and here I'm proposing these two extra properties:

default.key/value.list.serde.impl = java.util.ArrayList

default.key/value.list.serde.element = org.apache.kafka.common.serialization.Serdes$IntegerSerde

Properties default.key/value.list.* will be ignored as long as default.key/value.serde is not set to org.apache.kafka.common.serialization.Serdes$ListSerde

Compatibility, Deprecation, and Migration Plan

Does not apply

Rejected Alternatives

Not known

  • No labels