Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Status

Current stateAcceptedAdopted

Discussion thread: Link

JIRA

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-8326

...

I believe there are many use cases where List Serde could be useful.:

...

For the performance purposes the following serialization strategy was strategies were put in place:

enumSerializationStrategy {
    CONSTANT_SIZE,
    VARIABLE_SIZE;
}

. Depending on the type of an inner serde (a list's element type) the serialization will be performed in the following ways:

  1. If For SerializationStrategy.CONSTANT_SIZE, if an inner serde has one of the following serializers (ShortSerializer.class, IntegerSerializer.class, FloatSerializer.class, LongSerializer.class, DoubleSerializer.class, UUIDSerializer.class), then the final payload will not contain each element's size encoded since sizes of presented types are static (2 bytes, 4 bytes, 8 bytes, etc.)
  2. If For SerializationStrategy.VARIABLE_SIZE, if the inner serde doesn't have one of the serializers listed above, then a size of each element will be encoded in the final payload (see below)

Additionally, there are two different ways of serializing NULL values within the payload:

  1. For SerializationStrategy.CONSTANT_SIZE, the list serializer will generate a null index list that contains indexes of all null entries within the payload
  2. For SerializationStrategy.VARIABLE_SIZE, the list serializer instead will write Serdes.ListSerde.NULL_ENTRY_VALUE (-1 by default) for the size of a null entry
                                                                                                            
        CONSTANT_SIZE                                    VARIABLE_SIZE                                      
                                                                                                            
+----------------------------+                   +----------------------------+                             
|    SerializationStrategy   |                   |    SerializationStrategy   |                             
|            Flag            |                   |            Flag            |                             
|----------------------------|                   |----------------------------|                             
|    NullIndexList.size()    |                   |     PayloadList.size()     |                             
|----------------------------|                   |----------------------------|                             
|        Null index 1        |                   |      Size of entry 1       |                             
|----------------------------|                   |----------------------------|                             
|        Null index 2        |                   |                            |                             
|----------------------------|                   |          Entry 1           |                             
|            ...             |                   |                            |                             
|----------------------------|                   |----------------------------|                             
|     PayloadList.size()     |                   |      Size of entry 2       |                             
|----------------------------|                   |----------------------------|                             
|                            |                          |                            |                             
|          Entry 1     Case  1    |                   |      Case 2   Entry 1           
|                             
|                            |               
    |     +------------------+                    +------------------+   |     
         |                  |        -
|----------------------------|       |                  |        
     Int |   Size of list   ||----------------------------|                       Int |   Size of list
|   |        
         |        |          |         |      |                  |    |    
         |------------------|               |------------------|  
|         
 Entry 2       |    |              |     |          |                  |        
         |     Entry 1      
|           Int |  Size of entry 1 |        
  |       |            |      |                |      |            |        
         |--------
|----------|               |------------------|          
         |                            |                             
|                            |          
         |     Entry  2      |               |           Entry    1      |        
|          |                  |               |    |            ...  |        
   |      |------------------|               |------------------|        
|          |                  |               |    |              |        
         |                  |           Int 
|   Size of entry 2 |        
         |    |              |     |          |                  |         
         |           
|       |     ...          |------------------|   |     
         |     |             |               |                  |        
   
|      |                  |    |           |     Entry 2  |    |        
         |       ... |       |               |       
|           |        
         |                   |               |------------------|        
     |    |                  |       
|        |                  |  |      
         |     |             |               |                  |        
   
|      |                  |    |           |       ... |       |        
         |    |              |               
|                   |        
 |        |           |       |               |      |            |        
         
+----------------------------+                   +----------------------------+                             

Compatibility, Deprecation, and Migration Plan

...