Status

Current state: Under Discussion

Discussion thread: https://lists.apache.org/thread/lxfq8y8co76mjwxhowwx6cjl0hw1mxpy

JIRA:  Unable to render Jira issues macro, execution error.

Motivation

JsonConverter is used by Connect to convert raw JSON payloads, in bytes, to and from Connect schemas an values objects. When converting Connect objects to bytes, if the value is null and the field has a default value, JsonConverter always emits the default value. In some cases, for example when the value is explicitly set to null, it would be preferable to keep the null value instead.

In particular, if we use Debezium to stream changes out of a table, t1, in mysql:

create table t1 {
   name varchar(40) not null,
   create_time datetime default '1999-01-01 11:11:11' null,
   update_time datetime default '1999-01-01 11:11:11' null
}

and then insert one record into table t1,

INSERT INTO `t1` (`name`, `update_time`) VALUES ('kafka', null);

the record will store in MySQL as:

{
  "name": "kafka",
  "create_time": "1999-01-01 11:11:11",
  "update_time": null
}

but the JsonConverter will always emit

{
  "name": "kafka",
  "create_time": "1999-01-01 11:11:11",
  "update_time": "1999-01-01 11:11:11"
}

It would be beneficial to be able to specify whether fields with a default value and set to null should be converted to null or the default value.

Public Interfaces

Add two configurations to JsonConverter:

Name: serialize.use.optional.null
Description: Whether to serialize fields that have a default value and that are null to the default value or to null. When set to false, the default value is used, otherwise null is used.
Type: Boolean
Default: false

Name: deserialize.use.optional.null
Description: Whether to deserialize fields that have a default value and that are null to the default value or to null. When set to false, the default value is used, otherwise null is used.
Type: Boolean
Default: false

Proposed Changes

Update JsonConverter and JsonConverterConfig to handle the new configurations.

Compatibility, Deprecation, and Migration Plan

The new configurations keep the current behavior by default so it will not affect existing users. Users can optionally opt-in the new behavior by setting the configuration to true.

Rejected Alternatives

1. Always take null on an optional null field which has default value. It will break the compatibility and only cover partial cases.
2. Add the configuration to ConverterConfig