Status
Current state: Under Discussion
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Single Message Transforms (SMT), KIP-66, have greatly improved Connector's usability by enabling processing input/output data without the need for additional streaming applications. These benefits have been limited as most SMT implementations are limited to fields available on the root structure:
Therefore, this KIP is aimed to include support for nested structures on the existing SMTs — where this makes sense —and to include the abstractions to reuse this in future SMTs.
Public Interfaces
From the existing list of SMTs, there are the following to be impacted by this change:
New configuration flags
Add a new configuration flag: field.style
to enable nested and potentially other styles to iterate schemas and apply.
Accepted values:
plain
(default): SMTs will access fields as it is today, with no lookup for nested fields.nested
: if the dotted notation is used, then SMTs will look up for nested fields.
SMTs affected
Extending the support for field configuration for dotted separation:
Cast
: extendspec
to support nested notation.ExtractField
: extendfield
to support nested notation.HeaderFrom
: extendfields
list to support nested notation.MaskField
: extendfields
list to support nested notation.ReplaceField
: extendinclude
andexclude
lists to support nested notation.TimestampConverter
: extendfield
to support nested notation.ValueToKey
: extendfields
list to support nested notation.InsertField
: Extend field configs to support nested notation.
Will require additional configurations:
HoistField
: add ahoisted
config to point to a specific path to hoist.- For example:
hoisted = nested.val field = line value (before): { "nested": { "val": 42, "other val": 96 } } value (after): { "nested": { "line": { "val": 42, }, "other val": 96 } }
These SMTs do not require nested structure support:
Drop
: Drop the whole key or value.Filter
: Drops the whole message based on a predicate.InsertHeader
: Insert a specific message to the header.RegexRouter
: Acts on the topic name.SetSchemaMetadata
: Acts on root schema.TimestampRouter
: Acts on timestamp.Flatten
: Acts on the whole key or message.
Proposed Changes
Nested notation
Dotted notation nested.key
tends to be the most natural way to describe nested fields as part of the configuration.
Though, schemaless (Map<String, Object
) records can have a dotted notation included on their field names (e.g. { 'nested.key': { 'val':42 } }
).
As the scenarios where the dotted notation is used on JSON messages could be rare, this KIP proposed to stick with dots as separators.
For scenarios where dotted notations are present on JSON messages, an escape backlash approach is proposed:
"this.field" (which would refer to the nested field "field" under the top-level "this" field)
"this\.field" (which would refer to the field named "this.field")
Compatibility, Deprecation, and Migration Plan
Existing SMT configurations should work fine unless they are using schemaless JSON records relying on dotted notation. This will need to be assessed as part of the KIP discussion.
If further requests to support other values arrive, we should consider extending the configuration with a nested delimiter that should be restricted to a set of few values.
Renaming fields SMT could also be used as a workaround to replace dot-named fields on JSON messages.
Rejected Alternatives
Keep ExtractField
as it is and repeat it until reaching nested fields
This KIP proposes to simplify this configuration by replacing multiple invocations with one.
Potential KIPs
Future KIPs could extend this support for:
- Recursive notation: name a field and apply it to all fields across the schema matching that name.
- Access to arrays: Adding
[]
notation to represent access to arrays and applying SMTs to fields within an array.