Versions Compared

Old Version 9

changes.mady.by.user Nisarg Shah

Saved on

compared with

New Version 10

changes.mady.by.user Nisarg Shah

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Certain datasets require modifications to the messages either coming into Kafka (source) or going to another data store (sink). For example, we may require to remove user identifiable information like SSN from Kafka messages while exporting data into a Hadoop cluster for analysis.

Currently, the "data transfer" logic is locked within a single method of the Task classes, thus Kafka Connect API is inflexible for making such any modifications. This is more so when a Kafka Connect API is provided via third party where the user cannot modify the source. Custom modifications from user to user would require modifying the source of Connect API. 

...