Versions Compared

Old Version 4

changes.mady.by.user Vinoth Chandar

Saved on

compared with

New Version 5

changes.mady.by.user Bhavani Sudha

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Extracting change logs from MySQL : Surprisingly, this is  still a pretty tricky problem to solve and often Hudi users get stuck here. Thankfully, at-least for AWS users, there is a Database Migration service (DMS for short), that does this change capture and uploads them as parquet files on S3
  2. Applying these change logs to your data lake table :  Once there are change logs in some form, the next step is to apply them incrementally to your table. This mundane task can be fully automated using the Hudi DeltaStreamer tool. 

...