...
Table of Contents | ||||
---|---|---|---|---|
|
Proposer
...
- @rahuledavalath
Approver
- <approver1 JIRA username> Vinoth Chandar : [APPROVED/REQUESTED_INFO/REJECTED]<approver2 JIRA username>
- Balaji Varadarajan : [APPROVED/REQUESTED_INFO/REJECTED]...
Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here
JIRA: here
Released: <Hudi Version>
Abstract
<Describe the problem you are trying to solve and a brief description of why it’s needed>
Background
N/A
Prior doc link : https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit#heading=h.di66rda5xhp2
Abstract
Hudi delta Streamer does not have direct support for pulling data in csv format from kafa/HDFS-logs. The only possible alternative to ingesting Csv data to hudi dataset is to first convert them into json/avro before pulling in through delta-streamer. This HIP proposes a mechanism to directly support sources in csv format.
Background
Introduce <Introduce any much background context which is relevant or necessary to understand the feature and design choices.>
Implementation
...
Extend the DeltaStreamer by implementing a CSV Source(kafka/hdfs)
- We can use existing FilebasedSchemaProvider class for decoding csv data.
- If the Csv data does not contains header then, for field names in source schema we can use _c0,_c1,_c2..etc as the field names according to the record position in the csv data. (For kakfa data & non header csv files we can use this method).
- If header is present in the csv data then need to use header information for the field names.
- Need to introduce a configuration property to inform whether header is present or not.
- Need to introduce a new configuration property to support any configurable delimiters.
Rollout/Adoption Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Test Plan
There won’t be any impact for existing users. This is just a new feature
Test Plan
Unit and Manual Integration test<Describe in few sentences how the HIP will be tested. How will we know that the implementation works as expected? How will we know nothing broke?>