Div | ||
---|---|---|
| ||
HIP-1 : CSV Source Support for Delta Streamer |
Table of Contents | ||||
---|---|---|---|---|
|
Proposer
...
Approver
- <approver1 JIRA username> Vinoth Chandar : [APPROVED/REQUESTED_INFO/REJECTED]<approver2 JIRA username>
- Balaji Varadarajan : [APPROVED/REQUESTED_INFO/REJECTED]
- ...
Status
Current state:[One of "Under Discussion", "Accepted", "Rejected"] Status colour Yellow title In Progress
Discussion thread: here
JIRA: here
Released: <Hudi Version>
Abstract
<Describe the problem you are trying to solve and a brief description of why it’s needed>
Background
<Introduce any much background context which is relevant or necessary to understand the feature and design choices.>
Implementation
<Describe the new thing you want to do in appropriate detail, how it fits into the project architecture. Provide a detailed description of how you intend to implement this feature.This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.>
Rollout/Adoption Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Test Plan
...
N/A
Prior doc link : https://docs.google.com/document/d/1bj-xpkRomVtbzvLb_4BRngDIGkkMR5yzxXRRzkA7QVo/edit#heading=h.di66rda5xhp2
Abstract
Hudi delta Streamer does not have direct support for pulling data in CSV format from kafka/HDFS-logs. The only possible alternative to ingesting CSV data to Hudi dataset is to first convert them into json/avro before pulling in through delta-streamer. This RFC proposes a mechanism to directly support sources in CSV format.
Background
Spark supports loading a CSV file into a dataset of rows (example). Hudi has a RowSource that handles datasources with Rows. The idea is to transform CSV to Row format using Spark's functionality by passing in counterpart CSV options and use existing logic to go from Rows.
Currently, the conversion from Row to Avro in Hudi is not efficient. However, the improvement on this is on the way, so wrapping Spark's CSV reader makes the implementation easy and extensible compared to parsing CSV and constructing the record from scratch.
Implementation
Implementing a CSVSource by extending RowSource (similar to JDBCSource in this PR)
- Use `spark.read.format("csv")` as the reader of CSV files, using Spark's internal CSV parsing logic to convert it to Dataset of Rows
- Define new Hudi configurations related to the CSV source to be on par with Spark CSV options, and pass these configs to the reader through `.option()`
- The smallest unit of incremental pull will be one CSV file. Assume the CSV files are named after timestamps which are monotonically increasing, the filename of the last ingested CSV can be taken as the last checkpoint. A directory that holds all CSV files to be ingested will be given in the config.
Rollout/Adoption Plan
There won’t be any impact for existing users. This is just a new feature.
- New configurations will be added to the documentation.
Test Plan
- Add a new unit test class for the CSVSource
- Run integration tests around CSVSource
- Add one CSVSource example in the demo