THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- @<approver1 JIRA username> : [APPROVED/REQUESTED_INFO/REJECTED]
- @<approver2 JIRA username> : [APPROVED/REQUESTED_INFO/REJECTED]
- ...
Status
Current state:
| Current State | |||||||||
|---|---|---|---|---|---|---|---|---|---|
|  | ||||||||
| |||||||||
| |||||||||
| |||||||||
|
Discussion thread: here
JIRA: Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HUDI-648
...
- `uid`: uuid for the error record
- `ts`: creation unix timestamp for the error record
- `schema`: original schema for the record if any
- `record`: original serialized record in json if any
- `message`: additional message
Errors table
...
Users can choose to use local error tables or a global one, depending on their preferences.
Local error table
Users may apply `hoodie.write.error.table.global=false` (default) to Hudi writers such that the failed records will be written to a local Hudi table alongside with the target table.
Global error table
Users may apply `hoodie.write.error.table.global=true` to Hudi writers such that the failed records will be written to a global Hudi table.
Configurations
| key | default | |
|---|---|---|
| hoodie.write.error.table.enabled | set to true to activate error table handling feature | false |
| hoodie.write.error.table.suffix | suffix for local error table name, stored alongside the target table. If the Hudi table is "foo", errored records will be saved to "foo_errors" at the same base dir as configured via `hoodie.base.path` | "_errors" |
| hoodie.write.error.table.global | set to true to use global errors table | false |
| hoodie.write.error.table.global.base.path | base path for global error table | same as `hoodie.base.path` |
Write path
Start with
org.apache.hudi.client.HoodieWriteClient#postWrite
org.apache.hudi.client.HoodieWriteClient#completeCompaction
...
- Emit a count metric for the number of failed records
Rollout/Adoption Plan
- Add a writer config to Use configuration turn on this handling feature `hoodie.write.handleerror.failedtable.records`enabled=true`
- Default to false for smooth roll-out
Test Plan
...
- Functional test cases to cover both local and global cases.