...
| Current State |
---|
| ![(tick)](/confluence/s/34gb19/8804/10mvnxf/_/images/icons/emoticons/check.svg) |
|
Status |
---|
colour | Yellow |
---|
title | In Progress |
---|
|
| ![(tick)](/confluence/s/34gb19/8804/10mvnxf/_/images/icons/emoticons/check.svg) |
|
|
Status |
---|
colour | Green |
---|
title | Completed |
---|
|
|
|
|
|
...
Code Block |
---|
language | text |
---|
title | error record schema |
---|
|
{
"type": "record",
"namespace": "org.apache.hudi.common",
"name": "ErrorRecord",
"fields": [
{
"name": "uid",
"type": "string"
},
{
"name": "ts",
"type": "string"
},
{
"name": "schema",
"type": ["null", "string"],
"default": null
},
{
"name": "record",
"type": ["null", "string"],
"default": null
},
{
"name": "message",
"type": ["null", "string"],
"default": null
},
{
"name": "context",
"type": ["null", {"type": "map", "values": "string"}],
"default": null
}
} |
- `uid`: uuid for the error record
- `ts`: creation unix timestamp for the error record
- `schema`: original schema for the record if any
- `record`: original serialized record in json if any
- `message`: additional message or any string like error stacktrace to be attached
- `context`: kv pairs for any related context info like commitTime, tableName, partitionpath, recordKey, etc
Errors table
Users can choose to use local configure, based on their preferences, error tables as local or a global one, depending on their preferencesglobal ones.
Local error table
Users may apply `hoodie.write.error.table.global=false` (default) to Hudi writers such that the failed By default, if error table is enabled, it will be a local error table. Failed records will be written to a local Hudi table alongside with the target tableoriginal Hudi table with a suffix (configurable) like `_errors`.
Global error table
Users may apply To write to a global error table, users can configure `hoodie.write.error.table.base.path=<some file system path>` and `hoodie.write.error.table.global=true` to Hudi writers such that the failed records will be written to a global Hudi tablename=foobar`. If either of these 2 configs were set, error table is set to global mode and `hoodie.write.error.table.suffix` will be omitted.
Configurations
key |
| default |
---|
hoodie.write.error.table.enabled | set to true to activate error table handling feature | false |
hoodie.write.error.table.suffix | suffix for local error table name, stored alongside the target table. If the Hudi table is "foo", errored records will be saved to "foo_errors" at the same base dir as configured via `hoodie.base.path` | "_errors" |
hoodie.write.error.table. |
globalset to true to use global errors table | name | error table name | "hoodie_errors" |
false.global.base.path | base path for global error table | same as `hoodie.base.path` |
Write path
Start with
org.apache.hudi.client.HoodieWriteClient#postWrite
org.apache.hudi.client.HoodieWriteClient#completeCompaction
...