...

Proposers

• Raymond Xu

Approvers

• @<approver1 JIRA username> : [APPROVED/REQUESTED_INFO/REJECTED]
• @<approver2 JIRA username> Vinoth Chandar  : [APPROVED/REQUESTED_INFO/REJECTED]
• ...

Status

Current state: 

	Current State
Status title Under Discussion

Status colour Yellow title In Progress
Status colour Red title ABANDONED
Status colour Green title Completed
Status colour Blue title INactive

Discussion thread: here

JIRA:

...

{
  "type": "record",
  "namespace": "org.apache.hudi.common",
  "name": "ErrorRecord",
  "fields": [
    {
      "name": "uid",
      "type": "string"
    },
    {
      "name": "ts",
      "type": "string"
    },
    {
      "name": "schema",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "record",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "message",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "context",
      "type": ["null", {"type": "map", "values": "string"}],
      "default": null
    }
}

• `uid`: uuid for the error record
• `ts`: creation unix timestamp for the error record
• `schema`: original schema for the record if any
• `record`: original serialized record in json if any
• `message`: additional message or any string like error stacktrace to be attached

Errors table

...

• `context`: kv pairs for any related context info like commitTime, tableName, partitionpath, recordKey, etc

Errors table

Users can configure, based on their preferences, error tables as local or global ones.

Local error table

By default, if error table is enabled, it will be a local error table. Failed records will be written to a local Hudi table alongside with the original Hudi table with a suffix (configurable) like `_errors`.

Global error table

To write to a global error table, users can configure `hoodie.write.error.table.base.path=<some file system path>` and `hoodie.write.error.table.name=foobar`. If either of these 2 configs were set, error table is set to global mode and `hoodie.write.error.table.suffix` will be omitted.

Configurations

...

Write path

Start with

• org.apache.hudi.client.HoodieWriteClient#postWrite

• org.apache.hudi.client.HoodieWriteClient#completeCompaction

...

• Emit a count metric for the number of failed records

Rollout/Adoption Plan

• Add a writer config to Use configuration turn on this handling feature `hoodie.write.handleerror.failedtable.records`enabled=true`
• Default to false for smooth roll-out

Test Plan

...

• Functional test cases to cover both local and global cases.