THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
- accommodate the needs of the existing Kafka sinks
- accommodate the needs of the new JDBC sink:
- commits are retried in case of transient failures instead of failing the job
- rollbacks are retried
- need to distinguish between transactions started during this run and restored from the state; ignore commit failures (with reason “unknown”) for the latter; this is a consequence of a lack of timeouts
- when committing a group of transactions: an option to stop commits as soon as one failed; otherwise consistency can be violated (if the failure was transient then failed commit and all the further commits will be retried later)
- transaction timeouts aren’t used to ignore commit failures, as most DBs don’t support them
state will probably need to include all to-commit transactions (as union list)- minor API changes required
- accommodate the needs of other 2PC-sinks in future; these could be existing file sink, WAL; or potential DynamoDb, pulsar
- and non-sinks (see this question)
- improve testability; currently, TwoPhaseCommitSinkFunction requires a lot of mocking
...