Status
Current state: UNDER DISCUSSION and WIPACCEPTED
Vote: http://mail-archives.apache.org/mod_mbox/samza-dev/201909.mbox/browser
Discussion thread: http://mail-archives.apache.org/mod_mbox/samza-dev/201812.mbox/%3CCABpE9c1_N4hFBsj7Yr5v_7q89SVwnm6Zr-MGuLz1e%3De8MchLkg%40mail.gmail.com%3E
JIRA:
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
Authors: Daniel Nimishura, Shanthoosh Venkataraman
Released: TBDSamza-1.4
Table of Contents exclude Status
...
Metadata Store
Referenced in the General Workflow above.
The out-of-band metadata store used is described by the metadata store abstraction feature (SAMZA-1786) from SEP-11. The Startpoints are stored within its own namespaces in the metadata store.
...
Referred to in Step 7 of the Loading Startpoints Upon Job Startup section above.
Code Block | ||||
---|---|---|---|---|
| ||||
public interface SystemAdmin { ... /** * Resolves the startpoint to a system specific offset. * @param startpoint represents the startpoint. * @param systemStreamPartition represents the system stream partition. * @return the resolved offset. */ String resolveStartpointToOffset(SystemStreamPartition systemStreamPartition, Startpoint startpoint); ... } |
...
A key part of the core Startpoint feature is for individual task instances to fetch the appropriate Startpoint keyed by SSP-only. The two approaches, fan-out and intent-ACK, have been explored with the analysis detailed in the following subsections. The fan-out strategy is favored over the intent-ACK strategy. See analysis and explanation below.
Fan-out
See General Workflow above for details
Pros
- Follows the natural progression of the JobCoordinator calculating the job model and then applying the info in the job model to fan out the SSP to SSP-Task Startpoints.
- Cleaner and simpler book keeping of Startpoints. SSP-only keyed Startpoints are deleted after fan out and SSP+TaskName keyed Startpoints are deleted upon offset commits.
...