Page History

...

Instead of writing the new checkpoint message format to a new checkpoint topic (v2), we are writing to the same topic under a different namespace of the checkpoint topic (v1) is also possible for the migration. The checkpoint manager will selectively pull the checkpoints from the checkpoint topic which will match the key of the written checkpoint. The advantage of this approach is that the migration will no longer require creating a new Kafka topic. However, since we are planning to enable this for all users, the versions that are currently being used, could not have forward compatibility available for being able to read the new checkpoint format from the old checkpoint stream.

The steps for this rollout method will be the following:

1. Upgrade Samza users to a version (1.x) supporting read and write in the v2 checkpoint format to/from the same checkpoint topic
2. Writing checkpoint in both v2 and v1 format into the checkpoint topic
3. Enable the job to use the v2 format to restore the checkpoint
4. Ensure the job is stable with the new schema
5. Stop v1 format write to the checkpoint topic
   

State Backup and Commit Interface Changes

...