Page History

...

To support multiple keys for reading encrypted data it is required to store a key identifier on each encrypted page and on each encrypted WAL record (see more details). The key identifier is a sequential counter and should be the same on all nodes.

1. Start distributed process CACHE_GROUP_KEY_CHANGE_PREPARE, each node
   1. verifies that re-encryption is not in progress for the specified cache group.
      
   2. ensures that new key identifier does not exist
2. After successful completion of PREPARE, start distributed process CACHE_GROUP_KEY_CHANGE_FINISH, each node
   1. saves logical WAL record (ENCRYPTION_STATUS_RECORD) with current groups and key identifiers to start re-encryption on logical recovery.
   2. save the new key in metastore (as inactive key)
   3. adds new key and sets it for writing
   4. adds the mapping "WAL segment -> *old* key identifier" (to safely cleanup previous key in the future)
   5. saves logical WAL record (ENCRYPTION_STATUS_RECORD) with current page count in partitions.
   6. stores current page count as total pages for background re-encryption on partitions.
   7. starts background re-encryption of an existing data.

...