Page History

...

1. Start distributed process CACHE_GROUP_KEY_CHANGE_PREPARE, each node
   1. verifies that re-encryption is not in progress for specified cache group.
      
   2. ensures that new key identifier does not exist
2. After successful completion of PREPARE, start distributed process CACHE_GROUP_KEY_CHANGE_FINISH, each node
   1. adds new key and sets it for writing
   2. adds the mapping "WAL segment -> *old* key identifier" (to safely cleanup previous key in the future)
   3. saves logical WAL record (ENCRYPTION_STATUS_RECORD) with current page count in partitions.
   4. stores current page count as total pages for background re-encryption on partitions.
   5. starts background re-encryption of an existing data.

...

The process applies for all existing partitions including index.

Every time the cache group key changes, we store the current page count of the partition in the meta page (this value is used as the total page count to re-encrypt).

Scan all pages from specified range (metaPageId + [offset -> total])

1. acquire page
   1. if the checkpoint is finished (after key change) and page is dirty - skip this page.
   2. if the checkpoint is not finished or page is not dirty
      1. lock page
      2. unlock page (dirty=true)
2. release page

Re-encryption progress is stored into metapage (int offset, int total), it updates during the checkpoint.

...