...

1. Start distributed process CACHE_GROUP_KEY_CHANGE_PREPARE, each node
   1. verifies that re-encryption is not in progress for the specified cache group.
      
   2. ensures that new key identifier does not exist
2. After successful completion of PREPARE, start distributed process CACHE_GROUP_KEY_CHANGE_FINISH, each node
   1. saves logical WAL record (ENCRYPTION_STATUS_RECORD) with current groups and key identifiers to start re-encryption after logical recovery.
   2. save the new key in the metastore (as inactive key)
   3. sets it for writing
   4. adds the mapping "WAL segment -> *old* key identifier" (to safely cleanup previous key in the future)
   5. save current keys and WALl mappings into the metastore.
   6. starts background re-encryption of an existing data.

...

Scan all pages from specified range (metaPageId + [offset -> total])

1. acquire pageif the checkpoint is finished (after key change) and page is dirty - skip this page
   .
   1. if the checkpoint is not finished or page is not dirty
      1. lock page
      2. unlock page (dirty=true)
2. release page

...

Reencryption status requires an additional 8 bytes on the meta page of each partition.
Index partition uses PageMetaIO to read/write meta information (page type T_META).
Each other partition uses PagePartitionMetaIO to read/write meta information (page type T_PART_META).

Partition meta starts just after the end of the page meta.

To store an additional 8 bytes partition meta shifted by 8 bytes.support binary compatibility and keep code clean we creating a new successor of PageMetaIO - PageMetaIOV2 with the same type T_META.

We converting all existing T_META pages into a new version.

We storing additional 8 bytes at the end of each T_META and T_PART_META memory pages.

WAL delta records have also been modified to store re-encryption status.

...

• Background re-encryption may affect performance. Performance impact can be managed using the following configuration options:
  1. reencryptionThreadCnt - number of threads used for re-encryption.
     
  2. reencryptionBatchSize - number of pages that are scanned during re-encryption under checkpoint lock.
     
  3. reencryptionRateLimit - page scanning speed limit in megabytes per second.
     
  4. reencryptionThreadCnt - number of threads used for re-encryption(?).
• The WAL history can be not enough to store all entries between checkpoints (this should be carefully tuned by properly setting the size of the WAL history and tuning the re-encryption performance).
• The WAL history (for delta rebalancing) may be lost for all cache groups due to background re-encryption.

Process management

...

Public API changes

IgniteEncryption

...

• ReencryptionPagesLeft - (long) Total pages left for reencryption.
• ReencryptionFinished - (boolean) Indicates whether reencryption re-encryption is finished or not (it will set to true only when a checkpoint is finished).

Process management

The following commands should be added to the control.sh utility:

Rotate encryption key.

control.(sh|bat) --encryption change_cache_key cacheGroupName --yes

The encryption key has been changed for cache group "default".

View encryption key identifiers.

control.(sh|bat) --encryption cache_key_ids cacheGroupName

Encryption key identifiers for cache: default
  Node 6085d500-2736-4c1f-b47c-444cf0a00000:
    1 (active)
    0
  Node d98654c0-6dfb-4996-993e-387156300001:
    1 (active)
    0

View cache group re-encryption status.

control.(sh|bat) --encryption reencryption_status cacheGroupName

  Node 4ed26231-f92d-4b1c-86ba-7a117c200001:
    1552 KB of data left for re-encryption
  Node 89a456e5-59c5-4f13-a75b-39ab25000000:
    1552 KB of data left for re-encryption

Suspend cache group re-encryption.

control.(sh|bat) --encryption suspend_reencryption cacheGroupName

  Node ad1328e7-11e0-4ecb-8ef2-066519e00001:
    re-encryption of the cache group "default" has been suspended.
  Node 2a9e291f-e2d1-46e3-9954-18deb0e00000:
    re-encryption of the cache group "default" has been suspended.

Resume cache group re-encryption.

control.(sh|bat) --encryption resume_reencryption cacheGroupName

  Node 2ed43509-caab-48dc-a27d-3be65d800000:
    re-encryption of the cache group "default" has been resumed.
  Node b52d6451-a948-48d5-b79a-411956700001:
    re-encryption of the cache group "default" has been resumed.

View/change re-encryption rate limit.

control.(sh|bat) --encryption reencryption_rate [limit]

Parameters:
    limit  - decimal value to change rate limit (MB/s)

  Node 15cb8485-0c09-4361-b267-107d38400000:
    re-encryption rate has been limited to 0.01 MB/s.
  Node 909ed414-22e6-477b-b2ca-d1934cd00001:
    re-encryption rate has been limited to 0.01 MB/s.

Reference Links

1. PCI DSS Requirements and Security Assessment Procedures
   https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-2-1.pdf
2. How Often Do I Need to Rotate Encryption Keys on My SQL Server?
   https://info.townsendsecurity.com/bid/49019/How-Often-Do-I-Need-to-Rotate-Encryption-Keys-on-My-SQL-Server
3. PCI DSS and key rotations simplified
   https://www.crypteron.com/blog/pci-dss-key-rotations-simplified/
4. Transparent Data Encryption in MS SQL Server
   https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/transparent-data-encryption?view=sql-server-ver15
5. Oracle Transparent Data Encryption FAQ
   https://www.oracle.com/database/technologies/faq-tde.html
6. InnoDB Data-at-Rest Encryption
   https://dev.mysql.com/doc/refman/8.0/en/innodb-data-encryption.html
7. Transparent data encryption feature proposed in pgsql-hackers.
   https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Key_Rotation

...