...
To support multiple keys for reading encrypted data it is required to store a key identifier on each encrypted page and on each encrypted WAL record (see more details). The key identifier is a sequential counter and should be the same on all nodes.
- Start distributed process CACHE_GROUP_KEY_CHANGE_PREPARE, each node
- verifies that re-encryption is not in progress for the specified cache group.
- ensures that new key identifier does not exist
- After successful completion of PREPARE, start distributed process CACHE_GROUP_KEY_CHANGE_FINISH, each node
- saves logical WAL record (ENCRYPTION_STATUS_RECORD) with current groups and key identifiers to start re-encryption after logical recovery
- save the new key in the metastore (as inactive key)
- adds new key and sets it for writing
- adds the mapping "WAL segment -> *old* key identifier" (to safely cleanup previous key in the future)
- saves logical WAL record (ENCRYPTION_STATUS_RECORD) with current page count in partitions.
- stores current page count as total pages for background re-encryption on partitionssave current keys and WALl mappings into the metastore.
- starts background re-encryption of an existing data.
...
Background re-encryption of existing data will be completed sometime in the future, the new "pagesLeftForReencryptionreencryptionFinished" cache group metric can be used to track re-encryption progress ('0' means the process has ended).
Background re-encryption
The process applies for all existing partitions including index.
Every time the cache group key changes, we store the current page count of the partition in the meta page (this value is used as the total page count to re-encrypt).
Scan all pages from specified range (metaPageId + [offset -> total])
- acquire pageif the checkpoint is finished (after key change) and page is dirty - skip this page
.- if the checkpoint is not finished or page is not dirty
- lock page
- unlock page (dirty=true)
- release page
Re-encryption progress is stored into metapage (int offset, int total), it updates during the checkpoint.
The process aborts only when a partition is destroyed.
At node startup, during partition initialization, if the total number of pages for re-encryption is greater than zero - this cache group is scheduled for re-encryption.
Cleanup old key
Old cache group encryption key will be removed when
- re-encryption completed for cache group (and after that at least one checkpoint was successfully completed)
- last WAL segment in which the key was used is removed
Changes in memory page format
PageMetaIO and PagePartitionMetaIO
Reencryption status requires an additional 8 bytes on the meta page of each partition.
Index partition uses PageMetaIO to read/write meta information (page type T_META).
Each other partition uses PagePartitionMetaIO to read/write meta information (page type T_PART_META).
Partition meta starts just after the end of the page meta.
draw.io Diagram |
---|
border | true |
---|
| |
---|
diagramName | pagemeta_old |
---|
simpleViewer | false |
---|
width | |
---|
links | auto |
---|
tbstyle | top |
---|
lbox | true |
---|
diagramWidth | 501 |
---|
revision | 1 |
---|
|
To support binary compatibility and keep code clean we creating a new successor of PageMetaIO - PageMetaIOV2 with the same type T_META.
We converting all existing T_META pages into a new version.
We storing additional 8 bytes at the end of each T_META and T_PART_META memory pages.
draw.io Diagram |
---|
border | true |
---|
| |
---|
diagramName | PagePartMetaModV2 |
---|
simpleViewer | false |
---|
width | |
---|
links | auto |
---|
tbstyle | top |
---|
lbox | true |
---|
diagramWidth | 441 |
---|
revision | 5 |
---|
|
draw.io Diagram |
---|
border | true |
---|
| |
---|
diagramName | PageIndexMetaModV2 |
---|
simpleViewer | false |
---|
width | |
---|
links | auto |
---|
tbstyle | top |
---|
lbox | true |
---|
diagramWidth | 611 |
---|
revision | 5 |
---|
|
WAL delta records have also been modified to store re-encryption status.
Encrypted (persisted) page
Each encrypted page has reserved free space to store CRC of encrypted data.
The size of this free space depends on the size of the encryption block, but cannot be less than 8 bytes (Ignite default encryption implementation (KeystoreEncryptionSpi) uses AES with 16 bytes block size).
Added 1 byte for encryption key ID on each encrypted page (after CRC).
draw.io Diagram |
---|
border | true |
---|
| |
---|
diagramName | encrypted_page |
---|
simpleViewer | false |
---|
width | 400 |
---|
links | auto |
---|
tbstyle | top |
---|
lbox | true |
---|
diagramWidth | 317 |
---|
revision | 2 |
---|
|
(WAL records ENCRYPTED_RECORD and ENCRYPTED_DATA_RECORD have been changed accordingly)Fault tolerance
Distributed key rotation
The node Node join is rejected during the encryption key rotation, but this limitation may be revised in the future.
When a node joins the cluster (before/after key rotation), it receives the current encryption keys for the cache groups used for writing (it "rotates" encryption key automatically).
If the encryption key is a new key, then the node sets it for writing and starts the background re-encryption process (it starts re-encryption automaticallyin other words, the node automatically "rotates" the encryption key. when joining a cluster, if necessary).
Therefore, a node may leave the cluster during a key change, or a node may be absent and rejoin later (it does not matter if the baseline changes or not), it will receive a new key and schedule re-encryption, if necessary.
Background re-encryption
...
- If checkpoint failed it should restore physical records from WAL, as usual.
- If checkpoint was not invoked re-encryption is started from the beginning using saved logical WAL record (that was recorded during key rotation).
Risks and assumptions
- Background re-encryption may affect performance. Performance impact can be managed using the following properties:configuration options:
- reencryptionBatchSize
- IGNITE_REENCRYPTION_THREAD_POOL_SIZE - number of threads used for reencryption.
- IGNITE_REENCRYPTION_BATCH_SIZE - number of pages that is are scanned during reencryption re-encryption under checkpoint lock.
IGNITE_REENCRYPTION_THROTTLE - delay in milliseconds between batches during a partition scanning - reencryptionRateLimit - page scanning speed limit in megabytes per second.
reencryptionThreadCnt - number of threads used for re-encryption(?).
- The WAL history can be not enough to store all entries between checkpoints (this should be carefully tuned by properly setting the size of the WAL history and tuning the re-encryption performance).
- The WAL history (for delta rebalancing) may be lost for all cache groups due to background re-encryption.
Process management
...
Public API changes
IgniteEncryption
...
- ReencryptionPagesLeft - (long) Total pages left for reencryption.
- ReencryptionFinished - (boolean) Indicates whether reencryption re-encryption is finished or not (it will set to true only when a checkpoint is finished).
Process management
The following commands should be added to the control.sh utility:
Rotate encryption key.
Code Block |
---|
language | text |
---|
title | command syntax |
---|
|
control.(sh|bat) --encryption change_cache_key cacheGroupName --yes |
Code Block |
---|
language | text |
---|
title | command output |
---|
|
The encryption key has been changed for cache group "default". |
View encryption key identifiers.
Code Block |
---|
|
control.(sh|bat) --encryption cache_key_ids cacheGroupName |
Code Block |
---|
language | text |
---|
title | command output |
---|
|
Encryption key identifiers for cache: default
Node 6085d500-2736-4c1f-b47c-444cf0a00000:
1 (active)
0
Node d98654c0-6dfb-4996-993e-387156300001:
1 (active)
0 |
View cache group re-encryption status.
Code Block |
---|
language | text |
---|
title | command syntax |
---|
|
control.(sh|bat) --encryption reencryption_status cacheGroupName |
Code Block |
---|
language | text |
---|
title | command output |
---|
|
Node 4ed26231-f92d-4b1c-86ba-7a117c200001:
1552 KB of data left for re-encryption
Node 89a456e5-59c5-4f13-a75b-39ab25000000:
1552 KB of data left for re-encryption |
Suspend cache group re-encryption.
Code Block |
---|
language | text |
---|
title | command syntax |
---|
|
control.(sh|bat) --encryption suspend_reencryption cacheGroupName |
Code Block |
---|
language | text |
---|
title | command output |
---|
|
Node ad1328e7-11e0-4ecb-8ef2-066519e00001:
re-encryption of the cache group "default" has been suspended.
Node 2a9e291f-e2d1-46e3-9954-18deb0e00000:
re-encryption of the cache group "default" has been suspended. |
Resume cache group re-encryption.
Code Block |
---|
language | text |
---|
title | command syntax |
---|
|
control.(sh|bat) --encryption resume_reencryption cacheGroupName |
Code Block |
---|
language | text |
---|
title | command output |
---|
|
Node 2ed43509-caab-48dc-a27d-3be65d800000:
re-encryption of the cache group "default" has been resumed.
Node b52d6451-a948-48d5-b79a-411956700001:
re-encryption of the cache group "default" has been resumed. |
View/change re-encryption rate limit.
Code Block |
---|
language | text |
---|
title | command syntax |
---|
|
control.(sh|bat) --encryption reencryption_rate [limit]
Parameters:
limit - decimal value to change rate limit (MB/s) |
Code Block |
---|
language | text |
---|
title | command output |
---|
|
Node 15cb8485-0c09-4361-b267-107d38400000:
re-encryption rate has been limited to 0.01 MB/s.
Node 909ed414-22e6-477b-b2ca-d1934cd00001:
re-encryption rate has been limited to 0.01 MB/s. |
Reference Links
- PCI DSS Requirements and Security Assessment Procedures
https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-2-1.pdf - How Often Do I Need to Rotate Encryption Keys on My SQL Server?
https://info.townsendsecurity.com/bid/49019/How-Often-Do-I-Need-to-Rotate-Encryption-Keys-on-My-SQL-Server - PCI DSS and key rotations simplified
https://www.crypteron.com/blog/pci-dss-key-rotations-simplified/ - Transparent Data Encryption in MS SQL Server
https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/transparent-data-encryption?view=sql-server-ver15 - Oracle Transparent Data Encryption FAQ
https://www.oracle.com/database/technologies/faq-tde.html - InnoDB Data-at-Rest Encryption
https://dev.mysql.com/doc/refman/8.0/en/innodb-data-encryption.html - Transparent data encryption feature proposed in pgsql-hackers.
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#Key_Rotation
...
Jira |
---|
server | ASF JIRA |
---|
columns | key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution |
---|
maximumIssues | 20 |
---|
jqlQuery | project = Ignite AND labels in (tde-phase-3) ORDER BY status |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b | key | IGNITE-12843 |
---|
|