Table of Contents |
---|
Cache encryption key rotation required in case of it compromising or at the end of crypto period (key validity period). in addition, such feature is required to provide support for encrypt and descrypt existing caches in the future.
Local partition re-encryption strategy is similar to partition snapshotting - create partition snapshot encrypted with the new key and then replace the original partition file with the new one.
Payment card industry data security standard (PCI DSS) requires that key-management procedures include a defined cryptoperiod for each key type in use and define a process for key changes at the end of the defined cryptoperiod(s). An expired key should not be used to encrypt new data, but it can be used for archived data, such keys should be strongly protected (section 3.5 - 3.6) [1].
The maximum recommended key lifetime is 2 years [2], and on average it is supposed to be changed every few months [3].
MS SQL Server provide rotation of database encryption key with background re-encryption of existing data [4]. Oracle and MySQL, out of the box, do not provide an automatic procedure for rotating tablespace keys, master key rotation is supported [5][6], Currently, TDE is being developed for PostgreSQL, but support for tablespace key rotation is not planned [7].
The overall Cluster-wide process consists of the following steps:
After completion of the key change preparation process, a new distributed process is initiated to complete the key change.
The discovery event from the distributed process pushes a new exchange task to the exchange worker to start PME (PME is required to prevent reordering of WAL records when key will be changed and to simplify initial design, this could and will be changed in the future)
While updates are blocked each node:
After changing the encryption key, new WAL records will be encrypted with the new key. However, it must be possible to read older WAL records (at least to support historical rebalance).
For each cache, instead of a key, it is necessary to keep a history of keys in the form WALPointer -> key
(stored the maximum pointer for which the associated key is applicable).
When removing a WAL segment to which WALPointer(s) refers - key(s) should be also removed.
When the WAL is cleared, respectively, the key history must also be cleared (except the last one).
By canceling the re-encryption procedure is meant clearing all temporary data.
If a node crashes during the replacement of the partitions, the original backup copies of the partitions are restored when the node starts.
If major topology changes during key rotation - cancelling whole procedure.
If cache is stopping during re-encryption - cancelling whole procedure, other minor topology changes should not affect re-encryption procedure.
(TBD) When baseline node with data joins the cluster and the cache group has a different key:
1. If historical rebalancing is not applicable encryption key will be changed when node joins and the partitions are cleared.
2. If historical rebalancing is applicable - existing data should be re-encrypted with the new key before(?) node joins the cluster.
TBD
TBD
Re-encryption process state.
...
To support multiple keys for reading encrypted data it is required to store a key identifier on each encrypted page and on each encrypted WAL record (see more details). The key identifier is a sequential counter and should be the same on all nodes.
After the FINISH phase is complete, a new encryption key for writing is set on all nodes, i.e. the key change process is formally completed.
Background re-encryption of existing data will be completed sometime in the future, the new "reencryptionFinished" cache group metric can be used to track re-encryption progress.
The process applies for all existing partitions including index.
Every time the cache group key changes, we store the current page count of the partition in the meta page (this value is used as the total page count to re-encrypt).
Scan all pages from specified range (metaPageId + [offset -> total])
Re-encryption progress is stored into metapage (int offset, int total), it updates during the checkpoint.
The process aborts only when a partition is destroyed.
At node startup, during partition initialization, if the total number of pages for re-encryption is greater than zero - this cache group is scheduled for re-encryption.
Old cache group encryption key will be removed when
Reencryption status requires an additional 8 bytes on the meta page of each partition.
Index partition uses PageMetaIO to read/write meta information (page type T_META).
Each other partition uses PagePartitionMetaIO to read/write meta information (page type T_PART_META).
Partition meta starts just after the end of the page meta.
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
To support binary compatibility and keep code clean we creating a new successor of PageMetaIO - PageMetaIOV2 with the same type T_META.
We converting all existing T_META pages into a new version.
We storing additional 8 bytes at the end of each T_META and T_PART_META memory pages.
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
WAL delta records have also been modified to store re-encryption status.
Each encrypted page has reserved free space to store CRC of encrypted data.
The size of this free space depends on the size of the encryption block, but cannot be less than 8 bytes (Ignite default encryption implementation (KeystoreEncryptionSpi) uses AES with 16 bytes block size).
Added 1 byte for encryption key ID on each encrypted page (after CRC).
draw.io Diagram | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Node join is rejected during the key rotation, but this limitation may be revised in the future.
When a node joins the cluster (before/after key rotation), it receives the current encryption keys for the cache groups used for writing. If the encryption key is a new key, then the node sets it for writing and starts the background re-encryption process (in other words, the node automatically "rotates" the encryption key. when joining a cluster, if necessary).
Therefore, a node may leave the cluster during a key change, or a node may be absent and rejoin later (it does not matter if the baseline changes or not), it will receive a new key and schedule re-encryption, if necessary.
If the node stops/fails during re-encryption, after restarting it continue re-encryption from the stored offset:
New method will be introduced
public IgniteFuture<Void> changeCacheGroupKey(Collection<String> cacheOrGroupNames)
Re-encryption process state in CacheGroupMetrics
The following commands should be added to the control.sh utility:
Code Block | ||||
---|---|---|---|---|
| ||||
control.(sh|bat) --encryption change_cache_key cacheGroupName --yes |
Code Block | ||||
---|---|---|---|---|
| ||||
The encryption key has been changed for cache group "default". |
Code Block | ||
---|---|---|
| ||
control.(sh|bat) --encryption cache_key_ids cacheGroupName |
Code Block | ||||
---|---|---|---|---|
| ||||
Encryption key identifiers for cache: default
Node 6085d500-2736-4c1f-b47c-444cf0a00000:
1 (active)
0
Node d98654c0-6dfb-4996-993e-387156300001:
1 (active)
0 |
Code Block | ||||
---|---|---|---|---|
| ||||
control.(sh|bat) --encryption reencryption_status cacheGroupName |
Code Block | ||||
---|---|---|---|---|
| ||||
Node 4ed26231-f92d-4b1c-86ba-7a117c200001:
1552 KB of data left for re-encryption
Node 89a456e5-59c5-4f13-a75b-39ab25000000:
1552 KB of data left for re-encryption |
Code Block | ||||
---|---|---|---|---|
| ||||
control.(sh|bat) --encryption suspend_reencryption cacheGroupName |
Code Block | ||||
---|---|---|---|---|
| ||||
Node ad1328e7-11e0-4ecb-8ef2-066519e00001:
re-encryption of the cache group "default" has been suspended.
Node 2a9e291f-e2d1-46e3-9954-18deb0e00000:
re-encryption of the cache group "default" has been suspended. |
Code Block | ||||
---|---|---|---|---|
| ||||
control.(sh|bat) --encryption resume_reencryption cacheGroupName |
Code Block | ||||
---|---|---|---|---|
| ||||
Node 2ed43509-caab-48dc-a27d-3be65d800000:
re-encryption of the cache group "default" has been resumed.
Node b52d6451-a948-48d5-b79a-411956700001:
re-encryption of the cache group "default" has been resumed. |
Code Block | ||||
---|---|---|---|---|
| ||||
control.(sh|bat) --encryption reencryption_rate [limit]
Parameters:
limit - decimal value to change rate limit (MB/s) |
Code Block | ||||
---|---|---|---|---|
| ||||
Node 15cb8485-0c09-4361-b267-107d38400000:
re-encryption rate has been limited to 0.01 MB/s.
Node 909ed414-22e6-477b-b2ca-d1934cd00001:
re-encryption rate has been limited to 0.01 MB/s. |
...
Jira server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution maximumIssues 20 jqlQuery project = Ignite AND labels in (tde-phase-3) ORDER BY status serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key IGNITE-12843