Motivation

Master key rotation required in case of it compromising or at the end of crypto period(key validity period). 

Design assumes that administrator will provide ability to get new master key by EncryptionSPI from underlying storage.

Definitions

• MK – Master Key. Encrypts cache keys. Master key is stored in some key storage. Master keys identified by String masterKeyId .

Prerequisites

New master key should be available to EncryptionSPI for each server node.

Process management

Users can control the master key rotation process key via  some kind of user interface(CLI, JMX, Java API). 

• JAVA API:
  ignite.encryption().changeMasterKey(String masterKeyId)  - starts master key rotation process.
  String ignite.encryption().getMasterKeyId()  - gets current master key id.
  
  
• JMX:
  changeMasterKey(String masterKeyId)  - starts master key rotation process.
  String getMasterKeyId()   - gets current master key id.
  
  
• CLI:
  # Starts master key rotation.
  control.sh --encryption change_master_key newMasterKeyId 
  
  # Displays cluster's current master key id.
  control.sh --encryption get_master_key 
  
  # Starts ignite with MK recovery process. See details.
  ignite.sh --change-master-key newMasterKeyId 

Process description 

1. Check that the cluster is active (WAL should be available for a write to correctly log changes and survive cluster restarts). Otherwise, throw an error.
2. A node creates the ChangeMasterKeyMessage  message and sent it by discovery as a custom event. The goal is to verify that all nodes have the same master key. 
   
   1. Initiating message should contain: 
      1. New master key id. 
      2. New master key hash.
   2. When server node processed message following actions are executed: 
      1. It obtain hash of new master key.
      2. Compares it with the one in message
      3. If it differs then error added to the message.
      4. Store locally master key id and hash.
3. If on step1 there are some errors we log it and cancel process. Otherwise got to step3.
4. The ChangeMasterKeyFinishMessage  action message is sent by discovery as a custom event.
   1. Action message sould contain:
      
      1. New master key id.
      2. New master key hash.
   2. When server node processed message following actions are executed: 
      1. Checks that master key id and hash is the same as it was taken from the first message. Otherwice, we log it and cancel process.
      2. Blocks creation of encrypted cache key.
      3. Reencrypt all cache group keys with new master key in a temporary datastructure. No changes in MetaStore.
      4. Create WAL logical record (ChangeMasterKeyRecord ) that consist of:
         1. New master key id
         2. Reenctyped cache group keys.
      5. Write cache group keys to MetaStore .
      6. Unblock creation of encrypted cache key. 

Process completion

Process completes when all nodes in cluster will process action message.

Corner cases

Node was down during key rotation. ChangeMasterKeyRecord was not found.

If some node was unavailable during master key rotation process it will unable to join to the cluster because it has old master key.

To update this node user should run ignite with command to change master key before join:

ignite.sh --change-master-key newMasterKeyId 

The node will re-encrypt cache keys with new MK and try to join to cluster.

Node was down during key rotation. ChangeMasterKeyRecord found.

A node should not try to join to the cluster before the process of ChangeMasterKeyRecord. Regardless of whether the key rotation was finished successfully or not, the recovery will be from the record.

1. If during node recovery with logical records we found ChangeMasterKeyRecord  it passed to EncryptionManager .
2. When MetaStore becomes available for write, EncryptionManager  writes new cache group keys to it.

Node join during key rotation process

Reject node join. It may lead to inconsistent master keys in cluster. (Or if it possible to delay until key rotation process ends)

Cache start during key rotation process

Cache keys must not be created during the master key rotation process. So, a node will throw an exception if a user will start cache during the key rotation process.

Public java API changes

EncryptionSpi

The concept of the masterKeyId will be added to the cache keys encryption process in EncryptionSpi :

New methods will be introduced:

• setMasterKeyId(String masterKeyId)  // Sets "current" master key id
• String getMasterKeyId()  // Gets "current" master key id

Follow methods will work with master key that setted by previous method:

• byte[] masterKeyDigest() 
• byte[] encryptKey(Serializable key) 
• Serializable decryptKey(byte[] key) 

This is necessary so that ignite can decrypt cache keys with the old master key and encrypt with the new one.

Code changes

Meta Storage

Meta storage will store master key id. Key id from meta storage has a higher priority to key id from EncryptionSpi .

Node attribute

Currently joining node send hash MK for validation in attributes. Attributes can't be modified at runtime. So joining node will send hash MK in JoiningNodeDiscoveryData .