Key rotation required in case of it compromising or at the end of crypto period(key validity period). 

Design assumes that administrator will provide ability to get new master key by EncryptionSPI from underlying storage.

Goal: 

To implement ability to rotate master encryption key. 

New processes: 

1. Master key rotation.

2. Master key rotation recovery start.

New administrator commands: 

1. Master keys view: node -> master key hash 

2. Cache group keys view: node -> group name -> encryption key hash 

Master Key rotation: 

Process start: 

Administrator initiates key rotation via  some kind of user interface(CLI, Visor, Web Console, JMX, etc). 

Process description: 

1. Initiating message is sent by discovery. 
   1. Initiating message should contain: 
      1. New master key hash. 
      2. New master key id.
   2. When server node processed message following actions are executed: 
      1. It obtain hash of new master key.
      2. Compares it with the one in message
      3. If it differs then error added to the message.
2. If on step1 there are some errors we log it and cancel process. Otherwise got to step3.
3. Action message is sent by discovery.
   1. Action message sould contain:
      1. New master key hash. 
      2. New master key id.
   2. When server node processed message following actions are executed: 
      1. Blocks creation of encrypted cache key. 
      2. Encrypt cache group keys with new master key. 
      3. Unblock creation of encrypted cache key. 
      4. EncryptionSPI executes keys rotation (implementation specific).

Atomic keys rotation should be done in the following way:

1. Reencrypt all cache group keys with new master key in a temporary datastructure.
   No changes in MetaStore.
2. Create WAL logical record (MasterKeyChangeRecord) that consist of:
   1. New master key hash
   2. Reenctyped cache group keys.
3. Write cache group keys to MetaStore.

Node recovery:

1. If during node recovery with logical records we found MasterKeyChangeRecord it passed to EncryptionManager.
2. When MetaStore becomes available for write, EncryptionManager writes new cache group keys to it.

Process completion: 

Process completes when all nodes in cluster will process action message.

Master key rotation recovery start

Motivation:

If some node was unavailable during master key rotation process it will unable to join to the cluster because it has old master key has.

To update this node design introduce master key recovery start option.

Process start:

Administartor initiates process by providing startup option.

Process description:

Node should execute following steps before join to the cluster:

1. Obtain old master key by id
2. Obtain new master key by id
3. Reencrypt cache group keys with new master key and store it to metastore.
4. EncryptionSPI executes keys rotation (implementation specific).