...

Process description

1. A node creates the ChangeMasterKeyMessagethe MasterKeyChangeMessage  message and sent it by discovery as a custom event. The goal is to verify that all nodes have the same master key. 
   
   1. Initiating message should contain: 
      1. New master key id. 
      2. New master key hash.
   2. When server node processed message following actions are executed:
      
      1. It obtain hash of new master key.
      2. Compares it with the one in message
      3. If it differs then error added to the message.
      4. Store locally master key id and hash.
2. If on step1 there are some errors we log it and cancel process. Otherwise got to step3.
3. The ChangeMasterKeyFinishMessageThe MasterKeyChangeMessage  ack action message is sent by discovery as a custom event.
   1. Action message sould contain:
      
      1. New master key id.
      2. New master key hash.
   2. When server node processed message following actions are executed: 
      1. It checks that there are no errors in the message and the cluster is active (WAL should be available for a write to correctly log changes and survive cluster restarts). Otherwise, error added to the messagecancel process with error.
      2. Checks that master key id and hash is the same as it was taken from the first message. Otherwice, we log it and cancel process.
      3. Blocks creation of encrypted cache key.
      4. Reencrypt all cache group keys with new master key in a temporary datastructure. No changes in MetaStore.
      5. Create WAL logical record (ChangeMasterKeyRecord ) that consist of:
         1. New master key id
         2. Reenctyped cache group keys.
      6. Write cache group keys to MetaStore .
      7. Unblock creation of encrypted cache key. 

...

Process completes when all nodes in cluster will process action message.

Corner cases

Node was down during key rotation.

...

MasterKeyChangeRecord was not found.

If some node was unavailable during master key rotation process it will unable to join to the cluster because it has old master key.

To update this node user should run ignite with system property (IGNITE_MASTER_KEY_ID_TO_CHANGE_ON_STARTUP=newMasterKeyId) or with command to change master key before join:

...

The node will re-encrypt cache keys with new MK and try to join to cluster.

Node was down during key rotation.

...

MasterKeyChangeRecord found.

A node should not try to join to the cluster before the process of ChangeMasterKeyRecord. Regardless of whether the key rotation was finished successfully or not, the recovery will be from the record.

...