Feature Design for: Key Management Service (KMS) with HSM Integration

Project Introduction

This document outlines the design for a comprehensive Key Management Service (KMS) in Apache CloudStack that provides envelope encryption for volume encryption using Hardware Security Modules (HSMs) or a database-backed fallback provider (also used for testing).

The primary goal is to allow users to manage encryption keys (KEKs) for their encrypted volumes through a secure, plugin-based architecture. KEKs can be stored in PKCS#11-compliant HSMs for hardware-grade security or in the CloudStack database for deployments without HSM hardware. The design supports key rotation, cross-HSM migration, and gradual re-encryption of wrapped keys.

GitHub PR: https://github.com/apache/cloudstack/pull/12711

CloudStack Version: 4.23.0.0

Glossary

This hierarchy:

KMS Provider (Plugin) → HSM Profile (Configuration) → KMS Key (KEK abstraction) → KEK Version → Wrapped Key (DEK)

KEK Size vs DEK Size (Important Distinction)

WhatSourceUsed For
KEK sizekeybits parameter in createKMSKey / rotateKMSKey (stored in kms_keys.key_bits)Size of the Key Encryption Key stored in the HSM or database. Used when creating or rotating a KEK.
DEK sizeGlobal config kms.dek.size.bitsSize of the Data Encryption Key generated per volume. Used in generateVolumeKeyWithKek().

Functional Description

Key Functional Components

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    CloudStack API Layer                     │
│  createKMSKey, listKMSKeys, rotateKMSKey, addHSMProfile     │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│              KMSManagerImpl (Implementation)                │
│  - HSM profile management        - Permission checks        │
│  - KEK version management        - Retry with back-off      │
│  - Background rewrap jobs        - Per-operation timeout    │
│  - Cross-HSM migration           - Transaction boundaries   │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│           KMSProvider (Plugin Interface)                    │
│  createKek(), wrapKey(), unwrapKey(), rewrapKey()           │
└─────────────────────────────────────────────────────────────┘
         ┌────────────────────┴────────────────────┐
         ↓                                         ↓
┌──────────────────────┐              ┌──────────────────────┐
│ DatabaseKMSProvider  │              │ PKCS11HSMProvider    │
│ - Database storage   │              │ - PKCS#11 interface  │
│ - AES-256-GCM        │              │ - Session pooling    │
│ - Encrypted KEKs     │              │ - AES/CBC/PKCS5      │
└──────────────────────┘              └──────────────────────┘

Key Hierarchy

HSM (PKCS#11) or Database
    └── KEK (Key Encryption Key) - Stored in HSM/DB
        └── KEK Version (supports rotation)
            └── DEK (Data Encryption Key) - Wrapped by KEK
                └── Volume Data - Encrypted by DEK on hypervisor

HSM Profile Scoping

  1. User-Owned Profile: account_id set → visible only to that account
  2. Zone Admin Profile: zone_id set, account_id NULL → visible to all accounts in that zone
  3. Global Admin Profile: zone_id NULL, account_id NULL, system = TRUE → visible to all accounts in all zones

Note: createKMSKey requires an explicit hsmprofileid parameter. There is no automatic profile resolution or fallback hierarchy during key creation.

Workflows

Volume Encryption Flow

  1. User creates volume or deploys VM with encryption enabled and KMS key specified
  2. VolumeOrchestrator calls kmsManager.generateVolumeKeyWithKek()
  3. KMSManagerImpl gets the active KEK version, generates a random DEK (size from kms.dek.size.bits), wraps it using the active KEK, and persists the wrapped key
  4. Volume is updated with kms_key_id and kms_wrapped_key_id
  5. When the hypervisor needs the DEK, kmsManager.unwrapVolumeKey() is called

Key Rotation Flow

  1. Admin calls rotateKMSKey with optional hsmprofileid for cross-HSM migration
  2. Provider creates a new KEK in the HSM
  3. Database updates (new KEK version, old version marked Previous, optional profile update) are executed atomically via Transaction.execute()
  4. Background rewrap job (ScheduledExecutorService) gradually rewraps wrapped keys in batches
  5. When all wrapped keys are migrated, old KEK version is marked Archived

API Changes

KMS Key APIs

createKMSKey

Creates a new KMS key (Key Encryption Key) for envelope encryption.

Parameters:

ParameterRequiredTypeDescription
nameYesStringName of the KMS key
descriptionNoStringDescription of the KMS key
purposeYesStringPurpose of the key (volume, tls)
zoneidYesUUIDZone ID where the key will be valid
hsmprofileidYesUUIDHSM profile ID to create the KEK in
keybitsNoIntegerKEK size in bits (128, 192, 256). Default: 256
accountNoStringAccount name (admin use)
domainidNoUUIDDomain ID (admin use)

CloudMonkey Example:

cmk createKMSKey \
  name="volume-encryption-key" \
  description="Production volume encryption" \
  purpose="volume" \
  keybits=256 \
  zoneid=<zone-uuid> \
  hsmprofileid=<hsm-profile-uuid>

listKMSKeys

Lists KMS keys available to the caller.

Parameters:

ParameterRequiredTypeDescription
idNoUUIDList KMS key by UUID
purposeNoStringFilter by purpose
zoneidNoUUIDFilter by zone
stateNoStringFilter by state (Enabled, Disabled)

CloudMonkey Example:

cmk listKMSKeys purpose=volume state=Enabled

updateKMSKey

Updates KMS key name, description, or state.

Parameters:

ParameterRequiredTypeDescription
idYesUUIDKMS key UUID
nameNoStringNew name
descriptionNoStringNew description
enabledNoBooleanEnable/disable the key

CloudMonkey Example:

cmk updateKMSKey id=<kms-key-uuid> enabled=false

deleteKMSKey

Deletes a KMS key (only if not referenced by volumes or wrapped keys).

Parameters:

ParameterRequiredTypeDescription
idYesUUIDKMS key UUID

CloudMonkey Example:

cmk deleteKMSKey id=<kms-key-uuid>

rotateKMSKey

Rotates KEK by creating a new version and scheduling gradual re-encryption of wrapped keys.

Parameters:

ParameterRequiredTypeDescription
idYesUUIDKMS key UUID to rotate
keybitsNoIntegerKey size for new KEK (default: same as current)
hsmprofileidNoUUIDTarget HSM profile for cross-HSM migration

CloudMonkey Examples:

# Same-HSM rotation
cmk rotateKMSKey id=<kms-key-uuid>

# Cross-HSM migration
cmk rotateKMSKey id=<kms-key-uuid> hsmprofileid=<target-hsm-profile-uuid>

migrateVolumesToKMS

Migrates passphrase-based volumes to KMS encryption.

Parameters:

ParameterRequiredTypeDescription
zoneidYesUUIDZone ID
idYesUUIDKMS key ID to migrate volumes to
accountNoStringMigrate volumes for specific account
domainidNoUUIDDomain ID

CloudMonkey Example:

cmk migrateVolumesToKMS zoneid=<zone-uuid> id=<kms-key-uuid>

HSM Profile APIs

addHSMProfile

Adds a new HSM profile for connecting to an HSM device.

Parameters:

ParameterRequiredTypeDescription
nameYesStringHSM profile name
protocolNoStringProtocol (PKCS11, KMIP, etc.). Default: pkcs11
zoneidNoUUIDZone ID (null = global scope)
domainidNoUUIDDomain ID
accountNoStringAccount name
systemNoBooleanSystem profile (globally available, root admin only)
vendornameNoStringHSM vendor name
detailsNoMapHSM configuration details

PKCS#11 details keys: library (path to PKCS#11 library), slot (slot number), pin (HSM PIN, encrypted at rest), token_label (token label), minSessions, maxSessions

CloudMonkey Example:

cmk addHSMProfile \
  name="production-hsm" \
  protocol="PKCS11" \
  zoneid=<zone-uuid> \
  system=true \
  details[0].library="/usr/lib/libsofthsm2.so" \
  details[0].slot="0" \
  details[0].pin="<pin>" \
  details[0].token_label="cloudstack"

listHSMProfiles

Lists HSM profiles visible to the caller.

Parameters:

ParameterRequiredTypeDescription
idNoUUIDHSM profile ID
zoneidNoUUIDZone ID
protocolNoStringProtocol filter
enabledNoBooleanEnabled filter

CloudMonkey Example:

cmk listHSMProfiles zoneid=<zone-uuid> protocol=PKCS11

updateHSMProfile

Updates an HSM profile name or enabled state.

Parameters:

ParameterRequiredTypeDescription
idYesUUIDHSM profile UUID
nameNoStringNew name
enabledNoBooleanEnable/disable

Note: Updating configuration details is not currently supported. To change PKCS#11 parameters (e.g., PIN), delete and re-create the HSM profile.

CloudMonkey Example:

cmk updateHSMProfile id=<profile-uuid> enabled=false

deleteHSMProfile

Deletes an HSM profile (only if not in use by any KEK versions).

Parameters:

ParameterRequiredTypeDescription
idYesUUIDHSM profile UUID

CloudMonkey Example:

cmk deleteHSMProfile id=<profile-uuid>

Global Settings

Setting KeyScopeTypeDefaultDescription
kms.dek.size.bitsGlobalInteger256Size of DEKs in bits for new volumes (128, 192, 256)
kms.retry.countGlobalInteger3Number of retry attempts for transient KMS failures
kms.retry.delay.msGlobalInteger1000Delay in milliseconds between retry attempts
kms.operation.timeout.secGlobalInteger30Per-attempt timeout for KMS operations
kms.rewrap.batch.sizeGlobalInteger50Wrapped keys rewrapped per batch in background job
kms.rewrap.interval.msGlobalLong300000Interval between background rewrap executions (5 min)

Database Changes

New Tables

cloud.kms_hsm_profiles

Stores HSM profile configurations. Scoped by account_id, domain_id, and zone_id.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_hsm_profiles` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `protocol` VARCHAR(32) NOT NULL COMMENT 'PKCS11, KMIP, AWS_KMS, etc.',
    `account_id` BIGINT UNSIGNED COMMENT 'null = admin-provided',
    `domain_id` BIGINT UNSIGNED,
    `zone_id` BIGINT UNSIGNED COMMENT 'null = global scope',
    `vendor_name` VARCHAR(64),
    `enabled` BOOLEAN NOT NULL DEFAULT TRUE,
    `system` BOOLEAN NOT NULL DEFAULT FALSE,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_account_name` (`account_id`, `name`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_hsm_profile_details

Key-value configuration details for HSM profiles. Sensitive values (PIN, password) are encrypted via DBEncryptionUtil.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_hsm_profile_details` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `profile_id` BIGINT UNSIGNED NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `value` TEXT NOT NULL COMMENT 'encrypted if sensitive',
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_profile_name` (`profile_id`, `name`),
    CONSTRAINT `fk_kms_hsm_profile_details__profile_id`
        FOREIGN KEY (`profile_id`) REFERENCES `kms_hsm_profiles`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_keys

KMS key (KEK) metadata. Account-scoped, zone-bound.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_keys` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `description` VARCHAR(1024),
    `kek_label` VARCHAR(255) NOT NULL,
    `purpose` VARCHAR(32) NOT NULL COMMENT 'VOLUME_ENCRYPTION, TLS_CERT, CONFIG_SECRET',
    `account_id` BIGINT UNSIGNED NOT NULL,
    `domain_id` BIGINT UNSIGNED NOT NULL,
    `zone_id` BIGINT UNSIGNED NOT NULL,
    `algorithm` VARCHAR(64) NOT NULL DEFAULT 'AES/GCM/NoPadding',
    `key_bits` INT NOT NULL DEFAULT 256,
    `enabled` TINYINT(1) NOT NULL DEFAULT 1,
    `hsm_profile_id` BIGINT UNSIGNED NOT NULL,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_kek_versions

KEK versions for gradual key rotation. Only one Active version per KMS key at a time.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_kek_versions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `kms_key_id` BIGINT UNSIGNED NOT NULL,
    `version_number` INT NOT NULL,
    `kek_label` VARCHAR(255) NOT NULL,
    `status` VARCHAR(32) NOT NULL DEFAULT 'Active' COMMENT 'Active, Previous, Archived',
    `hsm_profile_id` BIGINT UNSIGNED,
    `hsm_key_label` VARCHAR(255),
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_kms_key_version` (`kms_key_id`, `version_number`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_wrapped_key

Wrapped DEKs. Each volume references one wrapped key.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_wrapped_key` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `kms_key_id` BIGINT UNSIGNED,
    `kek_version_id` BIGINT UNSIGNED,
    `zone_id` BIGINT UNSIGNED NOT NULL,
    `wrapped_blob` VARBINARY(4096) NOT NULL,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_database_kek_objects

PKCS#11-compatible object storage for the database KMS provider. Key material is encrypted via DBEncryptionUtil (Base64-encoded KEK → DBEncryptionUtil.encrypt() → stored bytes).

CREATE TABLE IF NOT EXISTS `cloud`.`kms_database_kek_objects` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `object_class` VARCHAR(32) NOT NULL DEFAULT 'CKO_SECRET_KEY',
    `label` VARCHAR(255) NOT NULL,
    `object_id` VARBINARY(64),
    `key_type` VARCHAR(32) NOT NULL DEFAULT 'CKK_AES',
    `key_material` VARBINARY(512) NOT NULL COMMENT 'encrypted KEK material',
    `is_sensitive` BOOLEAN NOT NULL DEFAULT TRUE,
    `is_extractable` BOOLEAN NOT NULL DEFAULT FALSE,
    `purpose` VARCHAR(32) NOT NULL,
    `key_bits` INT NOT NULL,
    `algorithm` VARCHAR(64) NOT NULL DEFAULT 'AES/GCM/NoPadding',
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_label` (`label`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Modified Tables

cloud.volumes

Two new columns added via schema migration:

-- KMS key reference
ALTER TABLE `cloud`.`volumes` ADD COLUMN `kms_key_id` BIGINT UNSIGNED;
ALTER TABLE `cloud`.`volumes` ADD CONSTRAINT `fk_volumes__kms_key_id`
    FOREIGN KEY (`kms_key_id`) REFERENCES `kms_keys`(`id`);

-- Wrapped DEK reference
ALTER TABLE `cloud`.`volumes` ADD COLUMN `kms_wrapped_key_id` BIGINT UNSIGNED;
ALTER TABLE `cloud`.`volumes` ADD CONSTRAINT `fk_volumes__kms_wrapped_key_id`
    FOREIGN KEY (`kms_wrapped_key_id`) REFERENCES `kms_wrapped_key`(`id`);

User Interface

A new KMS top-level menu section is added with two sub-sections:

KMS Keys

HSM Profiles

Security Considerations

  1. Key Storage: KEKs never leave the HSM (for PKCS#11 provider) or are stored encrypted in database (for database provider). DEKs are always stored wrapped.

  2. Sensitive Data: HSM profile PINs/passwords encrypted via DBEncryptionUtil before storage. Encrypted values appear as ENC(...) in API responses.

  3. Access Control: Only root admins can manage HSM profiles (create, update, delete). All users can list HSM profiles visible to them. KMS keys are account-scoped with domain hierarchy permissions.

  4. Key Rotation: Background rewrap job gradually re-encrypts wrapped keys to avoid service disruption. Old KEK versions retained until all wrapped keys migrated. Cross-HSM migration supported.

  5. PKCS#11 Cipher: AES-CBC with PKCS5Padding (FIPS-compliant, universal driver support). AES-GCM preferred but SunPKCS11 support is unreliable across drivers.

  6. Error Handling: Per-operation timeout via Future.get(), retry with back-off for transient failures, fast-fail for non-retryable errors, graceful fallback to passphrase encryption.

  7. Cluster Safety: GlobalLock("kms.rewrap.worker") ensures only one management server runs the rewrap job. Key rotation DB updates use Transaction.execute() for atomicity with compensating HSM cleanup on failure.

Future Enhancements