Feature Design for: Key Management Service (KMS) with HSM Integration

Project Introduction

This document outlines the design for a comprehensive Key Management Service (KMS) in Apache CloudStack that provides envelope encryption for volume encryption using Hardware Security Modules (HSMs) or a database-backed fallback provider (also used for testing).

...

KEK Size vs DEK Size (Important Distinction)

• The KMS key's keybits controls only the KEK size (e.g., 256-bit AES key in the HSM).
• The global setting kms.dek.size.bits controls the DEK size for all new encrypted volumes.
• DEK size is independent of the KMS key; all volumes get DEKs of the configured global size.

...

• HSM Profile Management (Admin only)

  • Add HSM Profile: Configure connection to HSM devices (PKCS#11-compliant)
  • Profile Scoping: User-owned, zone-level, or global (systempublic) profiles
  • Profile Validation: Health check on configured HSM connections
  • Sensitive data encryption: PINs and passwords are encrypted via DBEncryptionUtil before storage

• KMS Key Management

  • Key Creation: Create KMS keys (KEKs) bound to an HSM profile, zone, and account
  • Key Update: Enable/disable keys, update name and description
  • Key Deletion: Soft-delete keys (only if not in use by volumes or wrapped keys)
  • Key Listing: List keys with filtering by purpose, zone, state, and account

• Key Rotation

  • Same-HSM Rotation: Create a new KEK version in the same HSM with a new label
  • Cross-HSM Migration: Create a new KEK version in a different HSM profile
  • Transaction Atomicity: Database updates for new KEK version and key profile are wrapped in a Transaction.execute() block; orphaned HSM keys are cleaned up on DB failure
  • Background Rewrap: Gradual re-encryption of wrapped keys in configurable batches

• Volume Encryption Integration

  • DEK Generation: Generate random DEKs (configurable size via kms.dek.size.bits) and wrap them with the active KEK version
  • DEK Unwrapping: Unwrap DEKs on demand for volume access (plaintext DEKs are zeroized after use)
  • Volume Migration: Migrate legacy passphrase-encrypted volumes to KMS encryption

• Plugin Architecture

  • DatabaseKMSProvider: Database-backed KEK storage with AES/GCM/NoPadding encryption via DBEncryptionUtil
  • PKCS11HSMProvider: PKCS#11 HSM integration with per-profile session pooling and AES/CBC/PKCS5Padding wrapping

• Concurrency & Cluster Safety

  • Bounded thread pool for KMS operations: ThreadPoolExecutor(core=2, max=100, keepAlive=60s, SynchronousQueue) with daemon threads
  • Cluster-aware rewrap: GlobalLock("kms.rewrap.worker") prevents duplicate rewrap work across management server nodes
  • ScheduledExecutorService (replaces java.util.Timer) for robust periodic rewrap scheduling

...

1. User-Owned Profile: account_id set → visible only to that account
2. Zone Admin Profile: zone_id set, account_id NULL → visible to all accounts in that zone
3. Global Admin Profile: zone_id NULL, account_id NULL, system is_public = TRUE → visible to all accounts in all zones

...

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Async: No

Parameters:

CloudMonkey Example:

cmk createKMSKey \
  name="volume-encryption-key" \
  description="Production volume encryption" \
  purpose="volume" \
  keybits=256 \
  zoneid=<zone-uuid> \
  hsmprofileid=<hsm-profile-uuid>

listKMSKeys

Lists KMS keys available to the caller.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User

Parameters:

listKMSKeys

Lists KMS keys available to the caller.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User

Parameters:

...

cmk listKMSKeys purpose=volume state=Enabled

updateKMSKey

Updates KMS key name, description, or state.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Async: Yes

Parameters:

CloudMonkey Example:

cmk updateKMSKey id=<kms-key-uuid> enabled=false

deleteKMSKey

deleteKMSKey

Deletes a KMS key (Deletes a KMS key (only if not referenced by volumes or wrapped keys).

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Async: Yes

Parameters:

CloudMonkey Example:

...

rotateKMSKey

Rotates KEK by creating a new version and scheduling gradual re-encryption of wrapped keys.

• Authorization: Admin only
• Async: Yes

Parameters:

CloudMonkey Examples:

# Same-HSM rotation
cmk rotateKMSKey id=<kms-key-uuid>

# Cross-HSM migration
cmk rotateKMSKey id=<kms-key-uuid> hsmprofileid=<target-hsm-profile-uuid>

migrateVolumesToKMS

Migrates passphrase-based volumes to KMS encryption.

• Authorization: Admin only
• Async: Yes

Parameters:

migrateVolumesToKMS

Migrates passphrase-based volumes to KMS encryption.

• Authorization: Admin only
• Async: Yes

Parameters:

CloudMonkey Example:

cmk migrateVolumesToKMS zoneid=<zone-uuid> id=<kms-key-uuid>

HSM Profile APIs

addHSMProfile

HSM Profile APIs

addHSMProfile

Adds a new HSM profile for Adds a new HSM profile for connecting to an HSM device.

• Authorization: Admin only
• Request has sensitive info: Yes (PIN, passwords)

Parameters:

PKCS#11 details keys: library (path to PKCS#11 library), slot (slot number), pin (HSM PIN, encrypted at rest), token_label (token label), minSessions, maxSessions

CloudMonkey Example:

...

PKCS#11 details keys: library (path to PKCS#11 library), slot (slot number), pin (HSM PIN, encrypted at rest), token_label (token label), minSessions, maxSessions

listHSMProfiles

Lists HSM profiles visible to the caller.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Response has sensitive info: Yes (encrypted values shown as ENC(...))

Parameters:

CloudMonkey Example:

...

updateHSMProfile

Updates an HSM profile name or enabled state.

• Authorization: Admin only

Parameters:

Note: Updating configuration details is not currently supported. To change PKCS#11 parameters (e.g., PIN), delete and re-create the HSM profile.

CloudMonkey Example:

...

deleteHSMProfile

Deletes an HSM profile (only if not in use by any KEK versions).

• Authorization: Admin only

Parameters:

CloudMonkey Example:

...

Global Settings

Database Changes

New Tables

...

Stores HSM profile configurations. Scoped by account_id, domain_id, and zone_id.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_hsm_profiles` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `protocol` VARCHAR(32) NOT NULL COMMENT 'PKCS11, KMIP, AWS_KMS, etc.',
    `account_id` BIGINT UNSIGNED COMMENT 'null = admin-provided',
    `domain_id` BIGINT UNSIGNED,
    `zone_id` BIGINT UNSIGNED COMMENT 'null = global scope',
    `vendor_name` VARCHAR(64),
    `enabled` BOOLEAN NOT NULL DEFAULT TRUE,
    

...

`is_public` BOOLEAN NOT NULL DEFAULT FALSE,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_account_name` (`account_id`, `name`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

...

cloud.kms_hsm_profile_details

Key-value configuration details for HSM profiles. Sensitive values (PIN, password) are encrypted via DBEncryptionUtil.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_hsm_profile_details` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `profile_id` BIGINT UNSIGNED NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    

...

`value` TEXT NOT NULL COMMENT 'encrypted if sensitive',
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_profile_name` (`profile_id`, `name`),
    CONSTRAINT `fk_kms_hsm_profile_details__profile_id`
        FOREIGN KEY (`profile_id`) REFERENCES `kms_hsm_profiles`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

...

cloud.kms_keys

KMS key (KEK) metadata. Account-scoped, zone-bound.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_keys` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `description` VARCHAR(1024),
    `kek_label` VARCHAR(255) NOT NULL,
    `purpose` VARCHAR(32) NOT NULL COMMENT 'VOLUME_ENCRYPTION, TLS_CERT, CONFIG_SECRET',
    `account_id` BIGINT UNSIGNED NOT NULL,
    `domain_id` BIGINT UNSIGNED NOT NULL,
    `zone_id` BIGINT UNSIGNED NOT NULL,
    `algorithm` VARCHAR(64) NOT NULL DEFAULT 'AES/GCM/NoPadding',
    `key_bits` INT NOT NULL DEFAULT 256,
    `enabled` TINYINT(1) NOT NULL DEFAULT 1,
    `hsm_profile_id` BIGINT UNSIGNED NOT NULL,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

...

cloud.kms_kek_versions

KEK versions for gradual key rotation. Only one Active version per KMS key at a time.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_kek_versions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `kms_key_id` BIGINT UNSIGNED NOT NULL,
    `version_number` INT NOT NULL,
    `kek_label` VARCHAR(255) NOT NULL,
    `status` VARCHAR(32) NOT NULL DEFAULT 'Active' COMMENT 'Active, Previous, Archived',
    `hsm_profile_id` BIGINT UNSIGNED,
    `hsm_key_label` VARCHAR(255),
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_kms_key_version` (`kms_key_id`, `version_number`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

...

cloud.kms_wrapped_key

Wrapped DEKs. Each volume references one wrapped key.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_wrapped_key` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `kms_key_id` BIGINT UNSIGNED,
    `kek_version_id` BIGINT UNSIGNED,
    `zone_id` BIGINT UNSIGNED NOT NULL,
    `wrapped_blob` VARBINARY(4096) NOT NULL,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

...

cloud.kms_database_kek_objects

PKCS#11-compatible object storage for the database KMS provider. Key material is encrypted via DBEncryptionUtil (Base64-encoded KEK → DBEncryptionUtil.encrypt() → stored bytes).

CREATE TABLE IF NOT EXISTS `cloud`.`kms_database_kek_objects` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `object_class` VARCHAR(32) NOT NULL DEFAULT 'CKO_SECRET_KEY',
    `label` VARCHAR(255) NOT NULL,
    `object_id` VARBINARY(64),
    `key_type` VARCHAR(32) NOT NULL DEFAULT 'CKK_AES',
    `key_material` VARBINARY(512) NOT NULL COMMENT 'encrypted KEK material',
    `is_sensitive` BOOLEAN NOT NULL DEFAULT TRUE,
    `is_extractable` BOOLEAN NOT NULL DEFAULT FALSE,
    `purpose` VARCHAR(32) NOT NULL,
    `key_bits` INT NOT NULL,
    `algorithm` VARCHAR(64) NOT NULL DEFAULT 'AES/GCM/NoPadding',
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_label` (`label`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

...

Modified Tables

cloud.volumes

Two new columns added via schema migration:

-- KMS key reference
ALTER TABLE `cloud`.`volumes` ADD COLUMN `kms_key_id` BIGINT UNSIGNED;
ALTER TABLE `cloud`.`volumes` ADD CONSTRAINT `fk_volumes__kms_key_id`
    FOREIGN KEY (`kms_key_id`) REFERENCES `kms_keys`(`id`);

-- Wrapped DEK reference
ALTER TABLE `cloud`.`volumes` ADD COLUMN `kms_wrapped_key_id` BIGINT UNSIGNED;
ALTER TABLE `cloud`.`volumes` ADD CONSTRAINT `fk_volumes__kms_wrapped_key_id`
    FOREIGN KEY (`kms_wrapped_key_id`) REFERENCES `kms_wrapped_key`(`id`);

...

User Interface

A new KMS top-level menu section is added with two sub-sections:

...