Key Management Service (KMS) with HSM Integration

This document outlines the design for a comprehensive Key Management Service (KMS) in Apache CloudStack that provides envelope encryption for volume encryption using Hardware Security Modules (HSMs) or a database-backed fallback provider (also used for testing).

The primary goal is to allow users to manage encryption keys (KEKs) for their encrypted volumes through a secure, plugin-based architecture. KEKs can be stored in PKCS#11-compliant HSMs for hardware-grade security or in the CloudStack database for deployments without HSM hardware. The design supports key rotation, cross-HSM migration, and gradual re-encryption of wrapped keys.

GitHub PR: https://github.com/apache/cloudstack/pull/12711

CloudStack Version: 4.23.0.0

Glossary

• KEK (Key Encryption Key): Master key stored in HSM or database, used to wrap/unwrap DEKs. Created via createKMSKey.
• DEK (Data Encryption Key): Per-volume data key used to encrypt volume data. The DEK is wrapped (encrypted) by a KEK and stored in the CloudStack database. Plaintext DEKs exist only transiently in memory.
• HSM Profile: Configuration for connecting to an HSM device (PKCS#11 library path, slot, PIN, etc.). Created via addHSMProfile.
• KMS Key: CloudStack abstraction representing a KEK with versioning support. A logical key that can have multiple KEK versions during rotation.
• Wrapped Key: A DEK encrypted by a KEK, stored as wrapped_blob in the kms_wrapped_key table.
• KEK Version: A specific version of a KEK under a KMS key. During rotation, a new version is created (status=Active) and old versions are marked Previous (still usable for decryption) then Archived.
• Envelope Encryption: A two-layer encryption pattern where data is encrypted with a DEK, and the DEK is encrypted (wrapped) by a KEK.

This hierarchy:

KMS Provider (Plugin) → HSM Profile (Configuration) → KMS Key (KEK abstraction) → KEK Version → Wrapped Key (DEK)

• KMS Provider: The type of cryptographic backend. Examples: database, pkcs11. This corresponds to the plugin implementation (DatabaseKMSProvider, PKCS11HSMProvider).
• HSM Profile: A specific, configured connection to an HSM or key store. Examples: "Production-SoftHSM", "DR-NetHSM". This corresponds to the kms_hsm_profiles table.
• KMS Key: A logical encryption key owned by an account in a zone. Each KMS key has one or more KEK versions. This corresponds to the kms_keys table.
• KEK Version: A specific cryptographic key material version under a KMS key. Only one version is Active at a time. This corresponds to the kms_kek_versions table.
• Wrapped Key: An encrypted DEK associated with a specific volume. This corresponds to the kms_wrapped_key table.

KEK Size vs DEK Size (Important Distinction)

• The KMS key's keybits controls only the KEK size (e.g., 256-bit AES key in the HSM).
• The global setting kms.dek.size.bits controls the DEK size for all new encrypted volumes.
• DEK size is independent of the KMS key; all volumes get DEKs of the configured global size.

Functional Description

Key Functional Components

• HSM Profile Management (Admin only)

  • Add HSM Profile: Configure connection to HSM devices (PKCS#11-compliant)
  • Profile Scoping: User-owned, zone-level, or global (public) profiles
  • Profile Validation: Health check on configured HSM connections
  • Sensitive data encryption: PINs and passwords are encrypted via DBEncryptionUtil before storage

• KMS Key Management

  • Key Creation: Create KMS keys (KEKs) bound to an HSM profile, zone, and account
  • Key Update: Enable/disable keys, update name and description
  • Key Deletion: Soft-delete keys (only if not in use by volumes or wrapped keys)
  • Key Listing: List keys with filtering by purpose, zone, state, and account

• Key Rotation

  • Same-HSM Rotation: Create a new KEK version in the same HSM with a new label
  • Cross-HSM Migration: Create a new KEK version in a different HSM profile
  • Transaction Atomicity: Database updates for new KEK version and key profile are wrapped in a Transaction.execute() block; orphaned HSM keys are cleaned up on DB failure
  • Background Rewrap: Gradual re-encryption of wrapped keys in configurable batches

• Volume Encryption Integration

  • DEK Generation: Generate random DEKs (configurable size via kms.dek.size.bits) and wrap them with the active KEK version
  • DEK Unwrapping: Unwrap DEKs on demand for volume access (plaintext DEKs are zeroized after use)
  • Volume Migration: Migrate legacy passphrase-encrypted volumes to KMS encryption

• Plugin Architecture

  • DatabaseKMSProvider: Database-backed KEK storage with AES/GCM/NoPadding encryption via DBEncryptionUtil
  • PKCS11HSMProvider: PKCS#11 HSM integration with per-profile session pooling and AES/CBC/PKCS5Padding wrapping

• Concurrency & Cluster Safety

  • Bounded thread pool for KMS operations: ThreadPoolExecutor(core=2, max=100, keepAlive=60s, SynchronousQueue) with daemon threads
  • Cluster-aware rewrap: GlobalLock("kms.rewrap.worker") prevents duplicate rewrap work across management server nodes
  • ScheduledExecutorService (replaces java.util.Timer) for robust periodic rewrap scheduling

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    CloudStack API Layer                     │
│  createKMSKey, listKMSKeys, rotateKMSKey, addHSMProfile     │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│              KMSManagerImpl (Implementation)                │
│  - HSM profile management        - Permission checks        │
│  - KEK version management        - Retry with back-off      │
│  - Background rewrap jobs        - Per-operation timeout    │
│  - Cross-HSM migration           - Transaction boundaries   │
└─────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────┐
│           KMSProvider (Plugin Interface)                    │
│  createKek(), wrapKey(), unwrapKey(), rewrapKey()           │
└─────────────────────────────────────────────────────────────┘
         ┌────────────────────┴────────────────────┐
         ↓                                         ↓
┌──────────────────────┐              ┌──────────────────────┐
│ DatabaseKMSProvider  │              │ PKCS11HSMProvider    │
│ - Database storage   │              │ - PKCS#11 interface  │
│ - AES-256-GCM        │              │ - Session pooling    │
│ - Encrypted KEKs     │              │ - AES/CBC/PKCS5      │
└──────────────────────┘              └──────────────────────┘

Key Hierarchy

HSM (PKCS#11) or Database
    └── KEK (Key Encryption Key) - Stored in HSM/DB
        └── KEK Version (supports rotation)
            └── DEK (Data Encryption Key) - Wrapped by KEK
                └── Volume Data - Encrypted by DEK on hypervisor

HSM Profile Scoping

1. User-Owned Profile: account_id set → visible only to that account
2. Zone Admin Profile: zone_id set, account_id NULL → visible to all accounts in that zone
3. Global Admin Profile: zone_id NULL, account_id NULL, is_public = TRUE → visible to all accounts in all zones

Note: createKMSKey requires an explicit hsmprofileid parameter. There is no automatic profile resolution or fallback hierarchy during key creation.

Workflows

Volume Encryption Flow

1. User creates volume or deploys VM with encryption enabled and KMS key specified
2. VolumeOrchestrator calls kmsManager.generateVolumeKeyWithKek()
3. KMSManagerImpl gets the active KEK version, generates a random DEK (size from kms.dek.size.bits), wraps it using the active KEK, and persists the wrapped key
4. Volume is updated with kms_key_id and kms_wrapped_key_id
5. When the hypervisor needs the DEK, kmsManager.unwrapVolumeKey() is called

Key Rotation Flow

1. Admin calls rotateKMSKey with optional hsmprofileid for cross-HSM migration
2. Provider creates a new KEK in the HSM
3. Database updates (new KEK version, old version marked Previous, optional profile update) are executed atomically via Transaction.execute()
4. Background rewrap job (ScheduledExecutorService) gradually rewraps wrapped keys in batches
5. When all wrapped keys are migrated, old KEK version is marked Archived

API Changes

KMS Key APIs

createKMSKey

Creates a new KMS key (Key Encryption Key) for envelope encryption.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Async: No

Parameters:

listKMSKeys

Lists KMS keys available to the caller.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User

Parameters:

updateKMSKey

Updates KMS key name, description, or state.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Async: Yes

Parameters:

deleteKMSKey

Deletes a KMS key (only if not referenced by volumes or wrapped keys).

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Async: Yes

Parameters:

rotateKMSKey

Rotates KEK by creating a new version and scheduling gradual re-encryption of wrapped keys.

• Authorization: Admin only
• Async: Yes

Parameters:

migrateVolumesToKMS

Migrates passphrase-based volumes to KMS encryption.

• Authorization: Admin only
• Async: Yes

Parameters:

HSM Profile APIs

addHSMProfile

Adds a new HSM profile for connecting to an HSM device.

• Authorization: Admin only
• Request has sensitive info: Yes (PIN, passwords)

Parameters:

PKCS#11 details keys: library (path to PKCS#11 library), slot (slot number), pin (HSM PIN, encrypted at rest), token_label (token label), minSessions, maxSessions

listHSMProfiles

Lists HSM profiles visible to the caller.

• Authorization: Admin, ResourceAdmin, DomainAdmin, User
• Response has sensitive info: Yes (encrypted values shown as ENC(...))

Parameters:

updateHSMProfile

Updates an HSM profile name or enabled state.

• Authorization: Admin only

Parameters:

deleteHSMProfile

Deletes an HSM profile (only if not in use by any KEK versions).

• Authorization: Admin only

Parameters:

Global Settings

Database Changes

New Tables

cloud.kms_hsm_profiles

Stores HSM profile configurations. Scoped by account_id, domain_id, and zone_id.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_hsm_profiles` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `protocol` VARCHAR(32) NOT NULL COMMENT 'PKCS11, KMIP, AWS_KMS, etc.',
    `account_id` BIGINT UNSIGNED COMMENT 'null = admin-provided',
    `domain_id` BIGINT UNSIGNED,
    `zone_id` BIGINT UNSIGNED COMMENT 'null = global scope',
    `vendor_name` VARCHAR(64),
    `enabled` BOOLEAN NOT NULL DEFAULT TRUE,
    `is_public` BOOLEAN NOT NULL DEFAULT FALSE,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_account_name` (`account_id`, `name`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_hsm_profile_details

Key-value configuration details for HSM profiles. Sensitive values (PIN, password) are encrypted via DBEncryptionUtil.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_hsm_profile_details` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `profile_id` BIGINT UNSIGNED NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `value` TEXT NOT NULL COMMENT 'encrypted if sensitive',
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_profile_name` (`profile_id`, `name`),
    CONSTRAINT `fk_kms_hsm_profile_details__profile_id`
        FOREIGN KEY (`profile_id`) REFERENCES `kms_hsm_profiles`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_keys

KMS key (KEK) metadata. Account-scoped, zone-bound.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_keys` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `name` VARCHAR(255) NOT NULL,
    `description` VARCHAR(1024),
    `kek_label` VARCHAR(255) NOT NULL,
    `purpose` VARCHAR(32) NOT NULL COMMENT 'VOLUME_ENCRYPTION, TLS_CERT, CONFIG_SECRET',
    `account_id` BIGINT UNSIGNED NOT NULL,
    `domain_id` BIGINT UNSIGNED NOT NULL,
    `zone_id` BIGINT UNSIGNED NOT NULL,
    `algorithm` VARCHAR(64) NOT NULL DEFAULT 'AES/GCM/NoPadding',
    `key_bits` INT NOT NULL DEFAULT 256,
    `enabled` TINYINT(1) NOT NULL DEFAULT 1,
    `hsm_profile_id` BIGINT UNSIGNED NOT NULL,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_kek_versions

KEK versions for gradual key rotation. Only one Active version per KMS key at a time.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_kek_versions` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `kms_key_id` BIGINT UNSIGNED NOT NULL,
    `version_number` INT NOT NULL,
    `kek_label` VARCHAR(255) NOT NULL,
    `status` VARCHAR(32) NOT NULL DEFAULT 'Active' COMMENT 'Active, Previous, Archived',
    `hsm_profile_id` BIGINT UNSIGNED,
    `hsm_key_label` VARCHAR(255),
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_kms_key_version` (`kms_key_id`, `version_number`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_wrapped_key

Wrapped DEKs. Each volume references one wrapped key.

CREATE TABLE IF NOT EXISTS `cloud`.`kms_wrapped_key` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `kms_key_id` BIGINT UNSIGNED,
    `kek_version_id` BIGINT UNSIGNED,
    `zone_id` BIGINT UNSIGNED NOT NULL,
    `wrapped_blob` VARBINARY(4096) NOT NULL,
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

cloud.kms_database_kek_objects

PKCS#11-compatible object storage for the database KMS provider. Key material is encrypted via DBEncryptionUtil (Base64-encoded KEK → DBEncryptionUtil.encrypt() → stored bytes).

CREATE TABLE IF NOT EXISTS `cloud`.`kms_database_kek_objects` (
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
    `uuid` VARCHAR(40) NOT NULL,
    `object_class` VARCHAR(32) NOT NULL DEFAULT 'CKO_SECRET_KEY',
    `label` VARCHAR(255) NOT NULL,
    `object_id` VARBINARY(64),
    `key_type` VARCHAR(32) NOT NULL DEFAULT 'CKK_AES',
    `key_material` VARBINARY(512) NOT NULL COMMENT 'encrypted KEK material',
    `is_sensitive` BOOLEAN NOT NULL DEFAULT TRUE,
    `is_extractable` BOOLEAN NOT NULL DEFAULT FALSE,
    `purpose` VARCHAR(32) NOT NULL,
    `key_bits` INT NOT NULL,
    `algorithm` VARCHAR(64) NOT NULL DEFAULT 'AES/GCM/NoPadding',
    `created` DATETIME NOT NULL,
    `removed` DATETIME,
    PRIMARY KEY (`id`),
    UNIQUE KEY `uk_uuid` (`uuid`),
    UNIQUE KEY `uk_label` (`label`, `removed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Modified Tables

cloud.volumes

Two new columns added via schema migration:

-- KMS key reference
ALTER TABLE `cloud`.`volumes` ADD COLUMN `kms_key_id` BIGINT UNSIGNED;
ALTER TABLE `cloud`.`volumes` ADD CONSTRAINT `fk_volumes__kms_key_id`
    FOREIGN KEY (`kms_key_id`) REFERENCES `kms_keys`(`id`);

-- Wrapped DEK reference
ALTER TABLE `cloud`.`volumes` ADD COLUMN `kms_wrapped_key_id` BIGINT UNSIGNED;
ALTER TABLE `cloud`.`volumes` ADD CONSTRAINT `fk_volumes__kms_wrapped_key_id`
    FOREIGN KEY (`kms_wrapped_key_id`) REFERENCES `kms_wrapped_key`(`id`);

User Interface

A new KMS top-level menu section is added with two sub-sections:

KMS Keys

• List View: Name, Enabled, Purpose, HSM Profile, Account, Domain columns
• Detail View: ID, Name, Description, Version, Enabled, Account, Domain, Created, HSM Profile
• Related: Volumes tab (linked via kmskeyid)
• Actions:
  • Create KMS Key (all users)
  • Update KMS Key (all users)
  • Rotate KMS Key (all users)
  • Migrate Volumes to KMS (admin only)
  • Delete KMS Key (all users)
• Search Filters: Zone, HSM Profile (admins also get: Account, Domain, Project)

HSM Profiles

• List View: Name, Enabled, Account, Domain columns
• Detail View: ID, Name, Description, Enabled, Account, Domain, Created, Details (key-value pairs)
• Related: KMS Keys tab (linked via hsmprofileid)
• Actions (admin only — non-admin users can only view the list):
  • Add HSM Profile
  • Update HSM Profile
  • Delete HSM Profile
• Search Filters: Zone (admins also get: Account, Domain, Project)

Security Considerations

1. Key Storage: KEKs never leave the HSM (for PKCS#11 provider) or are stored encrypted in database (for database provider). DEKs are always stored wrapped.

2. Sensitive Data: HSM profile PINs/passwords encrypted via DBEncryptionUtil before storage. Encrypted values appear as ENC(...) in API responses.

3. Access Control: Only root admins can manage HSM profiles (create, update, delete). All users can list HSM profiles visible to them. KMS keys are account-scoped with domain hierarchy permissions.

4. Key Rotation: Background rewrap job gradually re-encrypts wrapped keys to avoid service disruption. Old KEK versions retained until all wrapped keys migrated. Cross-HSM migration supported.

5. PKCS#11 Cipher: AES-CBC with PKCS5Padding (FIPS-compliant, universal driver support). AES-GCM preferred but SunPKCS11 support is unreliable across drivers.

6. Error Handling: Per-operation timeout via Future.get(), retry with back-off for transient failures, fast-fail for non-retryable errors, graceful fallback to passphrase encryption.

7. Cluster Safety: GlobalLock("kms.rewrap.worker") ensures only one management server runs the rewrap job. Key rotation DB updates use Transaction.execute() for atomicity with compensating HSM cleanup on failure.

Future Enhancements

• Usage Records: Track KMS key usage (wrap/unwrap operations, active wrapped key count) and emit usage events for billing integration with CloudStack's usage server

• Scheduled Key Rotation Policies: Allow admins to define rotation policies (e.g., rotate every 90 days) with automatic KEK rotation and rewrap scheduling per KMS key

• Managed HSM Offering: Provide HSM connectivity as a CloudStack service offering, allowing tenants to provision dedicated or shared HSM partitions (requires further investigation into HSM partitioning models and multi-tenancy isolation)

• HSM Profile Detail Updates: Support updating HSM profile configuration details (e.g., PIN rotation) without requiring delete and re-create
• Multi-Zone KEK Replication: Replicate KEKs across zones for disaster recovery, enabling volume failover without losing access to encryption keys
• AES-GCM for PKCS#11: Add AES-GCM support for DEK wrapping in PKCS11HSMProvider (pending broader SunPKCS11 driver compatibility)
• Key Access Audit Logging: Detailed audit trail for all KEK and DEK access events (who accessed which key, when, from which management server)
• KMIP Protocol Support: Add KMIP provider plugin for enterprise key management interoperability