Modernized CLVM: Enhancements and CLVM_NG support

Introduction:

KVM clusters that use shared block storage (SAN, iSCSI, Fibre Channel) manage disks as LVM Logical Volumes inside a Volume Group visible across all nodes. LVM's cluster-aware variants - CLVM and its modern successor CLVM_NG using `lvmlockd`/`sanlock` - provide the locking infrastructure that prevents simultaneous writes from multiple hosts.

CloudStack today has no first-class understanding of these pool types. When a VM migrates or a volume is attached to a different VM, CloudStack performs a full volume data copy even when the data has not moved at all. Worse, after a migration CloudStack has no record of which host holds the current LVM exclusive lock, so the next VM start may fail because the lock is stale on the wrong host.

CLVM/CLVM_NG are Linux LVM construct and this design only targets KVM.

Scope:

Introduce `CLVM` and `CLVM_NG` as first-class `StoragePoolType` entries in CloudStack. CLVM supports RAW volumes on LVM and CLVM_NG support QCOW2.
Design a lock-tracking layer that records and reconciles which KVM host holds the exclusive LVM activation for each volume.
Implement a lock-transfer mechanism so CloudStack can safely move the exclusive lock between hosts without copying data.
Introduce a three-tier migration strategy that avoids unnecessary data copying:
- Lock Transfer - same pool, wrong host: transfer lock only.
- Lightweight Migration - Same LV but different hosts - no data copy, just lock transfer
- Full Migration - different Volume Groups: full data copy (existing behavior).
Provide a secure-erase option for LVs at deletion time to prevent data leakage.
Incremental snapshots for CLVM_NG leveraging bitmap - not supported in 4.23
Note: Only full snapshots will be supported for clvm and clvm-ng

LVM Activation Modes

Flag	Meaning
lvchange -aey <lv>	Activate exclusively - one host only
lvchange -asy <lv>	Activate shared : read-only on all cluster nodes
lvchange -an <lv>	Deactivate

Architecture Overview:

┌─────────────────────────────────────────────────────────────┐
│              CloudStack Management Server                    │
└──────────────────────┬──────────────────────────────────────┘
                       │
    ┌──────────────────┼──────────────────┐
    │                  │                  │
┌───▼─────────┐  ┌────▼──────┐  ┌───────▼────────┐
│  VolumeApi  │  │  Volume   │  │     Default    │
│ ServiceImpl │  │Orchestrator│  │EndPointSelector│
│             │  │            │  │                │
│ • attach    │  │ • VM start │  │ • Route ops    │
│ • detach    │  │ • migrate  │  │   to lock host │
│ • migrate   │  │ • prepare  │  │ • Query lock   │
└─────────────┘  └────────────┘  └────────────────┘
       │                │                │
       └────────────────┼────────────────┘
                        │
              ┌─────────▼─────────┐
              │ VolumeServiceImpl │
              │                   │
              │ • performLock     │
              │   Migration       │
              │ • findLockHost    │
              │ • isLightweight   │
              │   Needed          │
              └─────────┬─────────┘
                        │
              ┌─────────▼─────────┐
              │  ClvmLockManager  │
              │                   │
              │ • transferVolume  │
              │   Lock            │
              │ • queryCurrent    │
              │   LockHolder      │
              │ • setClvmLock     │
              │   HostId          │
              └─────────┬─────────┘
                        │ (AgentManager)
                        │
        ┌───────────────┴───────────────┐
        │                               │
┌───────▼────────┐             ┌───────▼────────┐
│  KVM Agent 1   │             │  KVM Agent N   │
│  (Host A)      │             │  (Host B)      │
│                │             │                │
│  LibvirtClvm   │             │  LibvirtClvm   │
│  LockTransfer  │             │  LockTransfer  │
│  Wrapper       │             │  Wrapper       │
│                │             │                │
│  lvchange -an  │             │  lvchange -aey │
│  lvchange -aey │             │  lvs -o ...    │
│  lvs           │             │                │
└────────────────┘             └────────────────┘
        │                               │
        └───────────────┬───────────────┘
                        │
            ┌───────────▼────────────┐
            │  Shared Storage (SAN)  │
            │  LVM Volume Group      │
            │  /dev/vg-cluster01/... │
            └────────────────────────┘

Key design principles:

Source of truth: The actual LVM state is obtained via lvs, not the database.
Database as cache: volume_details.clvmLockHostId tracks which host has the lock for debugging purposes, but may be stale.
Sequential operations: Lock transfers execute sequentially via executeInSequence() to prevent races.
Idempotency: Lock transfer to the same host is a no-op

Lock Model and State Management

LVM Lock Types

Exclusive Lock: -aey

Only one host can have an exclusive lock on an LV at a time
Required for read-write access
Used when VM is running or volume is being modified
Command: lvchange -aey /dev/vg/lv

Shared Lock: -asy

Multiple hosts can have shared read-only access
Not typically used for VM volumes; used at the time of migration
Command: lvchange -asy /dev/vg/lv

Deactivated:-an

LV is not accessible on this host
Releases any locks held
Command: lvchange -an /dev/vg/lv

LV Attribute Interpretation

The lvs command returns attributes that reveal lock state:

$ lvs -o lv_name,lv_attr,lv_host --noheadings /dev/vg0/vm-123-disk-0
  vm-123-disk-0  -wi-a-e---  host5.example.com

Position 4 (0-indexed): Active flag

a = active on at least one host
- = not active anywhere

Position 5 (0-indexed): Exclusivity flag

e = exclusively locked (read-write)
s = shared lock (read-only)
- = not exclusively locked

lv_host field: Hostname of the lock holder

Database State Tracking

Table: volume_details
Key: clvmLockHostId (constant in {{ClvmPoolManager.java}})
Value: Host ID (Long)

This is a cache/hint, not the source of truth. It can become stale when:

A host crashes without releasing locks
Manual LVM operations occur outside CloudStack
Network partitions cause lock manager state divergence

Reconciliation: ClvmLockManager.queryCurrentLockHolder() queries actual LVM state and optionally updates the database.

Core Components

ClvmLockTransferCommand / ClvmLockTransferAnswer

Operations:

public enum Operation {
    DEACTIVATE("-an", "deactivate"),
    ACTIVATE_EXCLUSIVE("-aey", "activate exclusively"),
    ACTIVATE_SHARED("-asy", "activate in shared mode"),
    QUERY_LOCK_STATE("query", "query lock state");
}

Command Properties
Property Value Description
executeInSequence() true
Prevents concurrent lock operations on same LV
setWait(30) 30 secs Agent timeout per operation
Fields lvPath, operation, volumeUuid command parameters

Answer Fields

Field	Type	Description
currentLockHostname	String	Hostname holding the lock from lv_host field
isActive	boolean	Whether LV is active anywhere
isExclusive	boolean	Weather the active lock is exclusive
lvAttributes	String	Raw lv_attr string for debugging

ClvmLockManager
getClvmLockHostId(volumeId, volumeUuid, volumePath, pool, queryActual)
Purpose: Returns the host ID holding the exclusive lock
```
if queryActual == true:
return queryCurrentLockHolder(...) // Query LVM directly
else:
return database value from volume_details.clvmLockHostId
```
Parameters:
queryActual=true: Bypass database, query LVM state (expensive but accurate)
queryActual=false: Fast database lookup (may be stale)

queryCurrentLockHolder(volumeId, volumeUuid, volumePath, pool, updateDatabase)

Purpose: Queries actual LVM lock state from KVM hosts

queryCurrentLockHolder(volumeId, volumeUuid, volumePath, pool, updateDatabase):
├─ lvPath = "/dev/{vgName}/{volumePath}"
│
├─ [Fast path] dbHostId = DB lookup (volume_details.clvmLockHostId)
│  └─ If dbHostId != null:
│     ├─ If dbHost is Up and KVM:
│     │  ├─ Send QUERY_LOCK_STATE to dbHostId
│     │  ├─ If active == TRUE:
│     │  │    return dbHostId                        // confirmed, early exit
│     │  └─ Else: log "fast path miss, falling back to fan-out"
│     └─ Else: log "fast path skip (host down/missing), falling back to fan-out"
│
├─ [Fan-out] Resolve hosts for pool:
│  ├─ If pool.getClusterId() != null:
│  │    hosts = hostDao.findByClusterId(clusterId, Type.Routing)
│  └─ Else if zone-scoped:
│       hosts = hostDao.findByDataCenterId(zoneId)
│
├─ Filter to UP KVM routing hosts, skip dbHostId (already checked above)
│
├─ For each host in hosts:
│  ├─ Send QUERY_LOCK_STATE command
│  └─ If active == TRUE: add to activeHostIds
│
└─ Evaluate results:
   ├─ If activeHostIds is empty:
   │  ├─ If updateDatabase and dbHostId != null:
   │  │    remove CLVM_LOCK_HOST_ID from volume_details
   │  └─ return null
   ├─ If activeHostIds.size > 1:
   │  ├─ log warning "shared-mode LV (template?), skipping"
   │  └─ return null
   └─ Else (exactly one):
      ├─ If updateDatabase and lockHostId != dbHostId:
      │    setClvmLockHostId(volumeId, lockHostId)   ← correct stale DB record
      └─ return lockHostId

Performance Considerations:

Queries one host at a time until success
Network overhead
Worst case: Queries all hosts in cluster/zone

transferClvmVolumeLock(volumeUuid, volumeId, volumePath, pool, sourceHostId, destHostId)

Purpose: Transfers exclusive lock from source to destination host

1. hostToDeactivate = sourceHostId
   // Use actual holder if known, fallback to provided source

2. if hostToDeactivate != null AND hostToDeactivate != destHostId:
   ├─ if host is UP:
   │    send DEACTIVATE command to hostToDeactivate
   └─ if host is DOWN:
        log warning "Host down, will force claim on destination"

3. send ACTIVATE_EXCLUSIVE command to destHostId
   // LVM force-claims if previous holder left cleanly

4. if activation successful:
   ├─ setClvmLockHostId(volumeId, destHostId)
   └─ return true

5. return false

setClvmLockHostId(volumeId, hostId)

Purpose: Updates the lock host ID in the database

existingDetail = volumeDetailsDao.findDetail(volumeId, CLVM_LOCK_HOST_ID)
if existingDetail != null:
    existingDetail.setValue(String.valueOf(hostId))
    volumeDetailsDao.update(existingDetail.getId(), existingDetail)
else:
    volumeDetailsDao.addDetail(volumeId, CLVM_LOCK_HOST_ID, 
                                String.valueOf(hostId), false)

LibvirtClvmLockTransferCommandWrapper

It's the KVM agent side handler.

Execution Flow:

1. Receive ClvmLockTransferCommand from management server

2. if operation == QUERY_LOCK_STATE:
   return handleQueryLockState()

3. Map operation to lvchange flag:
   DEACTIVATE         → "-an"
   ACTIVATE_EXCLUSIVE → "-aey"
   ACTIVATE_SHARED    → "-asy"

4. Execute: /usr/sbin/lvchange <flag> <lvPath>
   timeout: 30 seconds

5. if command fails:
   return ClvmLockTransferAnswer(success=false, details=error)

6. return ClvmLockTransferAnswer(success=true)

Query Lock State Flow:

1. Execute: /usr/sbin/lvs -o lv_attr,lv_host --noheadings <lvPath>

2. Parse output:
   "  -wi-a-e---  host5.example.com"
   
3. Extract:
   lvAttr = "-wi-a-e---"
   hostname = "host5.example.com"
   
4. Derive state:
   isActive = (lvAttr[4] == 'a')
   isExclusive = (lvAttr[5] == 'e')

5. return ClvmLockTransferAnswer(
       success=true,
       currentLockHostname=hostname,
       isActive=isActive,
       isExclusive=isExclusive,
       lvAttributes=lvAttr
   )

Host Selection and Endpoint Routing

DefaultEndPointSelector: Responsible for routing storage operations to the correct KVM host

Volume creation with destinationHostId as hint

select(VolumeInfo volume, String operation):
    if volume.getDestinationHostId() != null:
        if pool is CLVM type:
            return getEndPointFromHostId(volume.getDestinationHostId())
            // Ensures volume is created with exclusive lock on target host

Volume operations - existing volumes

select(DataObject object):
    if object is VolumeInfo AND pool is CLVM type:
        lockHostId = clvmLockManager.getClvmLockHostId(..., queryActual=true)
        if lockHostId != null:
            return getEndPointFromHostId(lockHostId)
            // Route to host holding the lock
    return default endpoint selection

Copy Operations - Primary to secondary

select(DataObject srcData, DataObject destData):
    if srcData is VolumeInfo AND srcPool is CLVM type:
        lockHostId = getClvmLockHostId(srcVolume)
        if lockHostId != null:
            return getEndPointFromHostId(lockHostId)
            // Read from lock holder (volume is active there)

Host selection Hierarchy

When creating or accessing CLVM volumes, CloudStack uses this priority:

Explicit destination host (for new volumes during VM creation)
Current lock holder (for existing volumes)
VM's host (for volumes attached to running VMs)
First UP host in cluster (fallback)
First UP host in zone (fallback for zone-scoped pools)

Volume Lifecycle Operations

Volume Creation

VolumeOrchestrator.createVolume()

 Determine target host for VM placement
   (via DeploymentPlanner)

2. if pool is CLVM type:
   volume.setDestinationHostId(targetHostId)
   // Pre-set destination hint

3. Create volume via VolumeServiceImpl:
   ├─ DefaultEndPointSelector routes to targetHostId
   └─ KVM agent executes:
       lvcreate -L <size> -n <lv_name> <vg_name>
       lvchange -aey /dev/<vg>/<lv>
   
4. setClvmLockHostId(volumeId, targetHostId)

Result: Volume created with exclusive lock already on the host where VM will start.

2. Volume Deletion

VolumeServiceImpl.deleteVolume()

Delete volume via storage driver:
   ├─ KVM agent executes:
   │   lvchange -an /dev/<vg>/<lv>
   │   lvremove /dev/<vg>/<lv>
   
2. clvmLockManager.clearClvmLockHostDetail(volume)
   // Remove database tracking

3. volumeDao.remove(volumeId)

Considers StoragePool level setting `clvm.secure.zero.fill` if when set to true, zero fills the lv before deletion for security reasons. Recommended to be set to true in production

3. Volume Attach to Running VM

VolumeApiServiceImpl.attachVolumeToVM()

if volume and VM are on different pools:
    if both pools point to same VG:
        // Lightweight migration
        executeLightweightLockMigration(volume, vm)
    else:
        // Full migration
        volumeMigrationService.migrateVolume(volume, destPool)
else:
    // Same pool
    if volume lock is on different host than VM:
        // Lock transfer
        volService.performLockMigration(volume, vm.getHostId())
    
// Actual libvirt attach
attachVolume(vm, volume)

4. Detach Volume

VolumeApiServiceImpl.detachVolume()

1. Detach volume via libvirt API
   (volume remains activated exclusively on host)

2. volume_details.clvmLockHostId remains unchanged
   // Lock is retained on the host where VM ran

3. On next attach:
   ├─ queryCurrentLockHolder() will find actual state
   └─ performLockMigration() if needed

Note: Keeping the lock on detach avoids unnecessary lock churn if volume is quickly re-attached to the same VM.

VM Operations

1. VM Start

VolumeOrchestrator.prepare()

1. Select destination host via DeploymentPlanner

2. transferClvmLocksForVmStart(volumes, destHostId, vm):
   for each volume in VM's volumes:
       if pool is not CLVM type:
           continue
       
       currentLockHost = clvmLockManager.getClvmLockHostId(..., queryActual=true)
       
       if currentLockHost == null:
           clvmLockManager.setClvmLockHostId(volume.getId(), destHostId)
       else if currentLockHost != destHostId:
           transferClvmVolumeLock(volume, currentLockHost, destHostId)

3. Start VM via libvirt

Note: All CLVM volumes are transferred to the VM's host *before* VM start, ensuring all disks are accessible when QEMU launches.

2. VM Stop

VirtualMachineManagerImpl.stop()

1. Stop VM via libvirt

2. CLVM volumes remain activated exclusively on the stopped VM's host
   // No lock release on stop

3. volume_details.clvmLockHostId remains unchanged

Note: Locks are retained to optimize quick restarts. Next start will transfer locks if VM is scheduled to a different host.

3. VM Live Migration

VirtualMachineManagerImpl.migrate()

1. Pre-migration checks and preparation:
├─ volumeMgr.prepareForMigration(profile, dest)
├─ Generate MigrateCommand with VM and disk information
├─ PreMigrationCommand already executed (Phase 0: Source → SHARED)
└─ PrepareForMigrationCommand already executed (Phase 1: Dest → SHARED)

2. Initiate libvirt live migration:
├─ Command: domain.migrate(destConn, xmlDesc, migrateFlags)
├─ Libvirt uses shared storage
├─ Transfers memory state over network
├─ QEMU on destination opens block devices
└─ BOTH hosts have volumes in SHARED mode (dual activation)

3. After successful migration:
├─ Deactivate CLVM volumes on source host:
│ LibvirtComputingResource.modifyClvmVolumesStateForMigration(
│ disks, resource, vmSpec, ClvmVolumeState.DEACTIVATE)
│ → executes: lvchange -an /dev/<vg>/<lv> on source
│ → Destination volumes remain ACTIVE in SHARED mode
│
└─ Update CloudStack database tracking:
updateClvmLockHostForVmVolumes(vm.getId(), destHost.getId())
→ sets CLVM_LOCK_HOST_ID = destHost for all volumes

4. On migration failure:
├─ Revert CLVM volumes to EXCLUSIVE mode on source:
│ LibvirtComputingResource.modifyClvmVolumesStateForMigration(
│ disks, resource, vmSpec, ClvmVolumeState.EXCLUSIVE)
│ → executes: lvchange -aey /dev/<vg>/<lv> on source
│
└─ Destination volumes are deactivated/cleaned up via rollback

4. VM Cold Migration