Page History

...

1.  Introduce `CLVM` and `CLVM_NG` as first-class `StoragePoolType` entries in CloudStack. CLVM supports RAW volumes on LVM and CLVM_NG support QCOW2.
2. Design a lock-tracking layer that records and reconciles which KVM host holds the exclusive LVM activation for each volume.
3. Implement a lock-transfer mechanism so CloudStack can safely move the exclusive lock between hosts without copying data.
4. Introduce a three-tier migration strategy that avoids unnecessary data copying:
      - Lock Transfer - same pool, wrong host: transfer lock only.
      - Lightweight Migration - different CloudStack pool records sharing the same LVM Volume Group: transfer lock only.Same LV but different hosts - no data copy, just lock transfer
      - Full Migration - different Volume Groups: full data copy (existing behavior).
5. Provide a secure-erase option for LVs at deletion time to prevent data leakage.
6. Incremental snapshots for CLVM_NG leveraging bitmap - not supported in 4.23
   Note: Only full snapshots will be supported for clvm and clvm-ng

LVM Activation Modes

Architecture Overview:

┌─────────────────────────────────────────────────────────────┐
│              CloudStack Management Server                    │
└──────────────────────┬──────────────────────────────────────┘
                       │
    ┌──────────────────┼──────────────────┐
    │                  │                  │
┌───▼─────────┐  ┌────▼──────┐  ┌───────▼────────┐
│  VolumeApi  │  │  Volume   │  │     Default    │
│ ServiceImpl │  │Orchestrator│  │EndPointSelector│
│             │  │            │  │                │
│ • attach    │  │ • VM start │  │ • Route ops    │
│ • detach    │  │ • migrate  │  │   to lock host │
│ • migrate   │  │ • prepare  │  │ • Query lock   │
└─────────────┘  └────────────┘  └────────────────┘
       │                │                │
       └────────────────┼────────────────┘
                        │
              ┌─────────▼─────────┐
              │ VolumeServiceImpl │
              │                   │
              │ • performLock     │
              │   Migration       │
              │ • findLockHost    │
              │ • isLightweight   │
              │   Needed          │
              └─────────┬─────────┘
                        │
              ┌─────────▼─────────┐
              │  ClvmLockManager  │
              │                   │
              │ • transferVolume  │
              │   Lock            │
              │ • queryCurrent    │
              │   LockHolder      │
              │ • setClvmLock     │
              │   HostId          │
              └─────────┬─────────┘
                        │ (AgentManager)
                        │
        ┌───────────────┴───────────────┐
        │                               │
┌───────▼────────┐             ┌───────▼────────┐
│  KVM Agent 1   │             │  KVM Agent N   │
│  (Host A)      │             │  (Host B)      │
│                │             │                │
│  LibvirtClvm   │             │  LibvirtClvm   │
│  LockTransfer  │             │  LockTransfer  │
│  Wrapper       │             │  Wrapper       │
│                │             │                │
│  lvchange -an  │             │  lvchange -aey │
│  lvchange -aey │             │  lvs -o ...    │
│  lvs           │             │                │
└────────────────┘             └────────────────┘
        │                               │
        └───────────────┬───────────────┘
                        │
            ┌───────────▼────────────┐
            │  Shared Storage (SAN)  │
            │  LVM Volume Group      │
            │  /dev/vg-cluster01/... │
            └────────────────────────┘

...

• Source of truth: The actual LVM state is obtained via lvs, not the database.
• Database as cache: volume_details.clvmLockHostId tracks which host has the lock for debugging purposes, but may be stale.
• Sequential operations: Lock transfers execute sequentially via executeInSequence() to prevent races.
• Idempotency: Lock transfer to the same host is a no-op

...

Table: volume_details
Key: clvmLockHostId (constant in {{VolumeInfoClvmPoolManager.java}})
Value: Host ID (Long)

...

• ClvmLockTransferCommand / ClvmLockTransferAnswer
  1. Operations:
     

     Code Block
     public enum Operation { DEACTIVATE("-an", "deactivate"), ACTIVATE_EXCLUSIVE("-aey", "activate exclusively"), ACTIVATE_SHARED("-asy", "activate in shared mode"), QUERY_LOCK_STATE("query", "query lock state"); }

     
     

  2. Command Properties

     Property	Value	Description
     executeInSequence()	true	Prevents concurrent lock operations on same LV
     setWait(30)	30 secs	Agent timeout per operation
     Fields	lvPath, operation, volumeUuid	command parameters

     
     

  3. Answer Fields
     

     Field	Type	Description
     currentLockHostname	String	Hostname holding the lock from lv_host field
     isActive	boolean	Whether LV is active anywhere
     isExclusive	boolean	Weather the active lock is exclusive
     lvAttributes	String	Raw lv_attr string for debugging

     
     
     
• ClvmLockManager
  

  getClvmLockHostId(volumeId, volumeUuid, volumePath, pool, queryActual)

  Purpose:Returns the host ID holding the exclusive lock
  
  

  Code Block
  if queryActual == true: return queryCurrentLockHolder(...) // Query LVM directly else: return database value from volume_details.clvmLockHostId

  
  
  Parameters:

• queryActual=true: Bypass database, query LVM state (expensive but accurate)

• queryActual=false: Fast database lookup (may be stale)

...

          Purpose: Queries actual LVM lock state from KVM hosts

1. Resolve hosts for pool:
   ├─ If pool.getClusterId() queryCurrentLockHolder(volumeId, volumeUuid, volumePath, pool, updateDatabase):
├─ lvPath = "/dev/{vgName}/{volumePath}"
│
├─ [Fast path] dbHostId = DB lookup (volume_details.clvmLockHostId)
│  └─ If dbHostId != null:
│   │  ├─ If dbHost hostsis = hostDao.findByClusterId(clusterId, Type.Routing)
Up and KVM:
│   └─  │ Else if zone-scoped:
        hosts = hostDao.findByDataCenterId(zoneId)

2. Filter to UP KVM routing hosts

3. For each host in hosts:
   ├─ Send QUERY_LOCK_STATE command
   ├─ If successful:
   │  ├─ Parse ClvmLockTransferAnswer├─ Send QUERY_LOCK_STATE to dbHostId
│     │  ├─ If active == TRUE:
│     │  │    return dbHostId                        // confirmed, early exit
│     │  └─ Else: log "fast path miss, falling back to fan-out"
│     └─ Else: log "fast path skip (host down/missing), falling back to fan-out"
│
├─ [Fan-out] Resolve hosts for pool:
│  ├─ If pool.getClusterId() != null:
│  │    hosts = hostDao.findByClusterId(clusterId, Type.Routing)
│  └─ Else if zone-scoped:
│       hosts = hostDao.findByDataCenterId(zoneId)
│
├─ Filter to UP KVM routing hosts, skip dbHostId (already checked above)
│
├─ For each host in hosts:
│  ├─ Send QUERY_LOCK_STATE command
│  └─ If active == TRUE: add to activeHostIds
│
└─ Evaluate results:
   ├─ If activeHostIds is empty:
   │  ├─ If updateDatabase and dbHostId != null:
   │  │    remove CLVM_LOCK_HOST_ID from volume_details
   │  └─ return null
   ├─ If activeHostIds.size > 1:
   │  ├─ Extract hostname from lv_host log warning "shared-mode LV (template?), skipping"
   │  ├─└─ Resolve hostname → host IDreturn null
   │└─ Else ├─ if updateDatabase(exactly one):
   │  │ ├─ If updateDatabase setClvmLockHostId(volumeId, hostId)
   │  └─ return hostId
   └─ If failed, try next host

4. If all hosts fail:
   ├─ Log warning
   └─ return nulland lockHostId != dbHostId:
      │    setClvmLockHostId(volumeId, lockHostId)   ← correct stale DB record
      └─ return lockHostId

Performance Considerations:

...

Purpose: Transfers exclusive lock from source to destination host

1. actualLockHostId = queryCurrentLockHolder(..., queryActual=false)
   // Get real lock holder, not database hint

2. hostToDeactivate = actualLockHostId ?? sourceHostId
   // Use actual holder if known, fallback to provided source

32. if hostToDeactivate != null AND hostToDeactivate != destHostId:
   ├─ if host is UP:
   │    send DEACTIVATE command to hostToDeactivate
   └─ if host is DOWN:
        log warning "Host down, will force claim on destination"

43. send ACTIVATE_EXCLUSIVE command to destHostId
   // LVM force-claims if previous holder left cleanly

54. if activation successful:
   ├─ setClvmLockHostId(volumeId, destHostId)
   └─ return true

65. return false

setClvmLockHostId(volumeId, hostId)

...

VirtualMachineManagerImpl.migrate()

1. Pre-migration checks and preparation:
├─ volumeMgr.prepareForMigration(profile, dest)
├─ Generate MigrateCommand with VM and disk information
├─ PreMigrationCommand already executed (Phase 0: Source → SHARED)
└─ PrepareForMigrationCommand already executed (Phase 1: Dest → SHARED)

2. Initiate libvirt live migration:
├─ Command: domain.migrate(destConn, xmlDesc, migrateFlags)
├─ Libvirt uses shared storage
├─ Transfers memory state over network
├─ QEMU on destination opens block devices
└─ BOTH hosts have volumes in SHARED mode (dual activation)

3. After successful migration:
├─ Deactivate CLVM volumes on source host:
│ LibvirtComputingResource.modifyClvmVolumesStateForMigration(
│ disks, resource, vmSpec, ClvmVolumeState.DEACTIVATE)
│ → executes: lvchange -an /dev/<vg>/<lv> on source
│ → Destination volumes remain ACTIVE in SHARED mode
│
└─ Update CloudStack database tracking:
updateClvmLockHostForVmVolumes(vm.getId(), destHost.getId())
→ sets CLVM_LOCK_HOST_ID = destHost for all volumes

4. On migration failure:
├─ Revert CLVM volumes to EXCLUSIVE mode on source:
│ LibvirtComputingResource.modifyClvmVolumesStateForMigration(
│ disks, resource, vmSpec, ClvmVolumeState.EXCLUSIVE)
│ → executes: lvchange -aey /dev/<vg>/<lv> on source
│
└─ Destination volumes are deactivated/cleaned up via rollback

...

cloudmonkey create storagepool \
  zoneid=<zone-id> \
  podid=<pod-id> \
  clusterid=<cluster-id> \
  name="clvm-pool-01" \
  url="clvm:///<vg-name>" \
  scope=cluster



cloudmonkey create storagepool \
  zoneid=<zone-id> \
  podid=<pod-id> \
  clusterid=<cluster-id> \
  name="clvm-ng-pool-01" \
  url="clvm_ng:///<vg-name>" \
  scope=cluster

Global Setting

When enabled, KVM agents execute dd if=/dev/zero of=<lv> before lvremove. This prevents the next VM allocated to that storage space from reading previous tenant data.

...