Shared NFS Zone-wide (primary) Block Storage

Background: When a data disk is needed by a VM that resides in a different cluster (i.e. different primary storage) than the current cluster/primary storage, it involves a costly/time consuming data copy operation - which involved copying data from one primary storage to another primary storage via the secondary storage. A primary storage is not shared across multiple clusters. As an example, AWS EBS provides the following benefits: EBS volumes are placed in a specific Availability Zone, and can then be attached to any instance in that same Availability Zone. Even though, a volume can only be attached to one instance at a time, if an instance fails or is detached from an EBS volume, the volume can be attached to any other instance in that Availability Zone.

Requirement: CloudStack to support Shared NFS zone-wide (primary) block storage

- Multiple clusters can share the same NFS primary storage (as long as the underlying hypervisor platform supports it)

- Only one VM can mount/attach a disk at any time (i.e. multiple VMs cannot attach the same disk concurrently)

- snapshot/restore features are similar to current semantics i.e. no degradation in features

Open questions

- ideally, instead of making the NFS share be available as a primary storage for all clusters (unless explicitly excluded by a cluster), would it be better if we simply allow more than one cluster to specify the same NFS share as a primary storage (latter is preferable)

- what would be the migration issues/requirements - for example, do we allow an existing NFS share that is a primary storage already to be "upgraded" to a zone-wide primary storage? 

- is there any impact of doing maintenance of this nfs? Or removing the primary share for a cluster?

- what is the performance/scalability limitation?