Introduction

An Object Storage service can store a large amount of unstructured data of any content type. Object Stores are widely used for Big Data analytics, Backup/Archival, Web Hosting and media storage and delivery. Amazon S3, Google Cloud Storage, Azure Blob Storage and DigitalOcean Spaces are some of the popular Object Storage service providers. 

Building on the OpenSource object store providers, CloudStack can support Object Storage as a first-class entity. A general purpose object storage framework should be added, with support for various Object Storage providers as plugins. CloudStack will manage the lifecycle of the object storage pools and buckets. Objects will not be managed by CloudStack. The underlying object-storage provider is responsible for flexible storage management and data growth, multi-site replication, security, and backup.

Purpose

The purpose if this document is to provide functional specification for Object Storage feature in CloudStack.

References

Document History

VersionAuthor/ReviewerDate
1.0Kishan Kavala

 




Glossary

Object Store: Object storage (also known as object-based storage) is a data storage that manages data as objects

Bucket: Buckets are logical containers for storing objects

Object: The basic storage units of Object Store are objects. Any type of data, regardless of content type, is stored as an object.

Object Storage Pool: CloudStack admin can setup supported Object Storage systems and add them to CloudStack as an Object Storage Pool. Users can create buckets within the object storage pool 


Feature Specifications

  • Object storage as a first-class end user feature in CloudStack

  • Pluggable Object storage provider framework. MinIO will added as a supported provider in the initial implementation 

  • Ability to create buckets and manage their lifecycle

Use cases

  • CloudStack admin should be able to add supported Object Storage providers (min.io based infra, ceph object storage etc.).
  • CloudStack users should be able to create buckets and manage their lifecycle.
  • Users/projects get billed by aggregate data storage on per bucket basis

Architecture and Design description

ObjectStoreDriver interface will be defined. All supported object store providers will need to implement this interface. 

API and Schema Changes

New APIs

Object Storage Pool APIs

  • addObjectStoragePool

    • name
    • description
    • URL
    • Provider
    • Details - Map

  • updateObjectStoragePool

    • id
    • name
    • description

  • removeObjectStoragePool

    • id

  • listObjectStoragePools

    • id


Bucket APIs

  • createBucket

    • name
    • description
    • object_storage_id
    • encryption
    • versioning
    • object_lock
    • policy

  • deleteBucket

    • id

  • listBuckets

    • id
    • provider
    • object_storage_id
  • updateBucket
    • id
    • name
    • description
    • encryption
    • versioning
    • object_lock
    • policy


Schema Changes

New Tables:

  • object_store
    • id
    • uuid
    • name
    • description
    • provider
    • url
    • total_size
    • used_size
    • created
    • removed
  • object_store_details
    • id
    • store_id
    • name
    • value
  • bucket
    • id
    • uuid
    • name
    • description
    • store_id
    • size
    • url
    • created
    • removed

Service Layer Changes

<ToDo>

UI Changes

  • UI for managing object storage pools
  • UI for bucket operations
  • S3 compatible UI client for managing objects

Usage Impact

  • Buckets usage will be added as a new Usage resource type

    • Size of the bucket will be included as key metric for billing purposes

Appendix

Appendix A:

Appendix B: