...
The goal is to provide the S3 REST API calls in CloudStack. Specifically
Allow the solution to be installable via a self contained virtual system server within a CloudStack management server.
...
Scope of this document is to provide a functional specification for the EC2 integration and fidelity work planned for the Bonita release of CloudStack.
...
...
...
Ideally the following should be accomplished1.
...
...
...
...
...
...
Deployment - Solution to be installable via a self contained virtual system server within a CloudStack management server.
...
Fidelity to the Amazon S3 API embraces:
To allow the above to be configurable to the resources, especially simple storage resources, at a given cloud management installation.
...
No regions – The Amazon AWS S3 provision for geographic regions, plus additionally a default ‘US Standard’ pan-regional option, will not be present in this design. Consequently when a location constraint is processed it will be ignored and, if created, will be empty by default.
...
The list of supported operations will not be fully coextensive with those at s3.amazonaws.com at this release. A list of Amazon S3 operations which are not supported within the current release are as follows.
...
The configuration environment is controlled by a file which needs to be accurately defined at the time of installation. Within the cloud bridge installation directory, the file is at conf/cloud-bridge.properties. Typical configuration information defined in this file is
host=http://myhost:8080/awsapi
storage.root=/mounts/mymountpoint
storage.multipartDir=_multipartuploads_
bucket.dns=false
serviceEndpoint=myhost:8080
So configured, the S3 API REST translation service will be running at http://myhost:8080/awsapi/rest/AmazonS3/.
The following step, with tomcat running, is to set up user keys using the script awsapi-setup/setup/cloudstack-aws-api-register. This needs setting up in accordance with the following example
./cloudstack-aws-api-register -u http://localhost:8080/awsapi/rest/AmazonS3 -a MyAccessIDKey -s MySecretKey openssl_generated.mycert.pem
...
The capabilities of the S3 API are intended to satisfy the following use cases. For an overview of the expected capabilities, see docs.amazonwebservices.com/AmazonS3/latest/API/APIRest.htm. *
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
It is not envisaged that any other CloudStack software need access these tables.
Code Block |
---|
Tables |
| Tables | +------------------- |
...
acl
+ | acl | | bucket_policies |
...
meta
...
mhost
| | meta | | mhost | | mhost_mount |
...
| | multipart_meta |
...
| | multipart_parts |
...
| | multipart_uploads |
...
| | offering_bundle |
...
sbucket
...
shost
...
sobject
...
sobject_item
...
|
| sbucket |
| shost |
| sobject |
| sobject_item |
| usercredentials |
|
The tables used are: acl, bucket_policies, meta, mhost, mhost_mount, multipart_meta, multipart_parts, multipart_uploads, offering_bundle, sbucket , shost, sobject, sobject_item, usercredentials
...
The design imposes a service lifecycle in which#The
...
...
...
...
To validate the request data structure (termed the canonical string) the following rules are enforced by the design:#List
...
...
...
In processing the URI, three formats can be distinguished:#uri
...
...
In the current implementation of the solution we are concentrating on the first two of these.
...
A hibernate layer coordinates between the business logic executed in the rest servlet and the SQL definitions in the MySQL database. See also Appendix 2.
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Code Block |
---|
calling_format= OrdinaryCallingFormat() |
...
connection = S3Connection(aws_access_key_id=<your api key>, |
...
aws_secret_access_key=<your secret key> |
...
is_secure=False, |
...
host='<cloudstack-server>', |
...
port=7080, |
...
calling_format=calling_format, |
...
path="/awsapi/rest/AmazonS3") |
CLOUDBRIDGE data definitions
Code Block |
---|
acl |
...
+--------------------+--------------+------+----- |
...
+---------+ |
...
Field | Type | Null | Key | Default | Extra |
---------------- |
...
+ | Field | Type | Null | Key | Default | Extra | +--------------------+--------------+------+-----+---------+-------- |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
Target | varchar(64) | NO | MUL | NULL |
|
TargetID | bigint(20) | NO |
| NULL |
|
GranteeType | int(11) | NO |
| 0 |
|
GranteeCanonicalID | varchar(150) | YES |
| NULL |
|
Permission | int(11) | NO |
| 0 |
|
GrantOrder | int(11) | NO |
| 0 |
|
CreateTime | datetime | YES |
| NULL |
|
LastModifiedTime | datetime | YES | MUL | NULL |
|
--------+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | Target | varchar(64) | NO | MUL | NULL | | | TargetID | bigint(20) | NO | | NULL | | | GranteeType | int(11) | NO | | 0 | | | GranteeCanonicalID | varchar(150) | YES | | NULL | | | Permission | int(11) | NO | | 0 | | | GrantOrder | int(11) | NO | | 0 | | | CreateTime | datetime | YES | | NULL | | | LastModifiedTime | datetime | YES | MUL | NULL | | +--------------------+--------------+------+-----+--------- |
...
+ |
...
----------------+ bucket_policies +------------------+---------------- |
...
+------+ |
...
Field | Type | Null | Key | Default | Extra |
-----+--------- |
...
+---------------- |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
BucketName | varchar(64) | NO | UNI | NULL |
|
OwnerCanonicalID | varchar(150) | NO |
| NULL |
|
Policy | varchar(20000) | NO |
| NULL |
|
+ | Field | Type | Null | Key | Default | Extra | +------------------+----------------+------+-----+--------- |
...
+ |
...
---------------- |
...
Field | Type | Null | Key | Default | Extra |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
Target | varchar(64) | NO | MUL | NULL |
|
TargetID | bigint(20) | NO |
| NULL |
|
Name | varchar(64) | NO |
| NULL |
|
Value | varchar(256) | YES |
| NULL |
|
+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | BucketName | varchar(64) | NO | UNI | NULL | | | OwnerCanonicalID | varchar(150) | NO | | NULL | | | Policy | varchar(20000) | NO | | NULL | | +------------------+----------------+------+-----+----- |
...
----+----------------+ meta +----------+--------------+------+-----+---- |
...
Field | Type | Null | Key | Default | Extra |
-----+----------------+ | Field | Type | Null | Key | Default | Extra | +----------+--------------+------+-----+---------+---------- |
...
ID
------+ | ID | bigint(20) |
...
NO
...
PRI
...
NULL
...
auto_increment
...
MHostKey
...
varchar(128)
...
NO
...
MUL
...
NULL
...
...
Host
...
varchar(128)
...
YES
...
UNI
...
NULL
...
...
Version
...
varchar(64)
...
YES
...
...
NULL
...
...
LastHeartbeatTime
...
datetime
...
YES
...
MUL
...
NULL
...
| NO | PRI | NULL | auto_increment | | Target | varchar(64) | NO | MUL | NULL | | | TargetID | bigint(20) | NO | | NULL | | | Name | varchar(64) | NO | | NULL | | | Value | varchar(256) | YES | | NULL | | +----------+--------------+------+-----+---------+----------------+ |
...
mhost +-------------------+--------------+------+----- |
...
+---------+ |
...
Field | Type | Null | Key | Default | Extra |
---------------- |
...
+ | Field | Type | Null | Key | Default | Extra | +-------------------+--------------+------+-----+---------+----------------+ |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
MHostID | bigint(20) | NO | MUL | NULL |
|
SHostID | bigint(20) | NO | MUL | NULL |
|
MountPath | varchar(256) | YES |
| NULL |
|
LastMountTime | datetime | YES | MUL | NULL |
|
...
| ID | bigint(20) | NO | PRI | NULL | auto_increment | | MHostKey | varchar(128) | NO | MUL | NULL | | | Host | varchar(128) | YES | UNI | NULL | | | Version | varchar(64) | YES | | NULL | | | LastHeartbeatTime | datetime | YES | MUL | NULL | | +-------------------+--------------+------+-----+----- |
...
----+----------------+ mhost_mount +---------------+--------------+------+-----+ |
...
Field | Type | Null | Key | Default | Extra |
---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+ |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
UploadID | bigint(20) | NO | MUL | NULL |
|
Name | varchar(64) | NO |
| NULL |
|
Value | varchar(256) | YES |
| NULL |
|
------+-----+--------- |
...
+----------------+ |
...
Field | Type | Null | Key | Default | Extra |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
UploadID | bigint(20) | NO | MUL | NULL |
|
partNumber | int(11) | NO |
| NULL |
|
MD5 | varchar(128) | YES |
| NULL |
|
StoredPath | varchar(256) | YES |
| NULL |
|
StoredSize | bigint(20) | NO |
| 0 |
|
CreateTime | datetime | YES |
| NULL |
|
...
| ID | bigint(20) | NO | PRI | NULL | auto_increment | | MHostID | bigint(20) | NO | MUL | NULL | | | SHostID | bigint(20) | NO | MUL | NULL | | | MountPath | varchar(256) | YES | | NULL | | | LastMountTime | datetime | YES | MUL | NULL | | +---------------+--------------+------+-----+---------+ |
...
----------------+ multipart_meta +----------+--------------+------+----- |
...
Field | Type | Null | Key | Default | Extra |
+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +----------+--------------+---- |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
AccessKey | varchar(150) | NO |
| NULL |
|
BucketName | varchar(64) | NO |
| NULL |
|
NameKey | varchar(255) | NO |
| NULL |
|
x_amz_acl | varchar(64) | YES |
| NULL |
|
CreateTime | datetime | YES |
| NULL |
|
...
--+-----+---------+---------------- |
...
Field | Type | Null | Key | Default | Extra |
+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | UploadID | bigint(20) | NO | MUL | NULL | | | Name | varchar(64) | NO | | NULL | | | Value | varchar(256) | YES | | NULL | | +----------+--------------+------+-----+---------+---------------- |
...
ID | int(11) | NO | PRI | NULL | auto_increment |
AmazonEC2Offering | varchar(100) | NO | UNI | NULL |
|
CloudStackOffering | varchar(20) | NO |
| NULL |
|
+ multipart_parts +------------+--------------+------+-----+---------+---------------- |
...
+ | Field | Type | Null | Key | Default | Extra | +------------+--------------+------+-----+--------- |
...
+ |
...
Field | Type | Null | Key | Default | Extra |
...
----------------+ |
...
ID
| ID | bigint(20) |
...
NO
...
PRI
...
NULL
...
auto_increment
...
Name
...
varchar(64)
...
NO
...
UNI
...
NULL
...
...
OwnerCanonicalID
...
varchar(150)
...
NO
...
MUL
...
NULL
...
...
SHostID
...
bigint(20)
...
YES
...
MUL
...
NULL
...
...
CreateTime
...
datetime
...
YES
...
MUL
...
NULL
...
...
VersioningStatus
...
int(11)
...
NO
...
...
0
...
...
| NO | PRI | NULL | auto_increment | | UploadID | bigint(20) | NO | MUL | NULL | | | partNumber | int(11) | NO | | NULL | | | MD5 | varchar(128) | YES | | NULL | | | StoredPath | varchar(256) | YES | | NULL | | | StoredSize | bigint(20) | NO | | 0 | | | CreateTime | datetime | YES | | NULL | | +------------+--------------+------ |
...
+-----+ |
...
---------+----------------+ multipart_uploads +------------ |
...
+-------------- |
...
+ |
...
Field | Type | Null | Key | Default | Extra |
------+-----+---------+---------------- |
...
+ | Field | Type | Null | Key | Default | Extra | +------------+--------------+------+-----+---------+------------ |
...
ID
----+ | ID | bigint(20) |
...
NO
...
PRI
...
NULL
...
auto_increment
...
Host
...
varchar(128)
...
NO
...
MUL
...
NULL
...
...
HostType
...
int(11)
...
NO
...
...
0
...
...
ExportRoot
...
varchar(128)
...
NO
...
...
NULL
...
...
MHostID
...
bigint(20)
...
YES
...
MUL
...
NULL
...
...
UserOnHost
...
varchar(64)
...
YES
...
...
NULL
...
...
UserPasssword
...
varchar(128)
...
YES
...
...
NULL
...
...
UserPassword
...
varchar(255)
...
YES
...
...
NULL
...
| NO | PRI | NULL | auto_increment | | AccessKey | varchar(150) | NO | | NULL | | | BucketName | varchar(64) | NO | | NULL | | | NameKey | varchar(255) | NO | | NULL | | | x_amz_acl | varchar(64) | YES | | NULL | | | CreateTime | datetime | YES | | NULL | | +------------+--------------+------+----- |
...
+--------- |
...
+ |
...
----------------+ offering_bundle +--------------------+--------------+------ |
...
+ |
...
Field | Type | Null | Key | Default | Extra |
-----+---------+---------------- |
...
+ | Field | Type | Null | Key | Default | Extra | +--------------------+--------------+--- |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
SBucketID | bigint(20) | NO | MUL | NULL |
|
NameKey | varchar(255) | NO |
| NULL |
|
OwnerCanonicalID | varchar(150) | NO | MUL | NULL |
|
NextSequence | int(11) | NO |
| 1 |
|
DeletionMark | varchar(150) | YES |
| NULL |
|
CreateTime | datetime | YES | MUL | NULL |
|
---+-----+---------+---------------- |
...
+ | ID | int(11) | NO | PRI | NULL | auto_increment | | AmazonEC2Offering | varchar(100) | NO | UNI | NULL | | | CloudStackOffering | varchar(20) | NO | | NULL | | +--------------------+--------------+------ |
...
+ |
...
Field | Type | Null | Key | Default | Extra |
-----+---------+---------------- |
...
+ sbucket +------------------+--------------+------+-----+---------+--------- |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
SObjectID | bigint(20) | NO | MUL | NULL |
|
Version | varchar(64) | YES |
| NULL |
|
MD5 | varchar(128) | YES |
| NULL |
|
StoredPath | varchar(256) | YES |
| NULL |
|
StoredSize | bigint(20) | NO | MUL | 0 |
|
CreateTime | datetime | YES | MUL | NULL |
|
LastModifiedTime | datetime | YES | MUL | NULL |
|
LastAccessTime | datetime | YES | MUL | NULL |
|
-------+ | Field | Type | Null | Key | Default | Extra | +------------------+--------------+------+----- |
...
+--------- |
...
+ |
...
---------------- |
...
Field | Type | Null | Key | Default | Extra |
...
+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | Name | varchar(64) | NO | UNI | NULL | | | OwnerCanonicalID | varchar(150) | NO | MUL | NULL | | | SHostID | bigint(20) | YES | MUL | NULL | | | CreateTime | datetime | YES | MUL | NULL | | | VersioningStatus | int(11) | NO | | 0 | | +------------------+--------------+------+-----+---------+----------------+ shost +---------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-----+---------+----------------+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | Host | varchar(128) | NO | MUL | NULL | | | HostType | int(11) | NO | | 0 | | | ExportRoot | varchar(128) | NO | | NULL | | | MHostID | bigint(20) | YES | MUL | NULL | | | UserOnHost | varchar(64) | YES | | NULL | | | UserPasssword | varchar(128) | YES | | NULL | | | UserPassword | varchar(255) | YES | | NULL | | +---------------+--------------+------+-----+---------+----------------+ sobject +------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------------+--------------+------+-----+---------+----------------+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | SBucketID | bigint(20) | NO | MUL | NULL | | | NameKey | varchar(255) | NO | | NULL | | | OwnerCanonicalID | varchar(150) | NO | MUL | NULL | | | NextSequence | int(11) | NO | | 1 | | | DeletionMark | varchar(150) | YES | | NULL | | | CreateTime | datetime | YES | MUL | NULL | | +------------------+--------------+------+-----+---------+----------------+ sobject_item +------------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------------+--------------+------+-----+---------+----------------+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | SObjectID | bigint(20) | NO | MUL | NULL | | | Version | varchar(64) | YES | | NULL | | | MD5 | varchar(128) | YES | | NULL | | | StoredPath | varchar(256) | YES | | NULL | | | StoredSize | bigint(20) | NO | MUL | 0 | | | CreateTime | datetime | YES | MUL | NULL | | | LastModifiedTime | datetime | YES | MUL | NULL | | | LastAccessTime | datetime | YES | MUL | NULL | | +------------------+--------------+------+-----+---------+----------------+ usercredentials +--------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+--------------+------+-----+---------+----------------+ | ID | bigint(20) | NO | PRI | NULL | auto_increment | | AccessKey | varchar(150) | NO | UNI | NULL | | | SecretKey | varchar(150) | NO | | NULL | | | CertUniqueId | varchar(200) | YES | UNI | NULL | | +--------------+-------------- |
...
+ |
...
ID | bigint(20) | NO | PRI | NULL | auto_increment |
AccessKey | varchar(150) | NO | UNI | NULL |
|
SecretKey | varchar(150) | NO |
| NULL |
|
CertUniqueId | varchar(200) | YES | UNI | NULL |
|
------+-----+--------- |
...
+---------------- |
...
+
|
OR mapping definitions