Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-2163

Branch

master, 4.2.0

Introduction

In order for the admin to have better control over the first class objects in CS, admins needs to have the ability to add meta information to the objects and be able to expose/hide them from the end user. In addition admin should also be able to deploy the first class resources and have the capability to expose/hide them from the end user. These changes allow a service with higher functionality to manipulate objects managed by CS.  For this I propose following changes to be introduced.

Use Cases

One of the use cases can be 

  • Say for every resource created in a zone the admin wants to have a copy of the resource in another zone for high availability and recovering from disaster. For this the admin can deploy a third party service on top of CS which subscribes to the resource creation events and in turn takes the action of creating backup resources on a backup zone. With the mechanism proposed in this FS admin can deploy these copy resources hidden from the end user and make them available to the end user in case of some disaster. Using the mechanism proposed in the FS, the third party service can also store meta data in the resources about their backup resource ids, zone information etc. .  diag1

Data Schema Changes

Schema changes can be divided into two categories

  • Resource tables - Add flag  to expose or hide them from the end user
  • Resource detail tables - Ability to add meta information for the resources and be able to expose or hide them from the end user.  

vm_instance 

  • Add a flag to determine whether to display the VMInstance to end user.

user_vm_details

  • Add a flag to determine whether to display the detail to the end user.

user_vm_view

  • Add the display vm flag in the view.

volumes

  • Add a flag to determine whether to display the detail to the end user.

volume_view

  • Add the display volume flag in the view.

volume_details

  • Add a table to contain details on the volume.  The table contains the following schema
    • display - whether to display this detail to the end user
    • name - name of the volume detail
    • value - value of the volume detail

networks

  • Add a flag to control whether this network is displayable to the end user.

network_details

  • Add a table to contain details on the network.  The table contains the following schema
    • display - whether to display this detail to the end user
    • name - name of the network detail
    • value - value of the network detail

nics

  • Add a flag to control whehter this nic is displayable to the end user.

nic_details

  • Add a table to contain service details  on the nic.  The table contains the following schema
    • display - whether to display this detail to the end user
    • name - name of the nic detail
    • value - value of the nic detail

disk_offering_details

  • Add a table to contain disk offering details.  The table contains the following schema
    • name - name of the disk offering details
    • value - value of the disk offering details

disk_offering

  • Add a flag to control whether this disk offering is displayable to the end user

disk_offering view

  • Add a flag display offering to the view.

API Changes

CS APIs must be changed to allow for the above schema to be specified at the creation and update of the entities specified.

 

Regular user/domain admin can never request disabled resources via API.

 

Admin can do it by providing forDisplay=false flag in corresponding API request.

 

It works the following way:

 

  • no display is passed to the call. Return only objects set with display=true
  • display=false is passed to the call. Return only objects with display=false
  • display=true is passed to the call. Return only objects with display=true


API Modifications

DeployVM/UpdateVM/ - Add flag whether to display the VM to end user

CreateNetwork/UpdateNetwork - Add flag whether to display the network to end user

CreateDiskOffering - Add flag whether to display the disk offering to end user

CreateVolume - Add flag whether to display volume to the end user

New APIs

Add new api's to add meta information for the resources.

UpdateVolume - Exposes the path component for other services to update

AddResourceDetail - Add details to resource

  • resourceIds - List of resource ids
  • resourceType - Example - volume, nic etc.
  • details - Map of key/value pairs

RemoveResourceDetail - Remove detail to resource

  • resourceIds - List of resource ids
  • resourceType - Example - volume, nic etc.
  • details - Map of key/value pairs

ListResourceDetails - lists details for a resource

  • resourceId 
  • resourceType - Example - volume, nic etc.
  • key
  • value

Upgrade

For existing "resource" and "resource detail" tables the default value for the flag will be "true" so that they are displayed to the end user. This along with the new tables will be added to schema-410to420.sql

Resource Count Changes

Since the display flag will be used by admin to control first class objects and hide them from the end user, following changes are made for calculating resource count

  • Resource count shouldn't be incremented when creating a resource with display flag and decremented when the display flag is turned off.
  • Similarly when the flag is turned on we should increment the resource count.
  • Calculating resource count for an account/domain should respect the flag and count it only when its turned on for the resource.

     Following interface functions were introduced in ResourceLimitService.java so that any resource using this flag can use this interface.

ResourceLimitService.java
    /**
     * Checks if a limit has been exceeded for an account if displayResource flag is on
     *
     * @param account
     * @param type
     * @param displayResource
     * @param count
     * @throws ResourceAllocationException
     */
    void checkResourceLimit(Account account, ResourceType type, Boolean displayResource, long... count) throws ResourceAllocationException;

    /**
     * Increments the resource count  if displayResource flag is on
     *
     * @param accountId
     * @param type
     * @param displayResource
     * @param delta
     */
    void incrementResourceCount(long accountId, ResourceType type, Boolean displayResource, Long... delta);

    /**
     * Increments/Decrements the resource count  depending on the displayResource flag is turned on or off respectively
     *
     * @param accountId
     * @param type
     * @param displayResource
     * @param delta
     */
    void changeResourceCount(long accountId, ResourceType type, Boolean displayResource, Long... delta);

    /**
     * Decrements the resource count  if displayResource flag is on
     *
     * @param accountId
     * @param type
     * @param displayResource
     * @param delta
     */
    void decrementResourceCount(long accountId, ResourceType type, Boolean displayResource, Long... delta);

 

Action Events

If the resources have display flag turned off we should not show the action events corresponding to actions happening on them.

Off course this should be done for end users only. For root admins all the events should be displayed. Technically following changes will be introduced.

  1. I will introduce a "display" flag in the event schema. Listevents will only display those events which have this flag=1. For ROOT admins all events will be listed.  

  2. Action events are put in at generic place at the entry of the commands and at their manager method interceptions. I have put in a generic function isDisplay() in BaseCmd.java (inherited by all APIs) and introduced a Displayable interface. So at the entry point of the commands this function will be invoked for each api to find out whether the corresponding action event needs to be logged with display=0. The method isDisplay() will check if the resource entity has the Displayable attribute and if it does then it check the display flag.

  3. For method interceptors I will set the value of flag isDisplay in the CallContext so that while persisting action events these can be invoked.

Usage Changes

Since the display flag will be used by admin to control first class objects and hide them from the end user, we shouldnt generate usage for the resources while the flag is off. In short, all the usage events are registered only when the flag is on.

Basic rules :-

  • We generate creation event when
    • The flag=1 during creation,
    • The flag is updated to 1.
  • We generate the deletion event when
    • The flag=1 during deletion,
    • The flag is updated to 0.
  • Resource in destroyed state - no 'creation' / 'deletion' event is generated.
  • Currently done for Allocated Vm, running Vm, volumes. For public ip we should charge for ips reserved in the recovery zone as well.

UI Changes

TBD

Open questions

  • When the admin sets the display flag, will he be creating the duplicate resource where the ownership is with him or does it create on behalf of the user ? And if the ownership is with him then whats the use of the flag -?
  • Is this flag for displaying the resource only set by the admin ? I am guessing yes.
  • No labels

2 Comments

  1. Alena Prokharchyk

    The feature doesn't actually provide an API for creating the copy of the resource in the DB. Is it a correct statement to make?
    What feature does instead is:
    provides an ability to mark certain resources as unavailable for display. The resources have to be created using regular cloudStack api prior to this point. And you leave it up to the third party resource (it will create a copy upon reading the msg of original resource creation published to the event bus)
    Allows to set metadata for CS resources. Could you provide the samples of the metadata that customers would find useful?

    Please let me know if my understanding is correct. Also it would be very helpful to have a simple use case described in the FS consisting of all the steps third party deployer needs to do in order to use the feature, including the APIs he would use. Something like:

    the original resource is created. The event is published to the event bus
    Third party reads the resource, creates a copy of the record in the DB for zone id=2
    Third party calls the updateVm/updateVolume/updateNetwork API to update the display flag to be "false" for the resource copy

  2. Alena Prokharchyk

    Couple of questions on display flag implementation. I noticed that:

    • in different APIs its called differently (displayVm, displayNetwork, etc). Its better to add a generic name for this, say "displayFlag" so the third party customer can use more generic approach while building up his api calls.
    • listVolumes API misses the display Flag.
    • when I execute listVms call (tried as an admin), I see vms marked with display=false flag:

    mysql> select id, name, display_vm from vm_instance where id=4;
    --------------------------------------------------

    id

    name

    display_vm

    --------------------------------------------------

    4

    79aa2738-bf6d-43b3-ac8f-df961968d131

    0

    --------------------------------------------------
    1 row in set (0.00 sec)

    http://localhost:8096/?command=listVirtualMachines&id=79aa2738-bf6d-43b3-ac8f-df961968d131

    <listvirtualmachinesresponse cloud-stack-version="4.2.0-SNAPSHOT">
    <count>1</count>
    <virtualmachine>
    <id>79aa2738-bf6d-43b3-ac8f-df961968d131</id>
    <name>79aa2738-bf6d-43b3-ac8f-df961968d131</name>
    <account>admin</account>

    Sounds like a buggy behavior to me