Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

...

For Agent Load balancing functionality – please refer below spec

http://intranet.lab.vmops.com/engineering/release-2.2-features/agent-load-balancing?searchterm=agentImage Removed

 

Clustered MS configuration:
MS1: cluster DB
MS2   MS3 MS4  in cluster
Primary storage: NFS and ISCSI
Load balancer: netscaler
For agent_load-balancing cases
Minimum requirement  is at least 2 management servers and recommended is 2+
3 management servers - > zone> 1 pod -> 3 clusters each has at least 1 hosts. Have Vms running on all hosts
Set up : Avg time to setup cluster management setup is 4 to 6 hrs
The below testcases to be repeated for all the supported hyper-visors.
Also repeated for basic and advanced zone.

SNO

Testcase Name

Procedure

Expected  Results

Test Case Type ( Sanity | Functional |Negative)

is Automatable

Execution Time

Priority

Executed on Release

Reviewed By

Status

Comments

result Is automatable

1

Verify_Host_transition_states

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host        
2. MS1 start. check host transition state, MSs ownership of host        
3. MS2 start. check host transition state, MS ownership of host          
4. MS3 start. check host transition state, MS ownership of host         
5. MS1 stop. check Host transition state, MS ownership of host         

1. Check Host state change Alert-disconnect-Connecting-UP
2. MS1 owns 3 clusters of hosts     3. MS1 owns 2 clusters. MS2 own 1 cluster.              4. MS1, MS2, MS3 each own 1  cluster 5. MS1 cluster owned by MS2 or  MS3          

SANITY, FUNCTIONAL

N

 

P1

 

 

 

 

 

2

Only_One MS_UP

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host                 
2. MS1 start. check MSs ownership of host            
3. MS2 start. check MS ownership of host            
4. MS3 start. check MS ownership of host            
5. MS1 stop. check MS ownership of host            
6.. MS2 stop. check MS ownership of host 
7.Deploy  new VMs and perform  VM life-cycle operations
(vm stop/start/destroy/restore)
8.snapshot on the volumes
9.register new templates

7. Able to access the MS UI from MS3 and all system vms up running.
All hosts stats UP. deploy VM successful
8. snapshot successful
9. able to register the new templates successfully

SANITY, FUNCTIONAL

P1 Y

 

P2

 

 

 

 

 

3

All_MGMTServices

3

All_MGMTServices_UP_at_sameT

  1. MS1 MS2 MS3 each manage 1 cluster
    2. MS1 MS2 MS3 service cloud-management stop
    3. MS1 MS2 MS3 service cloud-management start
    4.Deploy  new VMs and perform snapshot on the volumes

3. Host clusters re-distributed to all 3 management servers.
4. deployVM ,
snapshot operation should be successful

FUNCTIONAL

Y

 

P2

 

 

 

  P1

 

4

restart_ALL_MGMT services's

1.MS1 MS2 MS3 each manage 1 cluster
2. MS1 MS2 MS3 service cloud-management  restart

All the MGMT server should up with out any erros and all hosts get distributed to corresponding MS's

FUNCTIONAL/Negative

 

 

P2

 

 

 

negative

P1  

 

5

Change_SystemDate_ON_1MGMT Server

1.Make sure each host is managed by each MGMT Server and
 and system date is sync on all the hosts and MS
2.Change the system date in one of MS(MS1) in cluster MS
3. Check the MS state in mshost table
4. Check the MS ownership on host tables
5. Synch the date on MS1(i.e change the sysdate again to synch with
 with other MGMT server)

When one of the MS date is not synched  management state in mshost should be down
 ownership of  Host belongs to above MS moved to another available MS . MS  state should be UP  and  Host ownership will  changed back to current only when agent load is in-balance.

FUNCTIONAL/ Negative

 

 

P3

  P2

 

 

 

 

6

Perform_task_From_non-managed_MS

1.host 1  is managed by MS1
2.check the  forward functionality by initializing the task from
non-managed MS (i.e deployVM  on host1 from MS2)

1. task should be forward to managed MS(MS1) from MS2 and executed at MS1 and response should be received back to task initiator(i.e MS2)

SANITY, FUNCTIONAL

Y

 

P1

 

 

 

 

 

7

Perform_Task_From_non-Managed MS_StopMS

1.host 1  is managed by MS1
2.perform add Instance on host1 from MS2
3.stop the MS2 before it receive the response from MS1

Async Job shouldn't be in progress and it should be Done state.

FUNCTIONAL/ Negative

 

 

P2

 

 

 

 

 

8

verify_ForwardAgent_becomes DA

1.Stop MS1
2. cluster HOST1 owned by MS1   gets transferred to MS2
3. From MS2 , Make a request to add Vm to the host HOST1
 ( use host tags to direct vm to a specific host)
4.Start/stop Vm instances that are in HOST1.
Take snapshots of the Vms that are in HOST1.

From MS2,Direct Agent Request  transferred to MS2 from MS1 and  Vm deployment should successful on MS2
all the operations should be successful with out any errors

SANITY, FUNCTIONAL

Y(Its possible to automate but it s difficult judge which MS server takes the ownership if we have more than 2 MS ..always manual intervention required to get the accurate results for this cases)

 

P1

 

 

 

the operations should be successful with out any errors

SANITY, FUNCTIONAL

P1  

 

9

verify ForwardAgent_functionality

1.Stop the MS1
2. cluster HOST1 owned by MS1   gets transferred to MS2
3. From MS3 , Make a request to add Vm to the host HOST1
( use host tags to direct vm to a specific host)
4.Start/stop Vm instances that are in HOST1. Take
 Take snapshots of the Vms that are in HOST1.

3.Request  get forwarded to MS2 from MS3  and  Vm deployment should successful on MS2.
4.all the operations should be successful with out any errors

SANITY, FUNCTIONAL

Y

 

P1

 

 

 

 

 

10

Host_Force_reconnect_From_non-managed MS

Find the management server that owns this host cluster. From
 From any of the other MS ( that is NOT the
owner of this host) , initiate a Force Re-connect of the host.

We should see the request being forwarded  from non-managed MS to o the owner management server and the action should succeed.

FUNCTIONAL

 

 

P2

 

 

 

 

 

 

 

 

 

.

FUNCTIONAL

P1  

 

 

 

 

 

 

 

 

11

Host-maintenance_Mode_Mgmg_status

1. On MS1 Put Host into maintenance mode
2. Check the host MGMT_ID value

1.Host should be maintenance mode with out any erros
2.MGMT_ID  should be Null

SANITY, FUNCTIONAL

 

 

P2

 

 

 

  P1

 

12

Cancel_MaintenanceMode

1.Select the Host from any MGMT server(say MS2)
2.Put the Host into maintenance mode
3. Check the host MGMT_ID value
4.cancel the maintenance mode 
5.Check the MGMT ID of the above host

3.MGMT_ID  should be Null
4.It should get either  same MGMT ID or mgmt_server_id or another mgmt_server id

SANITY, FUNCTIONAL

 

 

P1

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

13

Take the Host out of maintenance Mode

Find the management server that owns this host. From any of the
 the other MS ( that is NOT the owner of this host) ,
put the host in maintenance mode and take the host out of maintenance mode.

We should see the requests for both the above actions being forwarded to the owner management server and the actions should succeed.

SANITY, FUNCTIONAL

 

 

P1

 

 

 

SANITY, FUNCTIONAL

P1  

 

14

HostMaintenance_Join_MS

1.MS1 Put its Host into maintenance mode
2. Join another MS to cluster
3.check the agent load balancing will happen or not

when Host is in maintenance mode ,agent load balancing won't happen.
No errors observed in the logs

FUNCTIONAL

Y

 

P2

 

 

 

  P1

 

15

Cancel_Maintenance_Join_MS

1.Select the Host from any MS (Say MS1)  and make sure the
 the current cluster has more than Threshold value
2.Put the Host into maintenance mode
3. cancel the maintenance mode
4.Join another Management server to cluster(say MS4) or restart
 restart the existing MS(MS2)
5.check the agent load balancing will happen or not

2..MGMT_ID  should be Null
5.Either old and new MS ID will be assigned

SANITY, FUNCTIONAL

Y

 

P2

 

 

 

  P1

 

16

StopMS1_Cancel_Maintenance Mode_fromMS2

1.Select the Host from any MGMT server(eg:MS1)
2.Put the Host into maintenance mode
3. Check the host MGMT_ID value
4.stop the MGMT server service on MS
5.cancel the maintenance mode  from MS2
6.Check the MGMT ID of the above host

3.MGMT_ID  should be Null
6)It should successfully came out of Maintenance mode and get the suitable  MS ID

SANITY, FUNCTIONAL

Y

 

P2

 

 

 

 

 

17

AsyncJobState_StopMS_while_Job(Maintenance)_inProgress

1.Select the Host owned by  MGMT server(eg:MGMT 1) from MGMT Server MS 2
2.Put the Host into maintenance mode from non-managed management server MS (say M2)
3. when host is in prepare maintenance mode,stop the Management server MS service on Management
 Management service 1(MS1)
5.Check the asyn job status

Async job should  not be in-progress state and its Job state should be done with FAILED.

SANITY, FUNCTIONAL

Y

 

P2

 

 

 

 

 

18

DeleteHost_from_other_MS

Find the management server that owns this host. From any of the other management servers ( that is NOT the
owner of this host) , put the host in maintenance mode and then delete the host.

We should see the requests for both the above actions being forwarded to the owner management server and the
actions should succeed.

FUNCTIONAL

Y

 

P1

 

 

 

 

 

19

AsyncJobState_StopMS_while_Job(Snapshots)_inProgress

From the management server which is NOT the owner of the host in which the Vm is running , initiate a snapshot task.

When the task is still in progress , stop the management server.

Async job should  get marked as done (FAILED) but it should not be in progress.

FUNCTIONAL

Y

 

P2

 

 

 

 

 

20

AgentLB_HostState_UP_cluster_Disabled

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.
2. MS1 put one of the cluster in disable state.
3.Join another MS3

MS1 gives away 2nd cluster & Agent loadbalacing happens when hosts state is UP.
host will get the new MGMT server ID

FUNCTIONAL

 

 

P1

 

 

 

 

 

21

AgentLB_HostState_Down_cluster_Disabled

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.2. MS1 unmanage 1 cluster >disable state and make sure state of the  hosts are in disconnected state
3.Join MS3 to cluster

No agent load balancing will trigger

FUNCTIONAL

 

 

P2

 

 

 

FUNCTIONAL

P2  

 

22

AgentLB_ThresHold_lessThan_default value

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.2.agent.load.threshhold =0.45 less than  default thresh hold value(0.7)   
2.Join MS3 to cluster

No agent load balancing will trigger

SANITY, FUNCTIONAL

 

 

P1

 

 

 

 

 

23

AgentLB_ThresHold_moreThan_default value

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.
2.agent.load.threshhold =0.8 more than  default thresh hold value(0.7)  
3.Join MS3 to cluster

Agent load balancing will happen and host(selected host which are in giveway list) will get the new MGMT server ID

SANITY, FUNCTIONAL

 

 

P1

 

 

 

 

 

24

AgentLB_ThresHold_moreThan_Non_default_ value

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.2. change the  threshold is more than  default thresh hold value(0.2)
 i.e Global configuration change the value of agent.load.threshhold  parameter
2.Join MS3 to cluster

Agent load balancing will happen and host(selected host which are in giveway list) will get the new MGMT server ID

FUNCTIONAL

 

 

P2

 

 

 

 

 

25

Direct agent-loads

  1. Configure the direct.agent.load.size =2
  2. make sure 3 hosts should be consider for rebalacning
  3. check the number of hosts consider for rebalancing at same time

As per the configured  load size(2),At same time 2 hosts should be consider(processed) for re-balancing.

FUNCTIONAL

Y

 

 

 

 

 

 

 

26

AgentLB_sourceMS_down_during_Re-balancing

1. zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.
2.Join MS3  to cluster
3.during agent load balancing,stop the source MGMT server(MS1)

All hosts already transferred to MS2 should get new MGMT server ID.

FUNCTIONAL

 

 

P2

 

 

 

 

 

27

AgentLB_DestinationMS_Down_during_Rebalancing

1.  zone> pod1> cluster1  cluster2  cluster3 each with at least 1 host. all hosts  state  are in UP  "agent.lb.enabled" true. MS1 manage 2 clusters. MS2 manage 1 cluster.
2.Join MS2 to cluster
3.during agent load balancing(from MS1 to MS2),stop the destination MGMT server  MS2

If M2 dies, all re balancing should be stopped as it's running on M2. All hosts already transfered to M2 should be marked as Disconnected.

FUNCTIONAL

 

 

P2

 

 

 

 

 

27

dbServer_ UsageServer on different machines

1. configure cluster management setup and Make sure database and usage on separate machine
2.Deploy VM's on Host2 and generate usgae traffic from one management server  MGMT 2 on the GuestVMs.(download some  files from internet into guestVMs)
3.check the network usage statistics
4.stop the current Mangement server i.e MGMT 2
5.check the network statistics
6.From another management server ,generate public traffic and check the network usage statistics

In all the cases usage statistics should be collected and details should be accurate

SANITY, FUNCTIONAL

 

6 Hrs

P1

 

 

 

 

 

29

dbServer_ UsageServer on _Same_Machines

1. configure cluster management setup and Make sure database and usage on the machine
2.Deploy VM's on Host2 and generate usgae traffic from one management server  MGMT 2 on the GuestVMs.(download some  files from internet into guestVMs)
3.check the network usage statistics
4.stop the current Management server i.e MGMT 2
5.check the network statistics
6.From another management server ,generate public traffic and check the network usage statistics

In all the cases usage statistics should be collected and details should be accurate

SANITY, FUNCTIONAL

 

3 hrs

P1

 

 

 

 

 

30

Ping Timeout

1.make sure all the MS's are synch
2.On one management remove or comment the IPtable rule for 9090 port(to make sure MS communication failed with ping timed out but still other MS's can communicate to common mysql DB)

once ping command timeouts,The corresponding MS will be disconnect state and host on the current MS(M2) will get the another management ID(MS3 ID)

Negative

 

 

 

 

 

 

 

 

31

Upgrade test(2.2.14 to Bonita and 3.0 to Bonita)

1.prepare the cluster management setup before upgrade
2. perform basic operations(vm life-cycle,snapshots)
3.make sure before host ownership changed from MS1 to MS2
4.Upgrade the all the Management Servers in the cluster management-setup
5.check after upgrade able to perform basic sanity test

Upgrade should be successful,able to perform basic sanity cases(Agent load balancing &Direct agent and forward agent functionality)

SANITY, FUNCTIONAL

 

12 hrs  for each upgrade

P1

 

 

 

SANITY, FUNCTIONAL

P1  

 

Q&A:

1) When the cluster is disabled(Hosts are in UP state) ,does the agent load balancing will happen?

...

4) How to restrict the host to same management ID

Wiki Markup      A) Current design its not possible to  restrict the  host to same management ID \[Not advisable because if the management server dies no re-balancing happen \possible to  restrict the  host to same management ID [Not advisable because if the management server dies no re-balancing happen ]

5)If host is rebalanced and move from M1->M2->M3 and when we stop MS service on M3  ..how it will determine future_owner_id(assuming both M1

...