...
Processor
Dual core Intel(R) Xeon(R) CPU processor, 2.27GHz, ht enabled, 4 processor
Operating System
CentOS release 5.5 (Final), x86_64
Java version
java version "1.6.0"
OpenJDK Runtime Environment (build 1.6.0-b09)
OpenJDK 64-Bit Server VM (build 1.6.0-b09, mixed mode)
Processor
Quad-Core AMD Opteron(tm) Processor, 2.1GHz, ht enabled, 8 processor
Operating System
CentOS release 6.2 (Final), x86_64
Configuration Parameters
DB configurations for this run is detailed in the my.cnf attached: my.cnf.txt
Mysql version
MySQL-server-5.5.21-1.linux2.6.x86_64
Set up a clustered Management Server (with 3 MS)
...
Deploy VMs in batches and in sequence and following measures should be tracked in the Management Server:
Following config parameters were used in both the management servers
- Java heap size = 5 GB
- db.cloud.maxActive = 250
- db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&prepStmtCacheSqlLimit=4096&includeInnodbStatusInDeadlockExceptions=true&logSlowQueries=true
CPU Utilization during deploy VM
CPU Utilization measured in the Management server during deployment of 12K simulator VMs and post deploy when there were no external APIs being fired fired
The area marked in red is during Deploy VM
Response Time of async job:
Following graph shows the response time calculated for deploy VM jobs. It’s more or less constant between 3 and 5 seconds except for few spikes
Test Scenario 5:
Restart Management Servers and track the time for host re balancing. Also track the numbers when few hosts are put into maintenance mode.
Following shows the time taken for each restart:
TC1: Stop both management servers. Then start up both. Time taken for all hosts to get to Up state: ~13 mins
TC2: Stop one management server. Time taken for hosts of that management server to get to Up state: ~5 mins
TC3: Start up the stopped management server. Time taken for hosts to rebalance between the 2 management server: ~16 mins
Now, add few hosts into maintenance mode and repeat the above restarts. Following are the time taken in each scenario:
TC1: ~10 mins
TC2: ~5 mins
TC3: ~13 mins
Configuration:
Advanced zone, 112 Pods \ [Each Pod having 2 Clusters\] Wiki Markup
224 Clusters \ [Each cluster having 8 hosts and one primary storage\] Wiki Markup
2000 Hosts
4000 User accounts \ [Each account having one network\] Wiki Markup
4000 User instancesunmigrated-wiki-markup
8000 Virtual Routers \ [Since we are using Redundant Virtual Router offering\]
API | Params | Time taken in 4.1 | Time taken in 4.2 | Time taken in 4.2 |
|
|
|
| (with fix) |
listAccounts | listAll=true&pagesize=100&page=3 | 0m16.923s | 0m20.017s | 0m16.680s |
(# of obj: 4000) | listAll=true&pagesize=800&page=2 | 0m26.555s | 0m34.438s | 0m36.118s |
| listAll=true&pagesize=1500&page=2 | 0m40.140s | 1m1.036s | 0m55.497s |
| listAll=true&pagesize=3000&page=1 | 1m3.906s | 2m2.921s | 1m24.601s |
| listAll=true | 1m25.348s | 2m22.628s | 1m44.621s |
|
|
|
|
|
listRouters | listAll=true&pagesize=100&page=1 | 0m2.398s | 0m3.808s | 0m3.261s |
(# of obj: 8000) | listAll=true&pagesize=800&page=1 | 0m10.372s | 0m23.124s | 0m22.164s |
| listAll=true&pagesize=1500&page=1 | 0m20.857s | 0m25.727s | 0m38.656s |
| listAll=true&pagesize=3000&page=1 | 0m35.034s | 1m39.030s | 1m22.330s |
| listAll=true&pagesize=5000&page=1 | 1m0.932s | 2m31.960s | 2m5.248s |
| listAll=true | 1m30.935s | 4m26.163s | 3m30.292s |
|
|
|
|
|
listEvents | listAll=true&pagesize=100&page=1 | 0m5.967s | 0m2.169s | 0m3.423s |
(# of obj: 8000) | listAll=true&pagesize=800&page=1 | 0m3.006s | 0m6.356s | 0m5.192s |
| listAll=true&pagesize=1500&page=1 | 0m2.111s | 0m10.402s | 0m7.622s |
| listAll=true&pagesize=3000&page=1 | 0m4.122s | 0m13.979s | 0m13.520s |
| listAll=true&pagesize=5000&page=1 | 0m6.575s | 0m20.277s | 0m21.931s |
| listAll=true | 0m15.399s | 2m16.445s | 1m54.454s |
|
|
|
|
|
listHosts | listAll=true&pagesize=100&page=1 | 0m1.906s | 0m1.030s | 0m1.159s |
(# of obj: 1985) | listAll=true&pagesize=800&page=1 | 0m13.316s | 0m7.949s | 0m7.091s |
| listAll=true&pagesize=1500&page=1 | 0m27.115s | 0m19.168s | 0m9.309s |
| listAll=true |
| 0m42.024s | 0m41.504s |
|
|
|
|
|
listUsers | listAll=true&pagesize=100&page=1 | 0m0.253s | 0m0.406s | 0m0.704s |
(# of obj: 4000) | listAll=true&pagesize=800&page=1 | 0m2.105s | 0m3.526s | 0m3.484s |
| listAll=true&pagesize=1500&page=1 | 0m2.485s | 0m12.411s | 0m6.792s |
| listAll=true&pagesize=3000&page=1 | 0m6.975s | 0m14.739s | 0m14.165s |
| listAll=true | 0m6.787s | 0m22.466s | 0m19.312s |
|
|
|
|
|
listVirtualMachines | listAll=true&pagesize=100&page=1 | 0m1.671s | 0m9.009s | 0m2.464s |
(# of obj: 8000) | listAll=true&pagesize=800&page=1 | 0m9.191s | 0m25.621s | 0m14.625s |
| listAll=true&pagesize=1500&page=1 | 0m13.068s | 0m20.737s | 0m26.461s |
| listAll=true&pagesize=3000&page=1 | 0m22.950s | 0m18.503s | 0m51.964s |
| listAll=true&pagesize=5000&page=1 | 0m35.443s | 0m18.503s |
|
| listAll=true |
| 1m16.001s |
|
|
|
|
|
|
listVolumes | listAll=true&pagesize=100&page=1 | 0m0.830s | 0m0.889s | 0m1.092s |
(# of obj: 8000) | listAll=true&pagesize=800&page=1 | 0m3.900s | 0m7.091s | 0m7.148s |
| listAll=true&pagesize=1500&page=1 | 0m5.887s | 0m13.906s | 0m13.077s |
| listAll=true&pagesize=3000&page=1 | 0m11.037s | 0m8.737s | 0m27.093s |
| listAll=true&pagesize=5000&page=1 | 0m18.859s | 0m40.966s | 1m9.951s |
| listAll=true |
| 0m0.526s |
|
|
|
|
|
|
listStoragePools | listAll=true&pagesize=100&page=1 | 0m0.313s | 0m0.526s | 0m0.346s |
(# of obj: 248) | listAll=true | 0m0.454s | 0m1.115s | 0m1.182s |
|
|
|
|
|
listProjects | listAll=true&pagesize=100&page=1 | 0m0.304s | 0m27.884s |
|
| listAll=true&pagesize=800&page=1 | 0m1.019s | 4m5.661s |
|
| listAll=true&pagesize=1500&page=1 | 0m1.696s |
|
|
| listAll=true&pagesize=3000&page=1 | 0m2.986s |
|
|
| listAll=true | 0m5.213s |
|
|
|
|
|
|
|
listProjectAccounts | listAll=true&projectid=3 | 0m33.247s |
|
|
| listAll=true&projectid=4 | 0m15.992s |
|
|
| listAll=true |
| 0m13.772s |
|
|
|
|
|
|
listProjectInvitations | listAll=true&pagesize=100&page=1 | 0m0.129s |
|
|
| listAll=true&pagesize=800&page=1 | 0m0.515s |
|
|
| listAll=true | 0m0.843s |
|
|
|
|
|
|
|
listAsyncJobs | listAll=true&pagesize=100&page=120 | 0m0.544s |
| 0m2.210s |
(# of obj: 12000) | listAll=true&pagesize=800&page=5 | 0m8.027s | 0m21.987s | 0m18.309s |
| listAll=true&pagesize=1500&page=6 | 0m1.856s | 0m29.040s | 0m36.051s |
| listAll=true&pagesize=3000&page=3 | 0m3.396s | 0m27.300s | 1m3.663s |
| listAll=true&pagesize=5000&page=3 |
| 1m49.942s | 1m50.274s |
| listAll=true&pagesize=8000&page=1 | 0m7.731s |
|
|
| listAll=true | 0m9.139s | 1m32.254s |
|