...
Automate the semi-manual testing procedure that is being followed now.
Automate log gathering and bug filing.
Optimize the test runs so that they can be used as self service and can be run with out hardware, for continuous integration and validation of business logic.
Fix the simulator so that it is in sync with the new features added during last few releases
Add capability to the simulator for fault injection tests,timeouts etc.
Self service regression tests that community can use to verify their changes.
Provide Profiling and coverage Information.
Simulator now up to date with current features.
...
Simulator: Comparison (old vs new) :
Old Simulator | New Simulator Improvements |
Mocks only success scenarios | Supports simulating faults-injections, timeouts |
No way to change/cleanup mock | Allows defining test case specific mocks which can defined and so applicable for entire test be cleaned up as appropriate suite |
Test only basic scenarios | Ability to test complex scenarios like VM. deployment retry logic, HA, VM migration etc.
|
No way to change/cleanup mock once.No way to check if the mocked behavior is executed
. Limited code coverage | Ability to check if mock executed. This makes tests more deterministic . With the ability to tests failure/negative scenariostests . Tests additional tests can be quickly developed thereby improving overall coverage. Enables devs to do better upfront testing rather then relying on QA team, should result in less bugs late in the cycle |
Validating check-ins for your local changes, using Simulator
...