...

1. Single threaded full API tests
   1. Single threaded API calls with proper assertions
   2. Single threaded async API calls with proper assertions
   3. Input parameters validation
   4. Configuration parameters validation
   5. Single node topology
   6. Multi nodes topology
   7. Operations from server nodes
   8. Operations from client nodes
   9. Cache modes
      1. partitioned, replicated, local
      2. atomic, transactional
      3. tx concurrency and isolation modes
   10. Cache memory modes
       1. onheap, 
       2. offheap
   11. Cache backups count - 0, 1, 2, ..., N
   12. Cache with store
   13. Cache with swap
   14. Cache with eviction policies
   15. Cache with exipry policies
   16. Marshallers - JDK, Binary, Optimized
   17. Peer class loading - On, Off
   18. Different key and value types - primitives, arrays, collections, POJOs, Serializables, Externalizables, Binarylizables.
2. Failover tests
   1. Multithreaded various API calls involving random branch choices with proper assertions
   2. Should be performed on changing topology
   3. May involve following technics
      1. Certain messages delaying
      2. Communication problems simulation
      3. Exception throwing from user implemented logic (e.g. SPIs, listeners, filters, interceptors etc).
   4. For combinations to cover see pt.1
   5. Tests should also be targeted to discover memory usage problems and possible memory leaks (in case of batching, deferred buffers, etc)
3. Benchmarks and performance tests
   1. Yardstick benchmark should be added
4. Long running performance tests
   1. May involve fault-tolerance test scenarios
   2. Should also address possible memory usage problems
   3. Should check that system parameters (throughput, latency, CPU, memory, IO, etc) are stable.

Full API coverage enhancements

There is a plane to have Full API test coverage of Ignite functionality. See: 

The purpose of this ticket is to make sure that all ignite operations should work for any configuration properties combination.

Ignite already has Full API test suites, but currently it's hard to test all configuration combinations. 

Configuration variations test framework

...

The framework has VariationsIterator which will produce the following 4 variation vectors for the matrix above: 

• [0, 0] - that means we need to the framework will set marshaller property to BinaryMarshaller and perClassLoadingEnabled flag to true. 
• [1, 0] - that means we need to set the framework will set marshaller property to OptimizedMarshaller and perClassLoadingEnabled flag to true.
• [0, 1] - that means we need to set the framework will set marshaller property to BinaryMarshaller and perClassLoadingEnabled flag to false.
• [1, 1] - that means we need to set the framework will set marshaller property to OptimizedMarshaller and perClassLoadingEnabled flag to false.

The following methods should be used to provide custom matrixes:

• ConfigVariationsTestSuiteBuilder.igniteParams(ConfigParameter<IgniteConfiguration>[][] igniteParams) 
• ConfigVariationsTestSuiteBuilder.cacheParams(ConfigParameter<CacheConfiguration>[][] cacheParams)

ConfigVariations - contains ready to use configuration parameters matrixes.

Parameters - contains util methods to build parameters matrixes.

Filtration of congifurations

ConfigVariationsTestSuiteBuilder also provides possibility to filter Ignite and Cache configurations. See withIgniteConfigFilters(IgnitePredicate<IgniteConfiguration>... filters) and withCacheConfigFilters(IgnitePredicate<CacheConfiguration>... filters) methods.

Full API coverage enhancements

There is a plane to have Full API test coverage of Ignite functionality. See: 

The purpose of this ticket is to make sure that all ignite operations should work for any configuration properties combination.