This page describes ideas, thoughts, and planning around Apache Ignite 3.0 release.
One of the biggest challenges with the current version of Ignite is related to how it works with data schemas. Here are the main issues:
To solve this, we should switch to the "schema-first" approach, which means that the schema is defined for a cache/table before its creation. The proposed changes are:
BinaryObject
API is replaced with Record
API (the naming can be discussed, of course). Here, a record is essentially a binary tuple that represents a set of fields stored in a cache entry or a table row. Similar to current binary objects, records have a "field" method to extract individual fields. A record can also be deserialized to any subset of fields.Another pain point is the fact that all the configuration in Ignite is static. Almost any updates to the configuration require cluster restart. It is also not obvious which parameters can be updated and which cannot because all of them are located within the same set of beans – there is no separation.
The proposal is to go through all the parameters and create a clear separation between:
Once this separation is done, we will need to update internal logic to allow changes for the dynamic parameters – such changes must be possible without any node restarts. The set of dynamic parameters can be minimal in the first version, as long as there is a way to convert static parameters to dynamic going forward.
In addition, the current Spring XML format seems to be outdated – it's not even recommended for Spring itself. Not to mention that we have to add the whole Spring dependency solely for configuration conversion. We suggest using the LightBend Config library based on the HOCON format: https://github.com/lightbend/config. This format is designed specifically for configuration purposes, is compatible with JSON, Java Properties, and allows merges. The latter is very useful for dynamic configuration updates.
Currently, query API is attached to the cache API. I.e., to execute a query, one needs to create a cache first. This is incompatible with SQL-only use cases where all the tables are created via DDL. In this use case, executing a SQL query requires that the user creates a dummy cache without no data and uses this cache as an API gateway. This is very counterintuitive. Instead, we should have a designated API for SQL, which would be located on the top level (e.g. ignite.sql().executeQuery(..)
).
Ignite consists of many modules, most of which are optional and are not used by the majority of our users. Still, we deliver binaries as a single package, which the user has to modify manually by moving folders around. In addition, the package contains configuration files, as well as multiple startup scripts. One of the biggest issues is related to upgrades: if anything within the package is modified (one of the scripts, a configuration file, a set of enabled modules, etc.), these changes then must be somehow merged with the new version of the package. There is no clear standard way of doing this, which makes the whole process very counterintuitive and error-prone.
We need to come with a better way of modularizing the platform and apply it to all available ways of delivery (downloadable ZIP, RPM, DEB, etc.). Can we utilize Java modularization somehow?
Some of the modules (mainly, integration components and thin clients) can be isolated into separate projects with independent lifecycles.
GraalVM is gaining a lot of popularity, especially in the context of serverless environments and frameworks like Micronaut and Quarkus. Currently, even Ignite thin client blocks software from being built into a native GraalVM image – this needs to be fixed. Preferably, we should support this for thick clients as well, although this is a lower priority and might require much bigger effort.
Ignite 3.0 is a major release, which gives us a unique opportunity to make incompatible changes. This can be used to do a cleanup and remove deprecated APIs and features. Additionally, all APIs should be revisited from this point of view – anything that is not relevant anymore should be removed from the project; some APIs can be reworked and modernized.
List of proposed removals (WIP):
*Resource
annotations except for the IgniteInstanceResource
CacheRebalanceMode
and rebalance delay - should always work in the ASYNC
mode (current default)IgniteFuture
with Java's CompletableFuture