THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Page History
...
| ID | IEP-36 | |||||||
| Author | Denis Magda | |||||||
| Sponsor | Denis Magda | |||||||
| Created |
| |||||||
| Status |
colour | Grey |
|
| Table of Contents |
|---|
Motivation
...
- Spark Integration (to be discussed)
- SpringData and SpringBootKafka Integration
- TensorFlow Integration
- Cassandra Integration
...
Below is a list of existing integrations that won't be turned into Ignite modules but rather would be moved to separate Github repositories and won't be maintained by Ignite community for every core release. If later the community sees demand for an unsupported integration, it can be taken back and be officially supported (testing, dev, releases, compatibility with the core) as an Ignite module.
The list of integrations that to be moved to independent Github repositories and will not be supported by the community for every Ignite release:
We discussed in dev list[1] and agreed on creating a new repository for hosting our Ignite integrations.
As discussed [2] with respect to releases all the extensions need to be verified for an upcoming release and updated if needed (with the version increase only for those updated)
The list of integrations that to be moved to independent Github repositories and will not be supported by the community for every Ignite release:
| Integration Name | +1 (explain if needed) | -1 (with explanation) | |||
|---|---|---|---|---|---|
| Kafka | |||||
Denis Magda - that's a side project for apps that use Twitter | |||||
| Integration Name | +1 (explain if needed) | -1 (with explanation) | Denis Magda - that's a side project for apps that use Twitter. | ||
| ZeroMQ | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project) | |||
| RocketMQ | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | ||||
| ZeroMQ | Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project) | Storm | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project) | |
| FlumeRocketMQ | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | Alexey Zinoviev - all ETL streaming tools/modules should be kept in one place (as part of AI or as ETL Streaming AI separate project) I mean that Flume is tool for loading big datasets to AI | |||
| StormFlink | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project) | |||
| MQTTFlume | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | - all ETL tools/modules should be kept in one place (as part of AI or as ETL AI separate project) I mean that Flume is tool for loading big datasets to AI | |||
| FlinkCamel | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | ||||
| Hibernate | Denis Magda - Spring Data gets much bigger adoption for Ignite deployments. Don't see a lot of traction with Hibernate. It's hard to maintain it in various variations - Ignite goes with several modules of different versions. Better to have as an independent Github project with forks for specific Hibernate versions. | Alexey Zinoviev - I suppose it's useful feature for wide adoption among Java Devs who use AI not like cache, but like database | |||
Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project) | |||||
| MQTT | JMS | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project). Also, I didn't see the Kafka Integration in this list | ||
| Camel | AOP-Based Grid | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. Alexey Goncharuk Maybe drop it at all because moving this into a separate project may be a pain - a lot of internal API usages | |||
| Hibernate | Denis Magda - Spring Data gets much bigger adoption for Ignite deployments. Don't see a lot of traction with Hibernate. It's hard to maintain it in various variations - Ignite goes with several modules of different versions. Better to have as an independent Github project with forks for specific Hibernate versions. | JSR-107(JCache) | Denis Magda - don't see any value in supporting this JSR rather than claiming that specification. It's better to have much cleaner Ignite key-value API without any dependencies influenced by the specification. | Alexey Zinoviev - I suppose it's useful feature for wide adoption among Java Devs who use AI not like cache, but like database | |
| JMS | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody. | Alexey Zinoviev - all streaming tools/modules should be kept in one place (as part of AI or as Streaming AI separate project). Also, I didn't see the Kafka Integration in this list | |||
| AOP-Based Grid | Denis Magda - low usage, better to have as an independent Github project that can be maintained by anybody | Alexey Zinoviev we should ask about that question the user-community, I have heard many times that that the JCache implementation is important for Java Devs Ivan Pavlukhin It is quite natural for me to imagine integration with Ignite using some kind of standard API. The situation with JCache is similar to JDBC. AFAIR Spring has a JCache integration. If we are going to evolve caching trait then we should support easy integartion with Spring. If there alternatives to JCache then we should consider them. | OSGi | Denis Magda - this integration is already broken and badly maintained. Haven't come across anybody who uses OSGi in the projects Ignite is targeted for. Alexey Goncharuk Maybe drop it at all because moving this into a separate project may be a pain - a lot of internal API usages | |
| YARNJSR-107(JCache) | Denis Magda - not sure it's useful any longer and should be supported by the community.Alexey Kuznetsovdon't see any value in supporting this JSR rather than claiming that specification. It's better to have much cleaner Ignite key-value API without any dependencies influenced by the specification. | Alexey Zinoviev What was the purpose of this integration? | Mesos | Denis Magda - not sure it's useful any longer and should be supported by the community. Alexey Zinoviev What was the purpose of this integration? | we should ask about that question the user-community, I have heard many times that that the JCache implementation is important for Java Devs Ivan Pavlukhin It is quite natural for me to imagine integration with Ignite using some kind of standard API. The situation with JCache is similar to JDBC. AFAIR Spring has a JCache integration. If we are going to evolve caching trait then we should support easy integartion with Spring. If there alternatives to JCache then we should consider them. |
| OSGi | Denis Magda - this integration is already broken and badly maintained. Haven't come across anybody who uses OSGi in the projects Ignite is targeted for. | .NET: Legacy Entity Framework and ASP.NET integrations | Denis Magda - outdated, needs to be replaced with a new one version. Pavel Tupitsyn integrations with legacy technologies; also blocks .NET Core migration | Scalar | Alexey Goncharuk Not used, brings unnecessary dependency on scala, adds library conflicts |
The list of the integrations to be removed completely (don't even move them to an independent Github repo):
| YARN | Denis Magda - not sure it's useful any longer and should be supported by the community. Alexey Zinoviev What was the purpose of this integration? | ||
| Mesos | |||
| Integration | +1 (explain if needed) | -1 (with explanation) | |
|---|---|---|---|
Denis Magda - not sure | why these 2 were supported in the first place.it's useful any longer and should be supported by the community. Alexey Zinoviev What was the purpose of this integration? | ||
.NET: Legacy Entity Framework and ASP.NET integrations | we already have Thin Clients, duplicate features with fewer capabilities | outdated, needs to be replaced with a new one version. Pavel Tupitsyn integrations with legacy technologies; also blocks .NET Core migration | |
Denis Magda - community has already voted for the removal. | The community has already voted for the removal. Alexey Zinoviev IGFS should be moved to the separate package in the first | AOP-Based Grid | Alexey Goncharuk Unused, hard to move to a separate module due to many internal API usages | Ignite Schedule | Alexey Goncharuk same issues as with local caches Sergey Kozlov - it can be implemeted in user code Pavel Kovalenko Ignite integration with distributed schedulers like Airflow or Oozie can be a better decision. |
APIs for Removal
As part of the modularization, that needs to be considered for Ignite 3.0, it's worthwhile listing all the APIs that the community is planning to remove in Ignite 3.0. The APIs can belong to both Ignite core and modules that will stay in Ignite and will be officially supported by the community:
| Scalar | Alexey Goncharuk Not used, brings unnecessary dependency on scala, adds library conflicts |
The list of the integrations to be removed completely (don't even move them to an independent Github repo):
| Integration | +1 (explain if needed) | -1 (with explanation) |
|---|---|---|
Redis and Memcached protocols support | Denis Magda - not sure why these 2 were supported in the first place. Sergey Kozlov - thin clients provide full and rich replacement | |
| ignite-clients module | Denis Magda - we already have Thin Clients, duplicate features with fewer capabilities | |
Denis Magda - community has already voted for the removal. | The community has already voted for the removal. Alexey Zinoviev IGFS should be moved to the separate package in the first | |
| AOP-Based Grid | Alexey Goncharuk Unused, hard to move to a separate module due to many internal API usages | |
| Ignite Schedule | Alexey Goncharuk same issues as with local caches Sergey Kozlov - it can be implemeted in user code Pavel Kovalenko Ignite integration with distributed schedulers like Airflow or Oozie can be a better decision. |
APIs for Removal
As part of the modularization, that needs to be considered for Ignite 3.0, it's worthwhile listing all the APIs that the community is planning to remove in Ignite 3.0. The APIs can belong to both Ignite core and modules that will stay in Ignite and will be officially supported by the community:
| API | +1 (explain if needed) | -1 (with explanation) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Already deprecated APIs | |||||||||||||||
| Local caches | Alexey Kuznetsov Can we "emulate" local cache by partitioned with "node filter to one node"? Alexey Goncharuk Local cache is meaningless in a distributed system, especially when a transaction is involved: suppose a prepare step completed and a node with local cache goes down. According to 2PC, we cannot proceed until the node goes up again | ||||||||||||||
| Spatial indexes | Denis Magda - the API is broken and not suited for production. Can be designed from scratch with Ignite 3.1 and 3.2. | Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided in other case we will poor with indices | |||||||||||||
| Full-text search | Denis Magda - the API is broken and not suited for production. Can be designed from scratch with Ignite 3.1 and 3.2. | ||||||||||||||
| API | +1 (explain if needed) | -1 (with explanation) | Already deprecated APIs | Local caches | Alexey Kuznetsov Can we "emulate" local cache by partitioned with "node filter to one node"? Alexey Goncharuk Local cache is meaningless in a distributed system, especially when a transaction is involved: suppose a prepare step completed and a node with local cache goes down. According to 2PC, we cannot proceed until the node goes up again | ||||||||||
| Spatial indexes | Denis Magda - the API is broken and not suited for production. Can be designed from scratch with Ignite 3.1 and 3.2. | Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided in other case we will poor with indices | |||||||||||||
| Full-text search | Denis Magda - the API is broken and not suited for production. Can be designed from scratch with Ignite 3.1 and 3.2. | Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided in other case we will poor with full-text indices | |||||||||||||
| Checkpointing SPI | Denis Magda - Ignite caches/tables can be used to store the checkpoints. This API is redundant. | Ignite.lock | Alexey Goncharuk The distributed lock concept is broken (see corresponding Aphyr video). The usual lock usage pattern is lock - update cache - unlock. The same semantics can be achieved using pessimistic transactions with keys locking, but instead, for an end-user, it is clear how lock can fail in a distributed system (transaction rollback). Pavel Kovalenko Very fragile API. Not tested well. Some failures can lead to unavailability of controlling such locks. Can be returned back and reworked after implementation of some consensus algorithm like Raft. | Denis Magda - unless an alternate solution is provided as part of existing Transactional APIs. There has to be a clear migration guide for Ignite.lock customers. | Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided | ||||||||||
| Cache.lock | Alexey Goncharuk Same arguments as with Ignite.lock, but even more redundant API | Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided | |||||||||||||
| Ignite Data Structures | Pavel Kovalenko Can be marked as not safe and reworked after implementation of some consensus algorithm like Raft. Ivan Pavlukhin The implementation quality is really not sufficient. As for me data structures in current flavor should not be used in production. We can deprecate current implementation as a first step. Later on we can add all necessarry implementations as they are ready and a required development effort seems not trivial. | Denis Magda - that's basic functionality of every IMDG and in-memory cache. We can't remove it. Instead, let's plan through activities that improve or reconsider current implementations. Alexey Zinoviev Agree with Denis Magdaalso we should extend the list of basic structures (ready to help) | |||||||||||||
provided in other case we will poor with full-text indices | |||||||||||||||
| Checkpointing SPI | Denis Magda - Ignite caches/tables can be used to store the checkpoints. This API is redundant. | ||||||||||||||
| Ignite.lock | Alexey Goncharuk The distributed lock concept is broken (see corresponding Aphyr video). The usual lock usage pattern is lock - update cache - unlock. The same semantics can be achieved using pessimistic transactions with keys locking, but instead, for an end-user, it is clear how lock can fail in a distributed system (transaction rollback). Pavel Kovalenko Very fragile API. Not tested well. Some failures can lead to unavailability of controlling such locks. Can be returned back and reworked after implementation of some consensus algorithm like Raft. | Denis Magda - unless an alternate solution is provided as part of existing Transactional APIs. There has to be a clear migration guide for Ignite.lock customers. Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided | |||||||||||||
| Cache.lock | Alexey Goncharuk Same arguments as with Ignite.lock, but even more redundant API | Alexey Zinoviev It could be removed only in the 3.1 then the new implementation will be provided | |||||||||||||
| Ignite Data Structures | Pavel Kovalenko Can be marked as not safe and reworked after implementation of some consensus algorithm like Raft. Ivan Pavlukhin The implementation quality is really not sufficient. As for me data structures in current flavor should not be used in production. We can deprecate current implementation as a first step. Later on we can add all necessarry implementations as they are ready and a required development effort seems not trivial. | Denis Magda - that's basic functionality of every IMDG and in-memory cache. We can't remove it. Instead, let's plan through activities that improve or reconsider current implementations. Alexey Zinoviev Agree with Denis Magdaalso we should extend the list of basic structures (ready to help) | |||||||||||||
| GAR files | Denis Magda - to be replaced with JARs | ||||||||||||||
"Force server mode" for client nodes | GAR files | Denis Magda - to be replaced with JARs | "Force server mode" for client nodes | Denis Magda - don't see any value any longer. Alexey Goncharuk Already deprecated | Daemon nodes | Denis Magda - Visorcmd has to be preserved and updated to another protocol Alexey Kuznetsov - Visorcmd should be merged with controls.sh | CacheRebalanceMode.NONE and Rebalance Delay | Denis Magda - it can break data consistency in a cluster. Also, remove force rebalance mode as it can be used only if rebalance delay is set. Alexey Goncharuk The mode does not make sense, cannot be explained to an end-used | Indexing SPI | Denis Magda - it's highly unlikely that anybody used this. The community supports all the querying engines on its own. | QueryEntities and Annotations based configuration of SQL | Denis Magda - Alexey Goncharuk is going to propose an alternate API that unites these concepts. Sergey Kozlov - require full redesign, it should be fully compatible JDBC SQL and visa-versa | @CentralizedAffinityFunction | Alexey Goncharuk This API is no longer needed after exchange merge was introduced | |
| IgniteCache.localPeek, localEntries, localClear, localSize | Ivan Pavlukhin These methods look more like a debug stuff and have confusing semantics. At least should be moved outside of IgniteCache facade. | Anton Vinogradov Such API useful for tests and bug analysis |
Risks and Assumptions
- Challenges with a modification of existing build and testing procedures
- Release policies have to be updated to ensure that modules & core versions compatibility matrix is updated regularly.
Discussion Links
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html
Reference Links
This initiative is also related to the discussion of Apache Ignite APIs update/removal - Apache Ignite 3.0 Wishlist
Tickets
| Daemon nodes | Denis Magda - Visorcmd has to be preserved and updated to another protocol Alexey Kuznetsov - Visorcmd should be merged with controls.sh | |
| CacheRebalanceMode.NONE and Rebalance Delay | Denis Magda - it can break data consistency in a cluster. Also, remove force rebalance mode as it can be used only if rebalance delay is set. Alexey Goncharuk The mode does not make sense, cannot be explained to an end-used | |
| Indexing SPI | Denis Magda - it's highly unlikely that anybody used this. The community supports all the querying engines on its own. | |
| QueryEntities and Annotations based configuration of SQL | Denis Magda - Alexey Goncharuk is going to propose an alternate API that unites these concepts. Sergey Kozlov - require full redesign, it should be fully compatible JDBC SQL and visa-versa | |
| @CentralizedAffinityFunction | Alexey Goncharuk This API is no longer needed after exchange merge was introduced | |
| IgniteCache.localPeek, localEntries, localClear, localSize | Ivan Pavlukhin These methods look more like a debug stuff and have confusing semantics. At least should be moved outside of IgniteCache facade. | Anton Vinogradov Such API useful for tests and bug analysis |
Risks and Assumptions
- Challenges with a modification of existing build and testing procedures
- Release policies have to be updated to ensure that modules & core versions compatibility matrix is updated regularly.
Discussion Links
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html
Reference Links
This initiative is also related to the discussion of Apache Ignite APIs update/removal - Apache Ignite 3.0 Wishlist
Tickets
Main issue : IEP-36: Modularization https://issues.apache.org/jira/browse/IGNITE-12355Jira server ASF JIRA columnIds issuekey,summary,issuetype,updated,assignee,customfield_12311032,customfield_12311037,customfield_12311022,customfield_12311027,priority,status,resolution columns key,summary,type,updated,assignee,Priority,Priority,Priority,Priority,priority,status,resolution maximumIssues 20 jqlQuery project = Ignite AND labels IN (iep-36) order by key serverId 5aa69414-a9e9-3523-82ec-879b028fb15b
Sub-Tasks
Migrate flink module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12356
Migrate Twitter module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12357
Migrate ZeroMQ module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12358
Migrate RocketMQ module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12359
Migrate Storm module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12360
Migrate Flume module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12361
Migrate MQTT module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12362
Migrate Camel module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12363
Migrate JMS module to ignite-extensions https://issues.apache.org/jira/browse/IGNITE-12364
Overview
Content Tools
Apps