Addition of Manifest File with "Dependent-Modules" Attribute to All Geode Projects

To be Reviewed By: August 20th, 2020

Authors: Patrick Johnson

Status: Draft | Discussion | Active | Dropped | Superseded

Superseded by: N/A

Related: ClassLoader Isolation

Problem

Geode is comprised of multiple sub-projects, many of which depend on other sub-projects at runtime. Normally, all of these sub-projects, as well as any libraries they use, are put on the system classpath where they can be easily accessed at runtime. Below is the runtime classpath from the gfsh-dependencies jar's manifest file:

Class-Path: geode-common-1.14.0-build.0.jar geode-common-services-1.14.0-build.0.jar geode-connectors-1.14.0-build.0.jar geode-core-1.14.0-build.0.jargeode-cq-1.14.0-build.0.jar
geode-gfsh-1.14.0-build.0.jargeode-log4j-1.14.0-build.0.jar geode-logging-1.14.0-build.0.jar geode-lucene-1.14.0-build.0.jar geode-memcached-1.14.0-build.0.jar geode-modules-1.14.0-build.0.jar
geode-old-client-support-1.14.0-build.0.jar geode-protobuf-1.14.0-build.0.jar geode-protobuf-messages-1.14.0-build.0.jar geode-rebalancer-1.14.0-build.0.jar geode-redis-1.14.0-build.0.jar
geode-serialization-1.14.0-build.0.jar geode-tcp-server-1.14.0-build.0.jar geode-wan-1.14.0-build.0.jar geode-module-management-1.14.0-build.0.jar geode-module-bootstrapping-1.14.0-build.0.jar
geode-management-1.14.0-build.0.jar jackson-databind-2.10.1.jar commons-lang3-3.10.jar jackson-annotations-2.10.1.jar jackson-core-2.10.1.jar log4j-api-2.13.1.jar
geode-membership-1.14.0-build.0.jar geode-http-service-1.14.0-build.0.jar geode-unsafe-1.14.0-build.0.jar httpclient-4.5.12.jar httpcore-4.4.13.jar HikariCP-3.4.2.jar jaxb-api-2.3.1.jar
log4j-jcl-2.13.1.jar spring-shell-1.2.0.RELEASE.jar rmiio-2.1.2.jar antlr-2.7.7.jar javax.activation-1.2.0.jar istack-commons-runtime-3.0.11.jar jaxb-impl-2.3.2.jar commons-validator-1.6.jar
shiro-core-1.5.3.jar shiro-config-ogdl-1.5.3.jar commons-beanutils-1.9.4.jar commons-codec-1.14.jar commons-collections-3.2.2.jar commons-io-2.6.jar commons-logging-1.2.jar classgraph-4.8.52.jar
micrometer-core-1.4.1.jar swagger-annotations-1.5.23.jar fastutil-8.3.1.jar javax.resource-api-1.7.1.jar jetty-webapp-9.4.21.v20190926.jar jetty-servlet-9.4.21.v20190926.jar
jetty-security-9.4.21.v20190926.jar jetty-server-9.4.21.v20190926.jar javax.servlet-api-3.1.0.jar jna-platform-5.5.0.jar jna-5.5.0.jar jopt-simple-5.0.4.jar snappy-0.4.jar
jgroups-3.6.14.Final.jar shiro-cache-1.5.3.jar shiro-crypto-hash-1.5.3.jar shiro-crypto-cipher-1.5.3.jar shiro-config-core-1.5.3.jar shiro-event-1.5.3.jar shiro-crypto-core-1.5.3.jar
shiro-lang-1.5.3.jar slf4j-api-1.7.30.jar spring-core-5.2.5.RELEASE.jar javax.activation-api-1.2.0.jar jline-2.12.jar HdrHistogram-2.1.12.jar LatencyUtils-2.0.3.jar
javax.transaction-api-1.3.jar spring-jcl-5.2.5.RELEASE.jar jetty-http-9.4.21.v20190926.jar jetty-io-9.4.21.v20190926.jar jetty-xml-9.4.21.v20190926.jar jetty-util-9.4.21.v20190926.jar
log4j-slf4j-impl-2.13.1.jar log4j-core-2.13.1.jar log4j-jul-2.13.1.jar lucene-analyzers-phonetic-6.6.6.jar lucene-analyzers-common-6.6.6.jar lucene-queryparser-6.6.6.jar lucene-core-6.6.6.jar
lucene-queries-6.6.6.jar jboss-modules-1.10.1.Final.jar vavr-0.10.3.jar vavr-match-0.10.3.jar protobuf-java-3.11.4.jar geo-0.7.7.jar netty-all-4.1.48.Final.jar spring-web-5.2.5.RELEASE.jar
spring-beans-5.2.5.RELEASE.jar

The ClassLoader Isolation RFC proposes that each sub-project of Geode be loaded as a separate module using JBoss Modules. This means that the classpath shown above will become much shorter as the majority of sub-projects, along with their dependencies, are removed from it and put into modules with their own ClassLoaders. Without everything being on the classpath, modules need a way to access classes from other modules that they depend on. JBoss Modules allows modules to link themselves to other modules that they depend on at runtime, but that requires knowing what modules depend on what other modules. We could link every module to every other module, but that would largely defeat the purpose of modularizing Geode in the first place.

At build-time, we know what other sub-project and libraries each sub-project depends on because those dependencies are declared in the sub-project's build.gradle file.

Normally, this is not a problem because all of Geode's projects are present on the Java classpath at runtime, allowing sub-projects to access classes from other sub-projects as required. However, the changes proposed by the ClassLoader Isolation RFC will result in all sub-projects being loaded as separate modules with different ClassLoaders and no longer being on the system classpath.

Once these modularizing changes go into effect, modules will no longer be able to access classes from other modules unless they explicitly create a dependency on them. In order to create dependencies between related modules, we need a way to determine at runtime what modules depend on what other modules, which, currently, we do not have. In addition to creating dependencies between modules, we may also need to load modules that are not already loaded when loading a module that depends on them. For example, imagine the scenario where module A depends on modules B and C which both have dependencies of their own, represented by the below graphic.

Loading module A would require all of the modules in its dependency hierarchy (B through F) to be loaded. If the modules required by module A are not already loaded, they would have to be loaded for module A to function correctly. Without knowledge of this hierarchy at runtime, we would be unable to ensure that all the modules required by module A are loaded when it is.

Anti-Goals

This proposal is intended only to solve the above problem of determining at runtime, which sub-projects/modules depend on which other sub-projects/modules and is not concerned with...

• The implementation of modules in Geode
• Refactoring Geode's sub-projects to break or change any existing dependencies between them
• Adding or removing sub-projects

Solution

 The proposed solution is to add a manifest file with a "Dependent-Modules" attribute (or add a "Dependent-Modules" attribute to existing manifest files) to the jar files of all Geode sub-projects. The "Dependent-Modules" attribute will consist of a space-delimited list of sub-projects required at runtime by the current sub-project in the form {name}-{version}. The "Dependent-Modules" attribute can then be read at runtime when loading the sub-project as a module and used to create the necessary dependencies on other modules. This list can be populated by a Gradle task at build time using the dependency information from each sub-project's build.gradle file, for example, these dependencies in a sub-project's build.gradle file:

compile(platform(project(':boms:geode-all-bom')))

api(project(':geode-core'))
api('org.apache.lucene:lucene-core')

compile(project(':geode-gfsh'))
implementation(project(':geode-logging'))
implementation(project(':geode-membership'))
implementation(project(':geode-serialization'))

implementation('org.apache.commons:commons-lang3')
implementation('org.apache.logging.log4j:log4j-api')

compileOnly(platform(project(':boms:geode-all-bom')))
compileOnly(project(':geode-common-services'))
compileOnly('com.fasterxml.jackson.core:jackson-annotations')

runtimeOnly('org.apache.lucene:lucene-analyzers-phonetic')

testImplementation(project(':geode-junit'))
testImplementation('org.apache.lucene:lucene-test-framework')

Would result in a "Dependent-Modules" attribute like this: 

Dependent-Modules: geode-gfsh-1.14.0-build.0 geode-core-1.14.0-build.0 geode-membership-1.14.0-build.0 geode-tcp-server-1.14.0-build.0 geode-http-service-1.14.0-build.0 geode-logging-1.14.0-build.0 geode-serialization-1.14.0-build.0 geode-common-1.14.0-build.0 geode-management-1.14.0-build.0 geode-unsafe-1.14.0-build.0

Notice that only project dependencies and only those scoped to be required at runtime (runtime, runtimeOnly, compile, implementation, and api) are included in "Dependent-Modules". Other library dependencies are excluded because they will not be loaded as modules and dependencies scoped as compileOnly or test are excluded because they are not required to be accessed by modules at runtime. You may also notice that the "Dependent-Modules" list contains modules that are not explicitly included in Gradle, specifically geode-tcp-server, geode-http-service, geode-common, geode-management, and geode-unsafe. These modules are included in the list because they are required by other modules that are explicitly called out in Gradle and therefore, are indirectly required by the module. While this list accurately represents the modules that must be loaded, it is not hierarchical, which makes it more difficult to determine how this module fits into the overall picture of Geode.

The sample hierarchy shown in the graphic above is much cleaner than many of the actual relationships between sub-projects, such as was shown by the sample Dependent-Modules attribute above. The below graph shows the actual relationships between all of Geode's sub-projects as defined by Gradle.

As you can see, the above graph resembles a tangled ball of yarn, making it hard to determine any kind of meaningful hierarchy, or even what sub-projects are involved; a structure like this is difficult to read and unnecessarily complicated for creating dependencies between modules since a module can access other modules indirectly via its dependencies (e.g. if A depends on B and B depends on C, then A can access C via B ). As long as there is a path between a module and the other modules it depends on, it can get what it needs, regardless of how many hops it takes. The above dependency graph can be simplified at build-time by removing dependencies that are reachable via other dependencies from the dependency list of each sub-project. The remaining dependencies in the list become the value of the "Dependent-Modules" attribute in the sub-project's manifest file. By doing this, the above graph can be simplified to the following:

This simplified dependency structure has significantly fewer paths, making it easier to understand and to construct a module hierarchy from. Despite the difference in appearance between the above graph and the original, both allow the same access between modules.  Additionally, using this method of simplifying dependencies, the sample "Dependent-Modules" attribute given earlier would be reduced to this: 

Dependent-Modules: geode-gfsh-1.14.0-build.0

The reduced version of this attribute may be minimal, but it provides us all the information necessary to load and link the module.

Changes and Additions to Public Interfaces

No anticipated changes to public interfaces.

Performance Impact

No anticipated performance impact.

Backwards Compatibility and Upgrade Path

No backward compatibility impact.

Prior Art

An alternative to this proposal would be to load every sub-project as a module at startup and create dependencies between every module and every other module. While this may solve the problem, it would also largely defeat the purpose of modularizing Geode in the first place.

FAQ

Answers to questions you’ve commonly been asked after requesting comments for this proposal.

Errata

What are minor adjustments that had to be made to the proposal since it was approved?