Page History

...

Generating the Samza API whitelist

In order to load the Samza API classes from the API classloader, we need to tell cytodynamics what those classes are. We can do this by providing a whitelist of packages/classes when building the cytodynamics classloader. All public interfaces/classes inside of samza-api should be considered an API class. One way to generate this whitelist is to use a Gradle task to find all the classes from samza-api and put that list in a file. Then, that file can be read by Samza when constructing the cytodynamics classloader. The Gradle task should also include classes from samza-kv.

...

Handling SamzaApplication.describe

The infrastructure classloader will include the concrete descriptors, and we will build an additional application classloader which can delegate to the infrastructure classloader when running describe.Since the application code is calling the descriptors directly, then the application classloader SamzaApplication.describe method needs to be able to delegate to the infrastructure classloader . However, we do not want to delegate for every class. We only want to delegate for certain components (e.g. descriptors , serdesprovided by the framework). We don't want to delegate for application dependencies or classes which are only implemented by the applicationTherefore, the application classloader from above can't be used, since it does not delegate to the infrastructure classloader.

We will build an additional classloader just to load the SamzaApplication, and then everything else will delegate to the existing infrastructure classloader. The infrastructure classloader still might delegate to the original application classloader, and that is good, because we want classes from the original application classloader to do the container processing.

Flow for loading a class from the additional application classloader for SamzaApplication.describe:

1. If a class is a Samza API classthe implementation of SamzaApplication specified by "app.class", then load it from the API this describe classloader.If
2. Load the class is on the infrastructure classpath and it is in the infrastructure whitelist (e.g. descriptor), load it from the infrastructure classloader.
3. If the class is on the application classpath, load it from the application classloader.
4. ClassNotFoundException

A consequence of this structure is that there are "multiple" application classloaders on the job coordinator: one in this describe flow and the one described above at "Application" classloader. Therefore, any classes loaded by one of the application classloaders cannot be used by the classes of the other application classloader. An example of when this could happen is in the low-level API. The application's TaskFactory implementation will be loaded by the application classloader described above, but the Kafka events deserialized into Avro objects will be loaded by the other application classloader. Even though the Avro objects are the same class (even associated with the same binary), the TaskFactory implementation won't be able to use the Avro objects since a different classloader instance was used. We can solve this by serializing the components specified through the descriptor and deserializing those components using the classloader that is used for the rest of the AM. This is consistent with the strategy to be able to serialize the whole job description. The interfaces have already been marked as Serializable.

Pros

• API classloader stays simpler
• Allows application to delegate to infrastructure for describe, and infrastructure to delegate to application for processing

Cons

...

• Includes having multiple classloaders associated with the same application classpath

...

1. The infrastructure classloader might do further delegation to other classloaders.

Classloader wiring

By using the special classloader to instantiate the "main" class, any dependencies will then be loaded using that classloader. Then Java will automatically propagate the special classloader through the rest of Samza. We can modify the "main" method to use reflection to load the "main" class and then trigger the actual Samza startup.

public static void main(String[] args) {
  ClassLoader isolatingClassLoader = buildIsolatingClassLoader();
  Class<?> isolatedClass = Class.forName(MainClass.class.getName(), true, isolatingClassLoader);
  isolatedClass.getDeclaredMethod("doMain").invoke(null);
}

Pros

• Cytodynamics provides an explicit and granular way to specify if a class should be from the parent classpath (i.e. API)
• Classloader propagation allows the correct external dependencies to be used, even if infrastructure and the application use different versions of the same dependency
• Do not need to modify existing Samza API classes
• Do not need to explicitly wire classloader through Samza

Cons

• Need to ensure proper specification of Samza API classes
  
  • Are there any classes that are not owned by Samza but are used as part of the Samza API? (e.g. java.lang)
• Need to generate separate classpaths for each classloader
• Multiple classloaders is not obvious, so certain assumptions are invalid (e.g. static variables are not shared across classloaders)
• Extra dependency for Samza
  • Seems like a very lightweight dependency though

...

The current working directory can be obtained from System.getProperty("user.dir"), and we can find the separate JAR directories from there in code. We can also generate the classpaths in code by finding all of the JAR files in a given directory.

Pros

• Easier to localize Samza infrastructure on its own, since it is separate from applications
• Evolves well into general split deployment, since can just localize different Samza packages to do an upgrade
• Leverages existing flow for localizing JARs
• Samza infrastructure can define the full runtime package of JARs (including dependencies) at build time

Cons

• Need to ensure that framework packages has consistent versions with the version of Samza used within the application
• Need to localize artifacts to multiple places
• Not all jobs use all infrastructure plugins, so this would localize more JARs than necessary for each job

...

For more context about why these changes are needed, see 135861549.

Pros

• Able to isolate log4j2 pluggable components built by Samza
• Can override Samza infrastructure logging configuration

Cons

• Samza ends up controlling log4j2 API version
• No support for isolation for log4j1 pluggable components, so existing apps would need to migrate to log4j2 to get isolation

...