Page History

...

Generating the Samza API whitelist

In order to load the Samza API classes from the API classloader, we need to tell cytodynamics what those classes are. We can do this by providing a whitelist of packages/classes when building the cytodynamics classloader. All public interfaces/classes inside of samza-api should be considered an API class. One way to generate this whitelist is to use a Gradle task to find all the classes from samza-api and put that list in a file. Then, that file can be read by Samza when constructing the cytodynamics classloader. The Gradle task should also include classes from samza-kv.

...

A consequence of this structure is that there are "multiple" application classloaders on the job coordinator: one in this describe flow and the one described above at "API" classloader. Therefore, any classes loaded by one of the application classloaders cannot be used by the classes of the other application classloader. An example of when this could happen is in the low-level API. The application's TaskFactory implementation will be loaded by the application classloader described above, but the Kafka events deserialized into Avro objects will be loaded by the other application classloader. Even though the Avro objects are the same class (even associated with the same binary), the TaskFactory implementation won't be able to use the Avro objects since a different classloader instance was used. We can solve this by serializing the components specified through the descriptor and deserializing those components using the classloader that is used for the rest of the AM. This is consistent with the strategy to be able to serialize the whole job description. The interfaces have already been marked as Serializable.

Pros

• API classloader stays simpler
• Allows application to delegate to infrastructure for describe, and infrastructure to delegate to application for processing

Cons

• Additional classloader component adds complexity
  • Includes having multiple classloaders associated with the same application classpath
• Need to serialize and deserialize components of the application description
  • Currently, some application descriptions are not actually Serializable (e.g. application context factory for both SQL and Beam)

...