...

• Many of the possible solutions for job coordinator isolation also apply to the application runners
  • Pros
    • Leveraging solution for the job coordinator
  • Cons
    • If we already don't have app logic in the app runners, then this is not necessary.

Handling SamzaApplication.describe

• Add descriptors to API classloader

  • Since application code is directly calling Samza infrastructure (e.g. LiKafkaInputDescriptor), then the classloader structure proposed for job coordinator isolation will not work for SamzaApplication.describe. If we used that classloader structure, then the Samza-owned components would be loaded by the application classloader, and that would mean that any dependencies would also be loaded by the application classloader. Those Samza-owned components should be loaded by a classloader associated with Samza infrastructure, so that the proper dependencies can be used. We can add the concrete descriptors to the API classloader. We would need to whitelist the concrete descriptors that we want loaded from the API classloader.

  • Pros
    • Descriptors and tables are part of the user API, so this organization is more intuitive
    • Classloader structure does not need to change
  • Cons
    • Table functions are passed to table descriptors in describe and are used in processing. The API classloader won't be able to delegate back to the application classloader for the processing case.
    • More dependencies in API classloader
• Extract descriptors and table functions into separate thin modules which can be made part of the API classloader
  
  • This would require decoupling the table function creation from the table function processing logic, so that different classloaders can be used for instantiation and processing.
    
    • One way of doing this could be to change the table descriptor to accept a table function factory instead of the table function directly.
  • Pros
    • Descriptors and tables are part of the user API, so this organization is more intuitive
    • Results in more consistency between table descriptors and other descriptors (e.g. system descriptors)
  • Cons
    • Requires changing table API
• Additional plugins classloader which just has plugins (e.g. descriptors, tables, system factories, etc.)
  • Pros
    • Descriptors and tables are part of the user API, so this organization is more intuitive
  • Cons
    • Still have the issue with how to delegate between plugins classloader and application classloader
      • In describe, the application should delegate to the plugins, but in regular processing, it should be the other way around

Existing solutions for other technologies

...