Summary

We are proposing to develop a modular and performant command processing component in parallel to the currently available one in Fineract and gradually migrate REST API controllers/resource classes, command handlers and where necessary business logic services. These changes will improve the developer experience (speed) significantly by (re-)introducing proper Class types where we currently use Strings as a common denominator to process various command payloads. Having proper type information is a requirement to delegate all JSON related de-/serialization to the Jackson framework and ultimately get rid of all handcrafted JSON parsing boilerplate  code (ca. 20% of the overall code base). With that we'll regain error checking at compile time (vs. tedious bug hunting at runtime, because of e.g. a simple refactoring of a command attribute). We are expecting visible performance gains by using Jackson and having a simpler command processing pipeline that has configurable performance levels (sync, async, non-blocking).

Status  

Under Development

Who is Involved   

Background/Motivation

Fineract accumulated some technical debt over the years. One area that is implicated is type-safety of internal and external facing APIs, the most prominent of which is Fineract's REST API. In general the package layout of the project reflects a more or less classic layered architecture (REST API, data transfer/value objects, business logic services, storage/repositories). The project predates some of the more modern frameworks and best practices that are available today and on occasions the data structures that are exchanged offer some challenges (e.g. generic types). Fineract's code base reflects that, especially where JSON de-/serialization is involved. Nowadays, this task would be simply delegated to the Jackson framework, but when Fineract (Mifos) started the decision was made to use Google's GSON library and create handcrafted helper classes to deal with JSON parsing. While this provided a lot of flexibility this approach had some downsides:

The list doesn't end here, but in the end things boil down to two main points:

There has been already some preparatory work done concerning type safety, but until now we avoided dealing with the real source of this issue. Fineract's architectures devises read from write requests ("CQRS", https://martinfowler.com/bliki/CQRS.html) for improved scalability.

The read requests are not that problematic, but all write requests pass through a component/service that is called "SynchronousCommandProcessingService. As the name suggests the execution of business logic is synchronous (mostly) due to this part of the architecture. This is not necessarily a problem (not immediately at least), but it's nevertheless a central bottleneck in the system. Even more important: this service is responsible to route incoming commands to their respective handler classes which in turn execute functions on one or more business logic services. The payload of these commands are obviously not always the same... which is the main reason why we decided to use the lowest common denominator to be able to handle these various types and rendered all payloads as strings. This compromise bubbles now up in the REST API and the business logic layers (and actually everything in between).

Over the years we've also added additional features (e.g. idempotency guarantees for incoming write requests) that make it now very hard to reason about the execution flow. Testing the performance impact of such additions to the critical execution path even can't be properly measured. Note: the current implementation of idempotency relies on database lookups (quite often, for each incoming request) and none of those queries are cached. If we wanted to store already processed requests (IDs) in a faster system (let's Redis) then this can't be done without major refactoring.

In conclusion, if we really want to fix those issues that are not only cosmetic and affect the performance and the developer experience equally then we urgently need to fix the way how we process write requests aka commands.

Goals

Exclusions

Change Proposed

Command Wrapper

Class contains some generic attributes like:

The actual payload (aka command input parameters) are defined as a generic parameter. It is expected that the modules implement classes that introduce the payload types and inherit from the abstract command class.

Command Processing Service

These different performance level implementations need to be absolute drop-in replacements (for each other). It is expected that more performant implementations need more testing due to increased complexity and possible unforseen side effects. In case any problems show up we can always roll back to the required default implementation (synchronous).

NOTE: we should consider providing a command processing implementation based on Apache Camel once this concept is approved and we migrated already a couple of services. They are specialized for exactly this kind of use cases and have more dedicated people working on it's implementation. Could give more flexibility without us needing to maintain code.

References to users

Keep things lightweight and only reference users by their user names.

Critical patch of execution

Again, keep things as lightweight as possible and avoid any database operations during the command processing.

Alternatives Considered

We could just opt to use an existing framework like Apache Camel which provides all necessary building parts to implement and efficient command processing infrastructure. We still recommend starting with a standalone solution that is properly abstracted with interfaces. Once the basic mechanics and the performance are tested we can still opt to provide an alternative implementation (aka command processing plugin) that is based on Apache Camel. That way we avoid any vendor (framework) lock-in.

Phases of work

NOTE: the actual refactoring of a feature/service for type-safety will be done with the following steps