Background and Motivation

Fineract accumulated some technical debt over the years. One area that is implicated is type-safety of internal and external facing APIs, the most prominent of which is Fineract's REST API. In general the package layout of the project reflects a more or less classic layered architecture (REST API, data transfer/value objects, business logic services, storage/repositories). The project predates some of the more modern frameworks and best practices that are available today and on occasions the data structures that are exchanged offer some challenges (e.g. generic types). Fineract's code base reflects that, especially where JSON de-/serialization is involved. Nowadays, this task would be simply delegated to the Jackson framework, but when Fineract (Mifos) started the decision was made to use Google's GSON library and create handcrafted helper classes to deal with JSON parsing. While this provided a lot of flexibility this approach had some downsides:

• the lowest common denominator is the string type (aka JSON blob); this is where we lose the type information
• the strings are transformed into JSONObjects; a little bit better than raw strings, but barely more than a hash map
• a ton of "magic" strings are needed to get/set values
• this approach makes refactoring unnecessarily more difficult
• to be able to serve an OpenAPI descriptor (as JSON and YAML) we had to re-introduce the type information at the REST API level with dummy classes that contain only the specified attributes; those classes are only used with the Swagger annotations and no were else
• some developers skipped the layered architecture and found it too tedious to maintain DTOs and JSON helper classes, and as a result just passed JSONObjects right to the business logic layer
• now the business logic is unnecessarily aware of how Fineract communicates to the outside world and makes replacing/enhancing the communication protocol (e.g. with GRPC) pretty much impossible

The list doesn't end here, but in the end things boil down to two main points:

• poor developer experience: boilerplate code and missing type safety cost more time
• bugs: the more code the more likely errors get introduced, especially when type safety is missing and we have to rely on runtime errors (vs. compile time).

There has been already some preparatory work done concerning type safety, but until now we avoided dealing with the real source of this issue. Fineract's architectures devises read from write requests ("CQRS", https://martinfowler.com/bliki/CQRS.html) for improved scalability.

The read requests are not that problematic, but all write requests pass through a component/service that is called "SynchronousCommandProcessingService. As the name suggests the execution of business logic is synchronous (mostly) due to this part of the architecture. This is not necessarily a problem (not immediately at least), but it's nevertheless a central bottleneck in the system. Even more important: this service is responsible to route

Background and Motivation

Fineract accumulated some technical debt over the years. One area that is implicated is type-safety of internal and external facing APIs, the most prominent of which is Fineract's REST API. In general the package layout of the project reflects a more or less classic layered architecture (REST API, data transfer/value objects, business logic services, storage/repositories). The project predates some of the more modern frameworks and best practices that are available today and on occasions the data structures that are exchanged offer some challenges (e.g. generic types). Fineract's code base reflects that, especially where JSON de-/serialization is involved. Nowadays, this task would be simply delegated to the Jackson framework, but when Fineract (Mifos) started the decision was made to use Google's GSON library and create handcrafted helper classes to deal with JSON parsing. While this provided a lot of flexibility this approach had some downsides:

• the lowest common denominator is the string type (aka JSON blob); this is where we lose the type information
• the strings are transformed into JSONObjects; a little bit better than raw strings, but barely more than a hash map
• a ton of "magic" strings are needed to get/set values
• this approach makes refactoring unnecessarily more difficult
• to be able to serve an OpenAPI descriptor (as JSON and YAML) we had to re-introduce the type information at the REST API level with dummy classes that contain only the specified attributes; those classes are only used with the Swagger annotations and no were else
• some developers skipped the layered architecture and found it too tedious to maintain DTOs and JSON helper classes, and as a result just passed JSONObjects right to the business logic layer
• now the business logic is unnecessarily aware of how Fineract communicates to the outside world and makes replacing/enhancing the communication protocol (e.g. with GRPC) pretty much impossible

The list doesn't end here, but in the end things boil down to two main points:

• poor developer experience: boilerplate code and missing type safety cost more time
• bugs: the more code the more likely errors get introduced, especially when type safety is missing and we have to rely on runtime errors (vs. compile time).

There has been already some preparatory work done concerning type safety, but until now we avoided dealing with the real source of this issue. Fineract's architectures devises read from write requests ("CQRS", https://martinfowler.com/bliki/CQRS.html) for improved scalability.

The read requests are not that problematic, but all write requests pass through a component/service that is called "SynchronousCommandProcessingService. As the name suggests the execution of business logic is synchronous (mostly) due to this part of the architecture. This is not necessarily a problem (not immediately at least), but it's nevertheless a central bottleneck in the system. Even more important: this service is responsible to route incoming commands to their respective handler classes which in turn execute functions on one or more business logic services. The payload of these commands are obviously not always the same... which is the main reason why we decided to use the lowest common denominator to be able to handle these various types and rendered all payloads as strings. This compromise bubbles now up in the REST API and the business logic layers (and actually everything in between).

Over the years we've also added additional features (e.g. idempotency guarantees for incoming write requests) that make it now very hard to reason about the execution flow. Testing the performance impact of such additions to the critical execution path even can't be properly measured. Note: the current implementation of idempotency relies on database lookups (quite often, for each incoming request) and none of those queries are cached. If we wanted to store already processed requests (IDs) in a faster system (let's Redis) then this can't be done without major refactoring.

In conclusion, if we really want to fix those issues that are not only cosmetic and affect the performance and the developer experience equally then we urgently need to fix the way how we process write requests aka commands.

Target Personas

• developers
• integrators
• end users
• BaaS

Goals

• new command processing will run independently next to the legacy mechanics
• self contained
• fully tested
• ensure that the REST API is 100% backward compatible
• try to contain the migration and make it as easy as possible for the community to integrate those changes
• introduce types where needed and migrate the (old) JAX-RS REST resource classes to Spring Web MVC (better performance and better testability)
• introduce DTOs if not already available and make sure if they exist that they are not outdated
• assemble one DTO as command payload from all incoming REST API parameters (headers, query/path paramters, request bodies)
• annotate attributes in the DTOs with Jakarta Validation annotations to enforce constraints on their values
• wired REST API to the new command processing, one service at a time/pull request
• take a non-critical service (like document management) and migrate it to the new command processing mechanics from top (REST API) to bottom (business logic service)
• refactor command handlers to new internal API
• make sure that the business service logic classes/functions take only one DTO request input parameter (aka don't let a function have 12 input parameters of type string...)
• when all integration tests run successfully then remove all legacy boilerplate code that is not used anymore
• make an ordered list of modules/features (easiest, lowest hanging fruit first)
• maintain at least the same performance as the current implementation
• optional: improve performance if it can be done in a reasonable time frame
• optional: improve resilience if it can be done in a reasonable time frame

Non-Goals

• current command processing will stay untouched, will run independently of new infrastructure
• don't try cleaning up the storage layer; that's a separate effort for later (type safe queries, query peformance, clean entity classes)
• maker-checker is tightly coupled in the current command processing implementation upstream; this is a separate concern for a separate proposal (domains: security, workflow)
• doesn't need to be optimized for speed immediately
• no changes in the integration tests

Proposed API Changes

Command Wrapper

Class contains some generic attributes like:

• username
• tenant ID
• timestamp

The actual payload (aka command input parameters) are defined as a generic parameter "payload". It is expected that the modules implement classes that introduce the payload types and inherit from the abstract command class.

Command Processing Service

Three performance levels are configurable via application.properties

• synchronously (required): this is pretty much as we do right now (use virtual threads optionally)
• asynchronously (optional): with executor service and completable futures (use virtual threads optionally)
• non-blocking (optional): high performance LMAX Disruptor non-blocking implementation

These different performance level implementations need to be absolute drop-in replacements (for each other). It is expected that more performant implementations need more testing due to increased complexity and possible unforeseen side effects (thread local variables, transactions). In case any problems show up we can always roll back to the required default implementation (synchronous).

NOTE: we should consider providing a command processing implementation based on Apache Camel once this concept is approved and we migrated already a couple of services. They are specialized for exactly this kind of use cases and have more dedicated people working on it's implementation. Could give more flexibility without us needing to maintain code.

Middlewares

TBD

Command Handlers

TBD

References to users (aka AppUser)

Keep things lightweight and only reference users by their user names.f

Risks

TBD

• feature creep

ETA

The module has been created and merged upstream ("fineract-command"). You can try things out locally with these commands:

./gradlew :fineract-command:build
./gradlew :fineract-command:jmh

Diagrams

TBD

Related Jira Tickets

• https://issues.apache.org/jira/browse/FINERACT-2021
• https://issues.apache.org/jira/browse/FINERACT-1744
• https://issues.apache.org/jira/browse/FINERACT-1909
• https://issues.apache.org/jira/browse/FINERACT-2452
• https://issues.apache.org/jira/browse/FINERACT-2485 

s implementation. Could give more flexibility without us needing to maintain code.

Middlewares

TBD

Command Handlers

TBD

References to users (aka AppUser)

Keep things lightweight and only reference users by their user names.f

Risks

TBD

• feature creep

ETA

The module has been created and merged upstream ("fineract-command"). You can try things out locally with these commands:

./gradlew :fineract-command:build
./gradlew :fineract-command:jmh

Diagrams

TBD

Related Jira Tickets

• https://issues.apache.org/jira/browse/FINERACT-2021
• https://issues.apache.org/jira/browse/FINERACT-1744
• https://issues.apache.org/jira/browse/FINERACT-1909
• https://issues.apache.org/jira/browse/FINERACT-2452
• https://issues.apache.org/jira/browse/FINERACT-2485 

To enable a more comprehensive Fineract project, there will be a new administrative backend User Interface (UI) component.  It will be a separate GitHub repository within the Apache Fineract project.  

It will be aimed explaining the key functionality of fineract to devs and to act as the demo infrastructure.  It will be aimed at being downloaded as part of the Docker container from the ASF, for example.  

It should include, for the system user and dev, a page showing all of the APIs organized in a sensible way, and generated automatically at each build.  

This back office component is NOT THE SAME as the end-user POC that is proposed in https://issues.apache.org/jira/browse/FINERACT-2440  

This does overlap partially with external open source projects that are offered under different licenses.  However, this will be apache 2.0 license.  

This project will use Angular.  

This project should re-imagine the Fineract use cases in a way that is visually simple, distinct, and relates to the several user groups that we see in the project:  Fintechs, embedded lending programs, non banking financial institutions (lenders), small banks, etc

Use cases will include, but not be limited to: 

• login and select user type
• configure other users 
• set up a new loan product 
• disburse a loan 
• create a savings account 
• configure global variables 
• run dashboards 

No one should work on this specific ticket unless assigned - the GSOC candidate we choose will be assigned this ticket.  

For more information, you should be reviewing emails on this subject and following the Wiki pages. 
https://lists.apache.org/list.html?dev@fineract.apache.org
https://cwiki.apache.org/confluence/display/FINERACT/GSOC+Program+at+Fineract 

"Moving away from RestAssured (low-level) API calls in integration tests and rather use fineract-client-feign would be a great improvement" 

Summary (with some assist from chatgpt for clarity) 

Apache Fineract has a large set of REST APIs and many integration tests currently call those APIs using RestAssured(low-level HTTP requests). This ticket is to help modernize the tests by switching them to use fineract-client-feign, which is Fineract’s higher-level API client. 

Goal

Create a simple migration approach and then migrate a small set of integration tests from RestAssured to fineract-client-feign.

Why we’re doing this

• Makes tests easier to read and maintain (less raw HTTP code).

• Encourages consistent API usage across tests.

• Reduces duplicated request-building logic (headers, base URLs, auth, etc.).

Scope of Work

1) Create a short migration plan

Write a short note (in the Jira ticket comments or a small doc) that answers:

• Where are the current RestAssured-based integration tests located?

• What’s the recommended pattern for using fineract-client-feign in tests?

• What should be migrated first (start small)?

2) Pick a small “starter set” of tests

Identify 2–5 integration tests that:

• Are simple (e.g., create/read/update a resource)

• Don’t involve complicated multi-step workflows

• Run reliably in CI

3) Implement the migration for the starter set

For each selected test:

• Replace RestAssured calls with fineract-client-feign client calls

• Keep the same assertions (same expected behavior)

• Ensure the tests still pass locally and in CI

4) Document the new pattern

Add a short README note or comments in the test code showing:

• How to initialize/configure the Feign client for tests

• How auth/session is handled

• A small “before vs after” explanation (1 paragraph is enough)

Acceptance Criteria

• A brief migration plan is written and linked in the ticket.

• At least 2 integration tests have been converted to use fineract-client-feign.

• All tests pass (locally and/or in CI).

• A short note exists explaining how to write future integration tests using fineract-client-feign.

Notes / Hints for a beginner

• Start by converting just one very small test to learn the pattern.

• Keep changes small and easy to review (one test per commit is ideal).

• If something is unclear (e.g., how auth is set up), add a comment in the ticket describing what you found.

Out of Scope (for this ticket)

• Migrating all integration tests across the repo

• Refactoring production API code

• Changing API behavior—this is only a test client swap

Current gradle plugin which is used to generate classes based on avro schema files is unmaintained...

We should find and start using an alternative solution:

1. Bakdata Avro

https://plugins.gradle.org/plugin/com.bakdata.avro 

2. Eventloop software

https://central.sonatype.com/artifact/eu.eventloopsoftware.avro-gradle-plugin/eu.eventloopsoftware.avro-gradle-plugin.gradle.plugin/versions

3. Martin's Java code

https://github.com/martinsjavacode/avro-gradle-plugin

We need to investigate which would be the best choice of these and make the necessary changes.

Acceptance criteria

• Current avro generation gradle plugin to be changed
  • Due to EOL and unmaintained
• The new plugin must provide the same functionalities and resulted classes
  • Requests
  • Responses
  • Methods

Synopsis

Apache Fluss currently incorporates the BITMAP data type within its metadata layer, but it remains inaccessible to end-users as it is trapped in the UnsupportedKeyword enum. While the aggregation merge engine in Fluss 0.9 supports rbm32/rbm64 at the storage level, BITMAP is not yet a first-class type. Users must currently declare bitmap columns as BYTES.

This GSoC project aims to enable end-to-end native support for the BITMAP data type to allow efficient server-side unique counting. By shifting the computational burden from the client side to the storage side, we can reduce network I/O and CPU utilization for high-cardinality DISTINCT-style aggregations. The project will introduce a proper BITMAP DDL type, SQL functions, and pushdown optimization via applyAggregates().

Benefits to Community

1. Network I/O Efficiency: With bitmap pushdown, only one serialized RoaringBitmap is transferred per group instead of all raw rows, reducing network cost from O(N) to O(G).

2. CPU Utilization Reduction: Heavy unique counting computation is offloaded to the Fluss TabletServer's native merge engine, reducing Flink TaskManager CPU overhead.

3. Ecosystem Interoperability: By using the standard RoaringBitmap binary serialization format, Fluss ensures bitmap data remains accessible to downstream consumers such as Flink, StarRocks, and Doris without requiring proprietary Fluss-specific headers or custom decoders.

4. UV Analytics Optimization: Enables efficient Unique Visitor analytics workflows with pre-aggregated bitmap fragments that can be efficiently merged on the storage side.

Deliverables

The student will deliver the following components:

1. Type System Enablement (fluss-common)

• Introduce BitmapType as a new logical type in fluss-common
• Extend DataTypeParser to support the BITMAP keyword in CREATE TABLE statements
• Define BITMAP type properties (nullable, not orderable, cannot be used as primary key or partition key)

2. Server-Side Aggregation Integration (fluss-server)

• Wire FieldRoaringBitmap32Agg to the new BITMAP logical type
• Extend FieldRoaringBitmap32AggFactory to accept DataTypeRoot.BITMAP in addition to DataTypeRoot.BYTES
• Update AggFunctionType.getSupportedDataTypeRoots() accordingly
• Perform comprehensive audit of AggregationMergeEngine for BITMAP type handling

3. Flink Connector Bridge (fluss-flink)

• Implement SQL UDFs: BITMAP_BUILD_AGG, BITMAP_OR_AGG, BITMAP_CARDINALITY, BITMAP_FROM_BYTES, BITMAP_TO_BYTES
• Extend PbDataTypeRoot with BITMAP = 16 for RPC message support
• Implement applyAggregates() pushdown optimization for BITMAP_OR_AGG
• Handle graceful fallback to Flink-side aggregation when pushdown is not applicable

4. Testing & Documentation

• Functional unit tests for BitmapType and FieldRoaringBitmap32Agg
• End-to-end integration tests (BitmapPushdownITCase) in flink-flink-common module
• Performance benchmarks measuring network I/O and CPU utilization improvements
• User documentation and SQL reference guides

Required Skills

• Proficiency in Java programming
• Understanding of distributed systems and data processing concepts
• Familiarity with Apache Flink or similar stream processing frameworks
• Knowledge of SQL and query optimization is a plus
• Experience with bitmap data structures (RoaringBitmap) is advantageous
• Good communication skills for community collaboration

Difficulty Level

Medium to Hard

This project requires understanding of multiple layers in the Fluss stack (common, server, flink connector) and involves type system changes, aggregation engine integration, and query optimization. A working prototype demonstrating BitmapType integration with FieldRoaringBitmap32Agg is available to help the student get started.

Mentors

• Giannis Polyzos (ipolyzos@apache.org)

Future Work (Stretch Goals)

• Native 64-bit support (BITMAP64 type) for IDs exceeding 32-bit range
• Advanced conversion functions: BITMAP_TO_ARRAY, BITMAP_TO_STRING, BITMAP_XOR_AGG
• Reverse materialization: UNNEST_BITMAP to explode bitmaps back into individual integer rows

Name and Contact Information

Project: Apache Fluss (Incubating)

Website: https://fluss.apache.org

Mailing List: dev@fluss.apache.org

GitHub: https://github.com/apache/fluss

Iceberg's Spark readers currently process scan tasks sequentially — each file is opened, fully consumed, and closed before moving to the next. For workloads with hundreds or thousands of small files (5 KB–1 MB), this creates significant overhead, especially on object stores with per-request latency. We would like to introduce an opt-in async mode that opens multiple small-file tasks concurrently and buffers their output into a shared iterator.

We have already been in discussion with a proposed contributor for this new feature

• GitHub issue: #15287
• WIP PR: #15341
• Design doc: Google Doc