Compiled 2017-2018

Apache Fineract Installations and Experiences: Led by Ramesh Padmanabhan, these notes capture use cases for Apache Fineract across some of its users/partners.Based on interviews with the partners, this document will capture the deployment and configuration environments, as well as challenges and lessons learned. The idea is to offer this as a shared resource to the community and help to influence hosting tools and recommendations for the forthcoming Apache Fineract CN architecture. 


Mifos Initiative (via Conflux staff)

contact: Nazeer Shaikh

 Resources used:

Webserver is Tomcat 7.0.6.4

MySQL: 5.7/5.8

Java 8 to run the platform

Challenges faced:

The database server and Webserver run in the same machine

With 150 tenants, MySQL becomes a bottleneck. Using the Amazon Aurora, RDS service resolved this problem.


Production Server:

EC2 instance: Large instance being used

  Quad core 2.5Ghz cpu

  160 GB SSD

  16 GB DRAM

scale: 250 tenants

Production Server Cost: <EC2+DB: Demo + prodn+ staging, costs $1000/mo>

Utilization of instance: not yet captured

Staging server: Medium sized instance.

OS: 14.04 Ubuntu

Mifos build: Built own environment.

Kuelap

 contact: Markus Geiss

 Finerat CN architecture:

  • Domain driven design

  • can have multiple microservices

  • Specific topics are grouped together: customers are its own domain, Accounting, Loans, Deposits etc

  • . Could be using 3 microservices for a Loan balancing app

 What binds the    microservices together:

 The API of the microservice. Every microservice has its own API. REST based with JSON objects.

 Uses Spring REST, not Swagger. Every microservice is a Spring boot application. It does all the wiring.

Use Spring JMS(msgq) + Spring Data + Access Cassandra

  • extended Spring security

  • implemented multi-tenancy for Cassandra

Use Java Money for financial calculation

Scale:

  • Running in the Cloud at scale is Kuelap’s secret sauce.

Cloud/Instance:

  • using Google Cloud Platform(GCP) as that is more flexible

  • GCP is more flexible in  definition of the Network and firewalls

        - Database Architecture

Database Cluster: Use Galera 3 node cluster for MySQL

  3 nodes for Cassandra

  2 nodes for ActiveMQ(for fault tolerance)


Cassandra usage: stores every state change.

All journal entries are stored in Cassandra. All loan+savings go through Cassandra. Data for 1.5M – 2M customers

 

MySQL is used for Typical Relational Data: eg customer name+info

Command gateway - hides which DB is accessed. Provided by Spring

 

Architecture for Microservices:

  • run Docker container

  • use Orchestrattion

  • 5 workers + 3 manager node

  • 5 VMs medium size.  Use 3 instances per service

  • Could run every microservice in its own VM. The framework supports discovery

  • 10 usvcs*3 + 10 VM for data = 45-50 VMs. Too many to manage

  • Hence Docker is recommended.  Docker packages multiple VMs into a container.

  • Orchestration: recommend Docker Swarm(this is a stripped down version of Kubernetes)

 

Current Fineract CN build

  • The demo-server instance expects MySQL+Cassandra if ‘persistent=true’ is set

  • Caveat: Data will be lost during shutdown, unless external MySQL/Cassandra is available

 

Security/Compliance:

  • No automatic compliance for PCI, relying on the cloud for ISO/PCI etc

  • Security inside the application: relying on RSA key pairs. Every message is password encrypted using tenant keys.

  • Once deployed: if password is lost, no more access. Need to wipe complete password

Next Fineract CN release:

  • no more demo server

  • need to use Docker for running Fineract


Thitsaworks 

contact: Thynn Win

Mifos X installed in

Production environment:

EC2 instance: X-large M4

Use ELB to load balance up to 2 servers so far

Using only one server for the most part, financial report generation at end of month puts enough load to require the second.

Concerns: Memory leak during report generation may have caused exceptions.

Mifos X: version 16.12.01. Planning to migrate to v17

Challenges:

  • lack of full blown accounting functionality. Will Mifos I/O address this ?

  • Release for fixes are delayed due to the Apache approval process

  • Takes around 6 months to make one release

  • Current process is to raise tickets for changes to the application. Takes a while for the tickets to get closed

Scale: Reporting crashes start to occur at around 12K borrowers. Expect to grow to 100K borrowers in a year. This issue  may since  been resolved.

Security concerns: Being addressed by getting Java developers to make changes at the Application level.

Will be using Trend Micro running in AWS for IDS functionality

Wishlist for Apache Fineract CN

  • Implement Audit tables: for user login, to figure out who is in the system

  • Beef up the security

  • Improve scaling for report generation. Do more stress testing.

  • Improve Release Turnaround time

Mines.io

contact: Arvind Sujeeth

Mines.io is a fintech startup for developing countries. People can dial a code and request a loan. Partners with Telcos in the backend for mobile data.  The company enables banks to approve loans for customers.

Use Mifos X for all customer loan accounts in Mifos ledger.

Use case: Mines.io uses only a small part of the functionality. Uses mainly core loan/savings functionality. All customers use mobile access, the UI is only used internally.

Why Mifos ? Came across Mifos and worked closely with architect. The API is really important, since the goal is to be programmatic or relatively high volumes of transactions that are expected.

Infrastructure used: AWS is used and Mifos X is part of an overall microservice architecture. Mifos X is used as a loan ledger service.

Production Environment:

Scaling: Plan to use multi-tenant capability in Mifos for horizontal sharding

Deployment in AWS: use Docker for Microservices deployment; Amazon scaling groups

Report generation: use external API for BI report for flexibility as well as to  circumvent any  potential performance issues in Mifos for this function

Concerns:  Security is a huge concern

  • encrypt everything(AWS encrypted DB)

  • use Mifos Access Control mechanism

  • fully firewalled, no external access

  • https: for all communication

Sync with Mifos: Sync with Master branch of Fineract periodically

Fineract CN wishlist:

  • If all microservices  are on one instance, concerned about heavy weight components like Cassandra and Zookeeper. The current usage only needs 10% of Mifos X functionality

  • Concern about using Cassandra for journal: noSQL is not a typical choice for strongly consistent use case

  • Would like instructions on how to get going: what is the bare minimum set.

  • Will there be API parity with Mifos X ?

  • Documentation: on github, readme, start scripts, Docker ?, separate out what is optional vs what is Core.

iDT Labs

contact: Salton Massally

Deployments

  • have a few standalone deployments of Mifos, institutions provide their own hosting. Mifos platform installed locally

  • Also have Cloud based deployments at AWS, using multi-tenancy

  • Use AWS Opsworks to automate the setup config etc

Production Environment: AWS

  • One large server: 15GB memory

  • Storage: S3 large + 200GB storage volume

  • Database: RDS, DB server M3, use MySQL

  • Chef scripts for automation

Future plans: use Mifos I/O when its stable. Will maintain Mifos X for clients onsite installations

Scale: how many users/borrowers ? What is current utilization ?: tbd, will provide this information

Security:

  • need security audits

  • ensure DB is not accessible from outside, only from instances in same Security group.

  • Production instance is behind ELB. Only select ports are open

  • Authorized IP range for people who need to ssh

  • Enforce policies through Security groups, using AWS provided firewalls

 Offering Banking as a Service:

  • This is offered to MFI, mostly in West Africa

  • Currently do not have the capacity for big banks

Wishlist for Fineract CN 

  • Will every microservice have its own instance ?

  • How can they transition from Opsworks to Container(Docker like from EC2)

DPC Hungary

contact: Istvan Molnar

Implemented Apache Fineract for German bank. Migrated 5000 loan accounts so far, expected to scale to 50K loans in a year.

Reasons to pick Apache Fineract:

- reduce license fees was the original reason.

- Also did formal evaluation using metrics of modern technology, ability to modify code and ISO certifications.

- Fits the expertise of the company: MariaDB, MySQL, Angular development

Production Environment:

All locally hosted in the Bank.

Development Environment: Hosted in AWS. EC2 + Jenkins for continuous build

EC2: Single VM(medium instance). Used for development and testing only. Up to 30 user accounts. With Userstories deployed immediately using Jenkins.

Using Tomcat for test environment in Bank as well as in Production.

Bank Release manager in Bank will take Git reports and move to ‘perforce’ for version control.

 

One test environment for System Integration

One test environment for User acceptance

Local environment for Production.


Fineract version: 0.60 in use, want to move to 1.0


Created new Security Audit

User Authentication/Roles/Groups/Permissions. Added some new roles in Mifos.


Wishlist for Fineract CN

  • use OAuth: (avoid sharing credentials with service provider by using tokens)

Scale for future: Current design can support up to 50K loans but not 100K


The Bank uses separate Data Warehouse, separate CRM, report generation, reconciliation in SAP.

The monolithic design in Mifos X is not seen as a bottleneck, since they took away CRM. Created a clone and threw away stuff. Created a payment hub on their own.

Ideally: Payment Hub is a microservice.

Multiple Tenants is an important use case: Ideally multiple loan services can migrate to this platform.

Migration to Fineract CN: Can this be automated as much as possible ?

Can we have documents that shows how loans and customers can be migrated ?

Dreamstart Labs

contact: CTO - Henrik Esbensen


Was the cofounder of Dreamstart: goal is to digitize Savings groups

Have developed an Android App and capabilities on mobile phones. Want to use Fineract CN on the backend server side.

Previous experience was at Elevate Africa: they were an early adopter of Mifos X. Henrik helped define the requirements for Mifos X.

Production Environment: Gen 3 Running on Google Cloud Platform

Details: pending

Scale:  Elevate Africa was Hosting Mifos on AWS for MFI only

5 admin users, 300 clients doing 100s of loans


DreamStart: For Savings groups

6-10 sample groups to scale to 30K-40K groups of 25 people groups.

Early interest from Urban users also shows potential demand for large scale

Mifos use case for Savings groups:

  • digitize all transactions that were previously done using physical  ledgers

  • provide metrics to non-profits

  • created Android smart phone App for Mifos.

  • Longer term using Machine Learning/Analytics,

  • Offline processing and still synchronize data is important.


Henrik’s team builds front end for the App. NGOs help create the groups, Mifos handles the group structure.

Scale: 5M to 20M clients expected from one NGO


Special Feature: Rural areas have no reliable Internet. The App is designed to run offline and synchronize later. Using sequence of SMS, can still synchronize with no dataplans. Dreamstart uses Mifos for synchronization and responses.


Advantage of Fineract: Interfaces are provided to connect to Mobile Money, consolidates to one interface.


Advantage of Multi-tenant vs. separate instances: Aggregate data across all MFIs in a region.

Fineract CN advantage: Microservices architecture and Mobile Money interfaces


Security:

  • need to assure security of user data which is not encrypted in Mifos.

  • The authentication system is very solid

  • Various legal issues around data, different laws for countries

  • Some customers use containerization for compliance


Key Use case for Fineract CN: Enable people to host Fineract CN as oppose to building their own Infra. This is preferred for Saving Groups.



RuPie

contact: Nayan Ambali

Initially built company (Conflux Technologies) to accelerate financial inclusion institutes by providing scalable platform and digitising the financial offerings to serve last mile customers at cost effectively. At present it is serving 60+ FIs in 15 countries.

RuPie is self-serivce urban micro-finance, offers micro-credit to semi-skilled and small business owner entirly on mobile phone. As customer repay the loan on time, RuPie unlocks larger loans at reduced cost with flexible terms.

At present, RuPie is serving in North Bengaluru and daily handling about 50 new loan applications and serving few hundred customers.

RuPie provides the guarantee to the Bank on behalf of the customer.

Microfinance interest rates are as high as 26%, Rupie wants to bring this down by offering efficient handing of loans.

Production Environment:

Uses Apache Fineract 1.0

Installed on AWS server, uses Aurora DB from AWS for database

Tomcat for front end.


Database: one master, one read replica: uses Medium EC2 instance

Scale: 250 end users, not very active: only around 20-25 requests during peak.

Bottleneck expected in a year: need Mifos I/O for scale

Questions for Fineract CN: feature parity, how to deploy ?

Security:

Relying on Fineract built in capabilities: Major security is protecting customer data.

Mobile data is encrypted.

For Bank integration use two accounts: set a limit on number of transactions.


Fineract CN wishlist: can the loan module in I/O support credit line feature: no amortization.

Can we reduce chattiness with client. Requires too many API calls in Mifos X

Tech and DevOps: All changes done in house, do not depend on community

Can we reduce chattiness with client. Requires too many API calls in Mifos X

Tech and DevOps: All changes done in house, do not depend on community

Conflux (Finflux)

contact: Ashok Auty

Use case: built own product using Mifos X.(Finflux). Hosting for customers.

Production: Using Aurora DB in AWS for Finflux to scale properly

Use Web App and Tomcat at different layers to provide more security to isolate the platform

For database, created read replica for reporting

Using multi-tenant for many customers

Types of customers:

  • Customers not needing frequent upgrades(Quarterly): use x-large M4 instance

Supports <50K clients, 250K loans and one instance

  • Customers who need monthly enhancements: hosting in M3 X-large instance(12 core)

Supports 250K clients, > 100K loan accounts

Use Java multi-threading to initiate multiple threads and utilize multi cores.

1000-5000 loans/thread

Devops/CI: built in house. Core contributors to Mifos X. Finflux is an Enterprise product

Created private copies of code to eliminate performance issues. Ended up fixing DB structures and rewriting bottlenecks.


Wishlist for Mifos I/O : make it simple to use: simple deployment model

Feature parity


Security: Encrypt content and password

oAuth2: use SSL certification for encrypting, integrated with banks. Lot of info in APP, needs to be secure. Use 128b encryption + certificate

Have all earlier XSS, SQL, data corruption issues from Mifos X been resolved ?

Will Fineract CN team provide Docker ?

Fintecheando

contact: Victor Romero

Use Case: Initial rollout is a Savings product. Working on fingerprint validation plugin.

Scale: Plan is for 250K users,  > 400 concurrent users

Security: Customers must fingerprint validate to National database. This is a challenge.

Planning to build an API to compare the finger prints (NBIS - https://www.nist.gov/services-resources/software/nist-biometric-image-software-nbis )

For now we are connecting the current applications using the existing APIs to MIFOS (we are adding JWT tokes using KONG) for security


Release Cadence: Code will go back to Fineract

Description:

Using Mifos, the company has developed its own Core Banking system, since it was looking for a stable system with a wide base of users.

Targeting small banks and small financial companies, since the larger banks like Citi Banamex, HSBC etc use Mainframe software for core banking needs.

There are opportunities in Mexico for banks currently using SAP to migrate to Mifos IO.

Motivation for Mifos IO:

- synchronization across all branches of financial institutions is not currently easy to do with other providers

- open source + availability of Java developers.


Instances used for development:

Using DB cluster + TomCat+ load balancer(HA proxy) on AWS instances

AWS instance sizing:

Micro-medium

For DB: 12GB DRAM, 100GB(files), 4 cores, shared partition on MariaDB

Tomcat: medium instance(16GB DRAM, 20GB storage, 2 cores)

HA LB: micro-instance(2GB DRAM, 10GB File system)


Production instances: Still being sized, testing in progress


 

API: using Swagger API for mobile App communications



Security: In cloud using private and public keys

All communications between systems are encrypted. Using encryption for MariaDB as provided by database


Finerat CN wishlist:

  • feature plug-in for payment gateway

  • Pentaho for reporting: need to evaluate performance for account statement generation

  • Galera cluster: will one node be enough ?

  • Generating Account statements must meet regulatory needs in Mexico



  • No labels