Compiled 2017-2018
Apache Fineract Installations and Experiences: Led by Ramesh Padmanabhan, these notes capture use cases for Apache Fineract across some of its users/partners.Based on interviews with the partners, this document will capture the deployment and configuration environments, as well as challenges and lessons learned. The idea is to offer this as a shared resource to the community and help to influence hosting tools and recommendations for the forthcoming Apache Fineract CN architecture.
contact: Nazeer Shaikh
Webserver is Tomcat 7.0.6.4
MySQL: 5.7/5.8
Java 8 to run the platform
The database server and Webserver run in the same machine
With 150 tenants, MySQL becomes a bottleneck. Using the Amazon Aurora, RDS service resolved this problem.
Production Server:
EC2 instance: Large instance being used
Quad core 2.5Ghz cpu
160 GB SSD
16 GB DRAM
scale: 250 tenants
Production Server Cost: <EC2+DB: Demo + prodn+ staging, costs $1000/mo>
Utilization of instance: not yet captured
Staging server: Medium sized instance.
OS: 14.04 Ubuntu
Mifos build: Built own environment.
contact: Markus Geiss
Finerat CN architecture:
Domain driven design
can have multiple microservices
Specific topics are grouped together: customers are its own domain, Accounting, Loans, Deposits etc
. Could be using 3 microservices for a Loan balancing app
What binds the microservices together:
The API of the microservice. Every microservice has its own API. REST based with JSON objects.
Uses Spring REST, not Swagger. Every microservice is a Spring boot application. It does all the wiring.
Use Spring JMS(msgq) + Spring Data + Access Cassandra
extended Spring security
implemented multi-tenancy for Cassandra
Use Java Money for financial calculation
Scale:
Running in the Cloud at scale is Kuelap’s secret sauce.
Cloud/Instance:
using Google Cloud Platform(GCP) as that is more flexible
GCP is more flexible in definition of the Network and firewalls
- Database Architecture
Database Cluster: Use Galera 3 node cluster for MySQL
3 nodes for Cassandra
2 nodes for ActiveMQ(for fault tolerance)
Cassandra usage: stores every state change.
All journal entries are stored in Cassandra. All loan+savings go through Cassandra. Data for 1.5M – 2M customers
MySQL is used for Typical Relational Data: eg customer name+info
Command gateway - hides which DB is accessed. Provided by Spring
Architecture for Microservices:
run Docker container
use Orchestrattion
5 workers + 3 manager node
5 VMs medium size. Use 3 instances per service
Could run every microservice in its own VM. The framework supports discovery
10 usvcs*3 + 10 VM for data = 45-50 VMs. Too many to manage
Hence Docker is recommended. Docker packages multiple VMs into a container.
Orchestration: recommend Docker Swarm(this is a stripped down version of Kubernetes)
Current Fineract CN build
The demo-server instance expects MySQL+Cassandra if ‘persistent=true’ is set
Caveat: Data will be lost during shutdown, unless external MySQL/Cassandra is available
Security/Compliance:
No automatic compliance for PCI, relying on the cloud for ISO/PCI etc
Security inside the application: relying on RSA key pairs. Every message is password encrypted using tenant keys.
Once deployed: if password is lost, no more access. Need to wipe complete password
Next Fineract CN release:
no more demo server
need to use Docker for running Fineract
contact: Thynn Win
Mifos X installed in
Production environment:
EC2 instance: X-large M4
Use ELB to load balance up to 2 servers so far
Using only one server for the most part, financial report generation at end of month puts enough load to require the second.
Concerns: Memory leak during report generation may have caused exceptions.
Mifos X: version 16.12.01. Planning to migrate to v17
Challenges:
lack of full blown accounting functionality. Will Mifos I/O address this ?
Release for fixes are delayed due to the Apache approval process
Takes around 6 months to make one release
Current process is to raise tickets for changes to the application. Takes a while for the tickets to get closed
Scale: Reporting crashes start to occur at around 12K borrowers. Expect to grow to 100K borrowers in a year. This issue may since been resolved.
Security concerns: Being addressed by getting Java developers to make changes at the Application level.
Will be using Trend Micro running in AWS for IDS functionality
Wishlist for Apache Fineract CN
Implement Audit tables: for user login, to figure out who is in the system
Beef up the security
Improve scaling for report generation. Do more stress testing.
Improve Release Turnaround time
contact: Arvind Sujeeth
Mines.io is a fintech startup for developing countries. People can dial a code and request a loan. Partners with Telcos in the backend for mobile data. The company enables banks to approve loans for customers.
Use Mifos X for all customer loan accounts in Mifos ledger.
Use case: Mines.io uses only a small part of the functionality. Uses mainly core loan/savings functionality. All customers use mobile access, the UI is only used internally.
Why Mifos ? Came across Mifos and worked closely with architect. The API is really important, since the goal is to be programmatic or relatively high volumes of transactions that are expected.
Infrastructure used: AWS is used and Mifos X is part of an overall microservice architecture. Mifos X is used as a loan ledger service.
Production Environment:
Scaling: Plan to use multi-tenant capability in Mifos for horizontal sharding
Deployment in AWS: use Docker for Microservices deployment; Amazon scaling groups
Report generation: use external API for BI report for flexibility as well as to circumvent any potential performance issues in Mifos for this function
Concerns: Security is a huge concern
encrypt everything(AWS encrypted DB)
use Mifos Access Control mechanism
fully firewalled, no external access
https: for all communication
Sync with Mifos: Sync with Master branch of Fineract periodically
Fineract CN wishlist:
If all microservices are on one instance, concerned about heavy weight components like Cassandra and Zookeeper. The current usage only needs 10% of Mifos X functionality
Concern about using Cassandra for journal: noSQL is not a typical choice for strongly consistent use case
Would like instructions on how to get going: what is the bare minimum set.
Will there be API parity with Mifos X ?
Documentation: on github, readme, start scripts, Docker ?, separate out what is optional vs what is Core.
contact: Salton Massally
Deployments
have a few standalone deployments of Mifos, institutions provide their own hosting. Mifos platform installed locally
Also have Cloud based deployments at AWS, using multi-tenancy
Use AWS Opsworks to automate the setup config etc
Production Environment: AWS
One large server: 15GB memory
Storage: S3 large + 200GB storage volume
Database: RDS, DB server M3, use MySQL
Chef scripts for automation
Future plans: use Mifos I/O when its stable. Will maintain Mifos X for clients onsite installations
Scale: how many users/borrowers ? What is current utilization ?: tbd, will provide this information
Security:
need security audits
ensure DB is not accessible from outside, only from instances in same Security group.
Production instance is behind ELB. Only select ports are open
Authorized IP range for people who need to ssh
Enforce policies through Security groups, using AWS provided firewalls
Offering Banking as a Service:
This is offered to MFI, mostly in West Africa
Currently do not have the capacity for big banks
Wishlist for Fineract CN
Will every microservice have its own instance ?
How can they transition from Opsworks to Container(Docker like from EC2)
contact: Istvan Molnar
Implemented Apache Fineract for German bank. Migrated 5000 loan accounts so far, expected to scale to 50K loans in a year.
Reasons to pick Apache Fineract:
- reduce license fees was the original reason.
- Also did formal evaluation using metrics of modern technology, ability to modify code and ISO certifications.
- Fits the expertise of the company: MariaDB, MySQL, Angular development
Production Environment:
All locally hosted in the Bank.
Development Environment: Hosted in AWS. EC2 + Jenkins for continuous build
EC2: Single VM(medium instance). Used for development and testing only. Up to 30 user accounts. With Userstories deployed immediately using Jenkins.
Using Tomcat for test environment in Bank as well as in Production.
Bank Release manager in Bank will take Git reports and move to ‘perforce’ for version control.
One test environment for System Integration
One test environment for User acceptance
Local environment for Production.
Fineract version: 0.60 in use, want to move to 1.0
Created new Security Audit
User Authentication/Roles/Groups/Permissions. Added some new roles in Mifos.
Wishlist for Fineract CN
use OAuth: (avoid sharing credentials with service provider by using tokens)
Scale for future: Current design can support up to 50K loans but not 100K
The Bank uses separate Data Warehouse, separate CRM, report generation, reconciliation in SAP.
The monolithic design in Mifos X is not seen as a bottleneck, since they took away CRM. Created a clone and threw away stuff. Created a payment hub on their own.
Ideally: Payment Hub is a microservice.
Multiple Tenants is an important use case: Ideally multiple loan services can migrate to this platform.
Migration to Fineract CN: Can this be automated as much as possible ?
Can we have documents that shows how loans and customers can be migrated ?
contact: CTO - Henrik Esbensen
Was the cofounder of Dreamstart: goal is to digitize Savings groups
Have developed an Android App and capabilities on mobile phones. Want to use Fineract CN on the backend server side.
Previous experience was at Elevate Africa: they were an early adopter of Mifos X. Henrik helped define the requirements for Mifos X.
Production Environment: Gen 3 Running on Google Cloud Platform
Details: pending
Scale: Elevate Africa was Hosting Mifos on AWS for MFI only
5 admin users, 300 clients doing 100s of loans
DreamStart: For Savings groups
6-10 sample groups to scale to 30K-40K groups of 25 people groups.
Early interest from Urban users also shows potential demand for large scale
Mifos use case for Savings groups:
digitize all transactions that were previously done using physical ledgers
provide metrics to non-profits
created Android smart phone App for Mifos.
Longer term using Machine Learning/Analytics,
Offline processing and still synchronize data is important.
Henrik’s team builds front end for the App. NGOs help create the groups, Mifos handles the group structure.
Scale: 5M to 20M clients expected from one NGO
Special Feature: Rural areas have no reliable Internet. The App is designed to run offline and synchronize later. Using sequence of SMS, can still synchronize with no dataplans. Dreamstart uses Mifos for synchronization and responses.
Advantage of Fineract: Interfaces are provided to connect to Mobile Money, consolidates to one interface.
Advantage of Multi-tenant vs. separate instances: Aggregate data across all MFIs in a region.
Fineract CN advantage: Microservices architecture and Mobile Money interfaces
Security:
need to assure security of user data which is not encrypted in Mifos.
The authentication system is very solid
Various legal issues around data, different laws for countries
Some customers use containerization for compliance
Key Use case for Fineract CN: Enable people to host Fineract CN as oppose to building their own Infra. This is preferred for Saving Groups.
contact: Nayan Ambali
Initially built company (Conflux Technologies) to accelerate financial inclusion institutes by providing scalable platform and digitising the financial offerings to serve last mile customers at cost effectively. At present it is serving 60+ FIs in 15 countries.
RuPie is self-serivce urban micro-finance, offers micro-credit to semi-skilled and small business owner entirly on mobile phone. As customer repay the loan on time, RuPie unlocks larger loans at reduced cost with flexible terms.
At present, RuPie is serving in North Bengaluru and daily handling about 50 new loan applications and serving few hundred customers.
RuPie provides the guarantee to the Bank on behalf of the customer.
Microfinance interest rates are as high as 26%, Rupie wants to bring this down by offering efficient handing of loans.
Production Environment:
Uses Apache Fineract 1.0
Installed on AWS server, uses Aurora DB from AWS for database
Tomcat for front end.
Database: one master, one read replica: uses Medium EC2 instance
Scale: 250 end users, not very active: only around 20-25 requests during peak.
Bottleneck expected in a year: need Mifos I/O for scale
Questions for Fineract CN: feature parity, how to deploy ?
Security:
Relying on Fineract built in capabilities: Major security is protecting customer data.
Mobile data is encrypted.
For Bank integration use two accounts: set a limit on number of transactions.
Fineract CN wishlist: can the loan module in I/O support credit line feature: no amortization.
Can we reduce chattiness with client. Requires too many API calls in Mifos X
Tech and DevOps: All changes done in house, do not depend on community
Can we reduce chattiness with client. Requires too many API calls in Mifos X
Tech and DevOps: All changes done in house, do not depend on community
contact: Ashok Auty
Use case: built own product using Mifos X.(Finflux). Hosting for customers.
Production: Using Aurora DB in AWS for Finflux to scale properly
Use Web App and Tomcat at different layers to provide more security to isolate the platform
For database, created read replica for reporting
Using multi-tenant for many customers
Types of customers:
Customers not needing frequent upgrades(Quarterly): use x-large M4 instance
Supports <50K clients, 250K loans and one instance
Customers who need monthly enhancements: hosting in M3 X-large instance(12 core)
Supports 250K clients, > 100K loan accounts
Use Java multi-threading to initiate multiple threads and utilize multi cores.
1000-5000 loans/thread
Devops/CI: built in house. Core contributors to Mifos X. Finflux is an Enterprise product
Created private copies of code to eliminate performance issues. Ended up fixing DB structures and rewriting bottlenecks.
Wishlist for Mifos I/O : make it simple to use: simple deployment model
Feature parity
Security: Encrypt content and password
oAuth2: use SSL certification for encrypting, integrated with banks. Lot of info in APP, needs to be secure. Use 128b encryption + certificate
Have all earlier XSS, SQL, data corruption issues from Mifos X been resolved ?
Will Fineract CN team provide Docker ?
contact: Victor Romero
Use Case: Initial rollout is a Savings product. Working on fingerprint validation plugin.
Scale: Plan is for 250K users, > 400 concurrent users
Security: Customers must fingerprint validate to National database. This is a challenge.
Planning to build an API to compare the finger prints (NBIS - https://www.nist.gov/services-resources/software/nist-biometric-image-software-nbis )
For now we are connecting the current applications using the existing APIs to MIFOS (we are adding JWT tokes using KONG) for security
Release Cadence: Code will go back to Fineract
Description:
Using Mifos, the company has developed its own Core Banking system, since it was looking for a stable system with a wide base of users.
Targeting small banks and small financial companies, since the larger banks like Citi Banamex, HSBC etc use Mainframe software for core banking needs.
There are opportunities in Mexico for banks currently using SAP to migrate to Mifos IO.
Motivation for Mifos IO:
- synchronization across all branches of financial institutions is not currently easy to do with other providers
- open source + availability of Java developers.
Instances used for development:
Using DB cluster + TomCat+ load balancer(HA proxy) on AWS instances
AWS instance sizing:
Micro-medium
For DB: 12GB DRAM, 100GB(files), 4 cores, shared partition on MariaDB
Tomcat: medium instance(16GB DRAM, 20GB storage, 2 cores)
HA LB: micro-instance(2GB DRAM, 10GB File system)
Production instances: Still being sized, testing in progress
API: using Swagger API for mobile App communications
Security: In cloud using private and public keys
All communications between systems are encrypted. Using encryption for MariaDB as provided by database
Finerat CN wishlist:
feature plug-in for payment gateway
Pentaho for reporting: need to evaluate performance for account statement generation
Galera cluster: will one node be enough ?
Generating Account statements must meet regulatory needs in Mexico