Bulkdata Import

Goals

• Allowing Micro Finance Organizations to move from their excel based systems to Apache Fineract by importing their existing portfolio

• Integrating the existing data import tool with Apache Fineract

• Fix all the present bugs in the import tool

• Add new category of imports (Offices, CoA, Users, staff)

• Enhance the import features based on user feedback

Background and strategic fit

If any Micro Finance Institution wants to move to Apache Fineract platform, there is no way they can import their existing portfolio into the system. It will consume too much time if they need add their portfolio one by one. So there is a need to provide an option to import their existing portfolio in bulk by supporting different entity level API(s). Mifosx had an external tool, but it is becoming out of sync with platform frequently. So there is a request from community to redesign this tool and add as part of platform.

Advantages of merging the code with platform:

• Bulk upload API will always be in sync with platform APIs.
• Easy of build UI for bulk upload inside Community-App.
• Import will be much faster as we are eliminating one tier.

Assumptions

• User will download excel based templates and input the data into those excels and imports the data into Fineract platform
• Before importing Centers, Groups, Clients user needs to add Offices and Staff by using Fineract prototype app or needs to import offices, staff
• Before importing Loan portfolio user has to configure/add Currency details, Funds details, Charges, Loan Product configurations, Payment Types, Code and Code values
• Before importing SA/FDA/RDA, user has to configure SA/FDA/RDA products by using Fineract prototype app
  
  

Requirements

User interaction and design

Mockup screens

Under Admin->Organization, there should new menu item to import bulk data. By selecting this menu item the following screen shall be displayed

By selecting these options user will be provided to load corresponding data. And the import option will be provided in screens where actual entities are created. For example user can import the clients from clients listing view.

By selecting Import Clients option, following screen should be displayed.

Groups import options

Savings Accounts and their transactions can be imported with the following screens. This screen(s) should be shown by selecting Import Savings Accounts and Import Savings Transactions option(s) under Import Bulkdata

Loan Accounts and their transactions can be imported with the following screens. This screen(s) should be shown by selecting Import Loan Accounts and Import Loan Accounts Transactions option(s) under Import Bulkdata

Similarly user should be able to import Centers, Chart of Accounts, Offices, Staff and Users by providing relevant options in the Import Bulkdata.

REST API(s)

•  http://serveraddress:port/fineract-provider/api/vi/bulkimport/{entityType}

  • Method: GET 

The above API will return template excel file for the defined entityType and supported entity types are clients, groups,  centers , users, staff, offices, savings,  savingtransactions, loans,  loanrepayments. User has to save this excel, fill the corresponding data according to the template format and then upload this excel file by using following API.

•  http://serveraddress:port/fineract-provider/api/vi/bulkimport/{entityType}

  • Method: POST

By using above API users an import the data related to defined entity type

High level architecture

Bulkdata Import

Implementation:

BulkImportApiResource is responsible to expose the REST API(s). By using BulkImportReadPlatformService implementation, it can return the template excel to the caller and by using BulkImportWritePlatformService implementation it will save the data into DB

org.apache.fineract.infrastructure.bulkimport.api;
@Path("/bulkimport/{entityType}")
@Component
@Scope("singleton")
public class BulkImportApiResource {
	@GET
    @Consumes({ MediaType.APPLICATION_JSON })
    @Produces({ MediaType.APPLICATION_OCTET_STREAM })
	public Response getTemplate(@PathParam("entityType") final String entityType) ;
	
	@POST
    @Consumes({ MediaType.MULTIPART_FORM_DATA })
    @Produces({ MediaType.APPLICATION_OCTET_STREAM })
    public Response importFromTemplate(@PathParam("entityType") final String entityType, 
                                   @FormDataParam("file") final InputStream uploadedInputStream,
                                   @FormDataParam("file") final FormDataContentDisposition fileDetails, 
                                   @FormDataParam("file") final FormDataBodyPart bodyPart) ;
}
 

The implementation of BulkImportWritePlatformService is responsible to load excel data based on entityType and should save the data into DB.

package org.apache.fineract.infrastructure.bulkimport.service;

import java.io.InputStream;
import javax.ws.rs.core.Response;

public interface BulkImportWritePlatformService {
    public Response importTemplateData(final String entityType, final InputStream uploadedInputStream);
}

BulkImportReadPlatformService implementation is responsible to construct the template excel based on entityType. The implementation of this interface can have its own methods to do the actual job.

package org.apache.fineract.infrastructure.bulkimport.service;
import javax.ws.rs.core.Response;

public interface BulkImportReadPlatformService {

    public Response getTemplate(final String entityType);
}

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing