Bulkdata Import

Goals

Allowing Micro Finance Organizations to move from their excel based systems to Apache Fineract by importing their existing portfolio

• Integrating the existing data import tool with Apache Fineract

• Add new category of imports (Offices, CoA, Users, staff)

• Enhance the import features based on user feedback

Background and strategic fit

If any Micro Finance Institution wants to move to Apache Fineract platform, there is no way they can import their existing portfolio into the system. It will consume too much time if they need add their portfolio one by one. So there is a need to provide an option to import their existing portfolio in bulk by supporting different entity level API(s). Mifosx had an external tool, but it is becoming out of sync with platform frequently. So there is a request from community to redesign this tool and add as part of platform.

Advantages of merging the code with platform:

• Bulk upload API will always be in sync with platform APIs.
• Easy of build UI for bulk upload inside Community-App.
• Import will be much faster as we are eliminating one tier.

Assumptions

• User will download excel based templates and input the data into those excels and imports the data into Fineract platform
• Before importing Centers, Groups, Clients user needs to add Offices and Staff by using Fineract prototype app or needs to import offices, staff
• Before importing Loan portfolio user has to configure/add Currency details, Funds details, Charges, Loan Product configurations, Payment Types, Code and Code values
• Before importing SA/FDA/RDA, user has to configure SA/FDA/RDA products by using Fineract prototype app
  
  

Requirements

User interaction and design

Mock up screens

Under Admin->Organization, there should new menu item to import bulk data. By selecting this menu item the following screen shall be displayed

By selecting these options user will be provided to load corresponding data. And the import option will be provided in screens where actual entities are created. For example user can import the clients from clients listing view.

By selecting Import Clients option, following screen should be displayed.

Groups import options

Savings Accounts and their transactions can be imported with the following screens. This screen(s) should be shown by selecting Import Savings Accounts and Import Savings Transactions option(s) under Import Bulkdata

Loan Accounts and their transactions can be imported with the following screens. This screen(s) should be shown by selecting Import Loan Accounts and Import Loan Accounts Transactions option(s) under Import Bulkdata

Similarly user should be able to import Centers, Chart of Accounts, Offices, Staff and Users by providing relevant options in the Import Bulkdata.

REST API(s)

•  http://serveraddress:port/fineract-provider/api/vi/bulkimport/{entityType}

  • Method: GET 

The above API will return template excel file for the defined entityType and supported entity types are clients, groups,  centers , users, staff, offices, savings,  savingtransactions, loans,  loanrepayments. User has to save this excel, fill the corresponding data according to the template format and then upload this excel file by using following API.

•  http://serveraddress:port/fineract-provider/api/vi/bulkimport/{entityType}

  • Method: POST

By using above API users an import the data related to defined entity type

High level architecture

Bulkdata Import

Implementation:

BulkImportApiResource is responsible to expose the REST API(s). By using BulkImportReadPlatformService implementation, it can return the template excel to the caller and by using BulkImportWritePlatformService implementation it will save the data into DB

org.apache.fineract.infrastructure.bulkimport.api;
@Path("/bulkimport/{entityType}")
@Component
@Scope("singleton")
public class BulkImportApiResource {
	@GET
    @Consumes({ MediaType.APPLICATION_JSON })
    @Produces({ MediaType.APPLICATION_OCTET_STREAM })
	public Response getTemplate(@PathParam("entityType") final String entityType) ;
	
	@POST
    @Consumes({ MediaType.MULTIPART_FORM_DATA })
    @Produces({ MediaType.APPLICATION_OCTET_STREAM })
    public Response importFromTemplate(@PathParam("entityType") final String entityType, 
                                   @FormDataParam("file") final InputStream uploadedInputStream,
                                   @FormDataParam("file") final FormDataContentDisposition fileDetails, 
                                   @FormDataParam("file") final FormDataBodyPart bodyPart) ;
}
 

The implementation of BulkImportWritePlatformService is responsible to load excel data based on entityType and should save the data into DB.

package org.apache.fineract.infrastructure.bulkimport.service;

import java.io.InputStream;
import javax.ws.rs.core.Response;

public interface BulkImportWritePlatformService {
    public Response importTemplateData(final String entityType, final InputStream uploadedInputStream);
}

BulkImportReadPlatformService implementation is responsible to construct the template excel based on entityType. The implementation of this interface can have its own methods to do the actual job.

package org.apache.fineract.infrastructure.bulkimport.service;
import javax.ws.rs.core.Response;

public interface BulkImportReadPlatformService {

    public Response getTemplate(final String entityType);
}

For every entity type, we have different template generators which should be implementing from the following interface. These populator services responsible for actual template excel creation and parsing and saving the template excel while user is trying to import entity data

package org.apache.fineract.infrastructure.bulkimport.service;

import org.apache.fineract.infrastructure.bulkimport.data.Result;

import org.apache.poi.ss.usermodel.Workbook;

public interface WorkBookPopulatorService {

    public Result generateTemplate(final Workbook workbook) ;
	public Result saveTemplate(final Workbook workbook) ;
}

package org.apache.fineract.infrastructure.bulkimport.data;

import java.util.ArrayList;
import java.util.List;

public class Result {
   private List<String> errors = new ArrayList<>();
 
   public void addError(String message) {
        errors.add(message);
    }
    public List<String> getErrors() {
        return errors;
    }
    public boolean isSuccess() {
        return errors.isEmpty();
    }
}

Security

Separate permissions should be defined to access this functionality. The user who doesn't have this permission should not be allowed to import the data. 

Points to remember

• The imported excel file should be saved into the platform by defining a new table. On any error, it should update the status cell in the same row. Once completion of the import, the file should be saved in Amazon S3/local system and the same file should be returned to the client as status
  
  

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing