Page History

Goals

Allowing Micro Finance Organizations to move from their excel based systems to Apache Fineract by importing their existing portfolio

• Integrating the existing data import tool with Apache Fineract

• Fix all the present bugs in the import tool

• Add new category of imports (Offices, CoA, Users, staff)

• Enhance the import features based on user feedback

...

User interaction and design

Mockup Mock up screens

Under Admin->Organization, there should new menu item to import bulk data. By selecting this menu item the following screen shall be displayed

...

Savings Accounts and their transactions can be imported with the following screens. This screen(s) should be shown by selecting Import Savings Accounts and Import Savings Transactions option(s) under Import Bulkdata

Image Added

Image Added

Loan Accounts and their transactions can be imported with the following screens. This screen(s) should be shown by selecting Import Loan Accounts and Import Loan Accounts Transactions option(s) under Import Bulkdata

Image Added

Image Added

Similarly user should be able to import Centers, Chart of Accounts, Offices, Staff and Users by providing relevant options in the Import Bulkdata.

REST API(s)

•  http://serveraddress:port/fineract-provider/api/vi/bulkimport/{entityType}

  • Method: GET 

The above API will return template excel file for the defined entityType and supported entity types are clients, groups,  centers , users, staff, offices, savings,  savingtransactions, loans,  loanrepayments. User has to save this excel, fill the corresponding data according to the template format and then upload this excel file by using following API.

•  http://serveraddress:port/fineract-provider/api/vi/bulkimport/{entityType}

  • Method: POST

By using above API users an import the data related to defined entity type

High level architecture

Image Added

Bulkdata Import

Image Added

Implementation:

BulkImportApiResource is responsible to expose the REST API(s). By using BulkImportReadPlatformService implementation, it can return the template excel to the caller and by using BulkImportWritePlatformService implementation it will save the data into DB

org.apache.fineract.infrastructure.bulkimport.api;
@Path("/bulkimport/{entityType}")
@Component
@Scope("singleton")
public class BulkImportApiResource {
	@GET
    @Consumes({ MediaType.APPLICATION_JSON })
    @Produces({ MediaType.APPLICATION_OCTET_STREAM })
	public Response getTemplate(@PathParam("entityType") final String entityType) ;
	
	@POST
    @Consumes({ MediaType.MULTIPART_FORM_DATA })
    @Produces({ MediaType.APPLICATION_OCTET_STREAM })
    public Response importFromTemplate(@PathParam("entityType") final String entityType, 
                                   @FormDataParam("file") final InputStream uploadedInputStream,
                                   @FormDataParam("file") final FormDataContentDisposition fileDetails, 
                                   @FormDataParam("file") final FormDataBodyPart bodyPart) ;
}
 

The implementation of BulkImportWritePlatformService is responsible to load excel data based on entityType and should save the data into DB.

package org.apache.fineract.infrastructure.bulkimport.service;

import java.io.InputStream;
import javax.ws.rs.core.Response;

public interface BulkImportWritePlatformService {
    public Response importTemplateData(final String entityType, final InputStream uploadedInputStream);
}

BulkImportReadPlatformService implementation is responsible to construct the template excel based on entityType. The implementation of this interface can have its own methods to do the actual job.

package org.apache.fineract.infrastructure.bulkimport.service;
import javax.ws.rs.core.Response;

public interface BulkImportReadPlatformService {

    public Response getTemplate(final String entityType);
}

For every entity type, we have different template generators which should be implementing from the following interface. These populator services responsible for actual template excel creation and parsing and saving the template excel while user is trying to import entity data

package org.apache.fineract.infrastructure.bulkimport.service;

import org.apache.fineract.infrastructure.bulkimport.data.Result;

import org.apache.poi.ss.usermodel.Workbook;

public interface WorkBookPopulatorService {

    public Result generateTemplate(final Workbook workbook) ;
	public Result saveTemplate(final Workbook workbook) ;
}

package org.apache.fineract.infrastructure.bulkimport.data;

import java.util.ArrayList;
import java.util.List;

public class Result {
   private List<String> errors = new ArrayList<>();
 
   public void addError(String message) {
        errors.add(message);
    }
    public List<String> getErrors() {
        return errors;
    }
    public boolean isSuccess() {
        return errors.isEmpty();
    }
}

Security

Separate permissions should be defined to access this functionality. The user who doesn't have this permission should not be allowed to import the data. 

Points to remember

• The imported excel file should be saved into the platform by defining a new table. On any error, it should update the status cell in the same row. Once completion of the import, the file should be saved in Amazon S3/local system and the same file should be returned to the client as status
  
  

Questions

Below is a list of questions to be addressed as a result of this requirements document:

...