Table of Contents

Introduction

OODT is a framework that allows distributed storage and cataloging of objects. To achieve its scalability OODT uses the concept of grid computing.

Proposal Idea

Currently OODT uses XMLRPC to enable communication between computers is the grid. "Our version of this library currently sits at 2.0.1, which was released on 28th December 2005." Apache Avro is a more moders data serialization system which also has a RPC implementation. Among the advantages of Avro to XMLRPC are smaller size (serialization can be done to binary or JSON) which means better speed.

Uses of XMLRPC are littered thoughout the OODT code. Sometimes XMLRPC is used in an implementation of an abstract class like XmlRpcCommunicationChannelClient or XmlRpcCommunicationChannelServer. Other times XMLRPC is used directly - like in CrawlDaemon.

So my idea is to first try to abstract the XMLRPC creation and calling logic behind an RPC interface. This will keep the existing functionality the code will have mentions of XMLRPC in as few places as possible.
Then, I will create similar implementations as for XMLRPC that will actually use Avro for object serialization and remote procedure call.
The code will gert the RPC implementation using a factory method for example.

I think abstraction of the method names and RPC client creations should be straightforward because method names are strings. I see difficutlies with this approach if OODT uses some special arguments that are specific to XMLRPC.

Timeline

Community bonding

• Get familiar with the OODT project source code
• Set up a GRID like environment which uses OODT
• Get familiar with the Avro library

Weeks 1-5

Refactor current code to use and abstract RPC interfacte which is implemented using XMLRPC

Weeks 6-12

Implement the RPC interface using Avro RPC

Throughout the implmentation I plan to write tests that exercise new functionality or that are missing for the old functionality.

Deliverables

• Abstractisation of the RPC framework used in OODT
• Implementation of RPC communication using Apache AVRO

Reporting

Week 1

Review of Previous Actions

At the start of bonding period I started to document about the xmlrpc API looking through tutorials for the beginners like: http://www.tutorialspoint.com/xml-rpc/ etc. than looking at more complex data. Like how server/client side works: https://ws.apache.org/xmlrpc/client.html. Read more about avro's rpc, got familiar with the classes through tutorial such as:https://avro.apache.org/docs/1.7.7/gettingstartedjava.html#download_install. Launched filemanager server, tried ingestion of files.

Objectives

Find a separate module to build a prototype for replacing xmlrpc with avrorpc.

Future Actions

Build a prototype for replacing xmlrpc with avrorpc in a standalone module. Establish a path for switching to avrorpc (with the mentor) - try to abstract the places where xmlrpc is used directly and support both technologies side by side, or directly replace xmlrpc with avrorpc.

Mentors Comments

Week 2

Review of Previous Actions

Looked thought the code and located in witch classes xmlrpc is used directly and where its used thought abstraction. Most used classes are WebServer, XmlRpcServer/Client and XmlRpcTransport. Studied the crawlerdeamon class and crawlerdeamonController and tried to build a prototype to replace xmlrpc with avrorpc, without abstraction.

Objectives

• Replace xmlrpc with avrorpc, without abstraction in CrawlerDaemon.

Future Actions

Establish a path for switching to avrorpc (with the mentor) - try to abstract the places where xmlrpc is used directly and support both technologies side by side, or directly replace xmlrpc with avrorpc

Mentors Comments

Week 3

Review of Previous Actions

Started studying the XMLRPC uses inside the project and created XMLRPC uses to document it. Continued work on Replace XMLRPC with Avro inside the crawler module.
Created the following JIRA tasks to track progress:

Objectives

• Finish studying the XMLRPC uses inside the project.
• Replace XMLRPC with Avro inside the crawler module.

Future Actions

Finish studying the XMLRPC uses inside the project. Replace XMLRPC with Avro inside the crawler module. After that create a proposal for the abstraction of XMLRPC/Avro after I get more experience with AvroRPC.

Also, currently XMLRPC is used as for serialization and also for the Server and Client creation. It is not clear how if in Avro this can also be assumes. And if Avro also has its own Server implementation, maybe these two things can be separated.

Mentors Comments

Week 4

Review of Previous Actions

I studied the XMLRPC uses in the project and posted the details in XMLRPC uses.

Objectives

• Replace XMLRPC with Avro inside the crawler module (continued). 

  Jira
  server ASF JIRA key OODT-852

• Come up with a way to abstract the use of XMLRPC in the existing modules based on the results in XMLRPC uses. 

  Jira
  server ASF JIRA key OODT-851

Future Actions

Replace XMLRPC with Avro inside the crawler module. After that create a proposal for the abstraction of XMLRPC/Avro after I get more experience with AvroRPC.

Mentors Comments

Week 5

Review of Previous Actions

• Replace XMLRPC with Avro inside the crawler module (done). 

  Jira
  server ASF JIRA key OODT-852

• Come up with a way to abstract the use of XMLRPC in the existing modules based on the results in XMLRPC uses. 

  Jira
  server ASF JIRA key OODT-851

  
  https://github.com/Cyan3/oodt/commit/a54f05c9111ee8d2bc95e1831a1bcc8fe7073268

Objectives

• Implement AvroRPC with String parameters in the catalog module: 

  Jira
  server ASF JIRA key OODT-855

• separate xmlrpc logic in filemrg module. 

  Jira
  server ASF JIRA key OODT-856

Future Actions

Replace XMLRPC with Avro inside the crawler module. After that create a proposal for the abstraction of XMLRPC/Avro after I get more experience with AvroRPC.

Week 6

Review of Previous Actions

• Implement AvroRPC with String parameters in the catalog module(done)
• separate xmlrpc logic in filemrg module.(done)

Objectives

• Implement AvroRPC logic in filemgr module  
• separate and Implement xmlrpc logic implement AvroRPC logic in commons module

Future Actions

Replace XMLRPC inside filemgr

Week 7

Review of Previous Actions

• Implement AvroRPC logic in filemgr module (partially done). Find some problem with AvroRPC imports for schemas, took some time to get it right.
• Separate and implement XmlRPC logic, implement AvroRPC logic in commons module(not done). I dropped this objective because work on filemgr module is a lot more that I thought.

Objectives

• Modify filemgr module so that he may choose to use avro-ipc or xmlrpc logic for interprocess communication
• Implement AvroTypeFactory for transforming from Avro types and versa. 
• Create tests for AvroStructType.

Future Actions

Replace XMLRPC 

Week 8

Review of Previous Actions

• Modify filemgr module so that he may choose to use avro-ipc or xmlrpc logic for interprocess communication (done)
• Implement AvroRPC logic in filemgr module (partially done). I have a bug that wont let me progress.
• Implement AvroTypeFactory for transforming from Avro types and versa. (partially done). 
• Create tests for AvroTypeFactory class(done)

Objectives

• Polish AvroTypeFactory class.
• Finish Implementation AvroRPC logic in filemgr
  
  

Future Actions

Finish AvroRPC implementation in filemgr

Week 9

Review of Previous Actions

• Finish Implementation AvroRPC logic infile mgr (mostly done). Minor bugs from test.
• Polish AvroStructType(done)

Objectives

• Polish avro rpc implementation in filemgr
• Separate and Implement xmlrpc logic implement AvroRPC logic in workflow module
• Create protocol and avro type schemas for workflow module

Future Actions

Start implementing AvroRPC in workflow module

About me

My name is Radu Manole and I am a student at the Alexandru Ioan Cuza Univerisy University of IASI, Romania. I am very interested in web services and distributed computing (although I am just a beginner) and I would like to expand my knowledge while contributing to an open source project.