Table of Contents |
---|
Introduction
OODT is a framework that allows distributed storage and cataloging of objects. To achieve its scalability OODT uses the concept of grid computing.
Proposal Idea
Currently OODT uses XMLRPC to enable communication between computers is the grid. "Our version of this library currently sits at 2.0.1, which was released on 28th December 2005." Apache Avro is a more moders data serialization system which also has a RPC implementation. Among the advantages of Avro to XMLRPC are smaller size (serialization can be done to binary or JSON) which means better speed.
Uses of XMLRPC are littered thoughout the OODT code. Sometimes XMLRPC is used in an implementation of an abstract class like XmlRpcCommunicationChannelClient or XmlRpcCommunicationChannelServer. Other times XMLRPC is used directly - like in CrawlDaemon.
So my idea is to first try to abstract the XMLRPC creation and calling logic behind an RPC interface. This will keep the existing functionality the code will have mentions of XMLRPC in as few places as possible.
Then, I will create similar implementations as for XMLRPC that will actually use Avro for object serialization and remote procedure call.
The code will gert the RPC implementation using a factory method for example.
I think abstraction of the method names and RPC client creations should be straightforward because method names are strings. I see difficutlies with this approach if OODT uses some special arguments that are specific to XMLRPC.
Timeline
Community bonding
- Get familiar with the OODT project source code
- Set up a GRID like environment which uses OODT
- Get familiar with the Avro library
Weeks 1-5
Refactor current code to use and abstract RPC interfacte which is implemented using XMLRPC
Weeks 6-12
Implement the RPC interface using Avro RPC
Throughout the implmentation I plan to write tests that exercise new functionality or that are missing for the old functionality.
Deliverables
- Abstractisation of the RPC framework used in OODT
- Implementation of RPC communication using Apache AVRO
Reporting
Week 1
Review of Previous Actions
At the start of bonding period I started to document about the xmlrpc API looking through tutorials for the beginners like: http://www.tutorialspoint.com/xml-rpc/ etc. than looking at more complex data. Like how server/client side works: https://ws.apache.org/xmlrpc/client.html. Read more about avro's rpc, got familiar with the classes through tutorial such as:https://avro.apache.org/docs/1.7.7/gettingstartedjava.html#download_install. Launched filemanager server, tried ingestion of files.
Objectives
Find a separate module to build a prototype for replacing xmlrpc with avrorpc.
Future Actions
Build a prototype for replacing xmlrpc with avrorpc in a standalone module. Establish a path for switching to avrorpc (with the mentor) - try to abstract the places where xmlrpc is used directly and support both technologies side by side, or directly replace xmlrpc with avrorpc.
Mentors Comments
Week 2
Review of Previous Actions
Looked thought the code and located in witch classes xmlrpc is used directly and where its used thought abstraction. Most used classes are WebServer, XmlRpcServer/Client and XmlRpcTransport. Studied the crawlerdeamon class and crawlerdeamonController and tried to build a prototype to replace xmlrpc with avrorpc, without abstraction.
Objectives
- Replace xmlrpc with avrorpc, without abstraction in CrawlerDaemon.
Future Actions
Establish a path for switching to avrorpc (with the mentor) - try to abstract the places where xmlrpc is used directly and support both technologies side by side, or directly replace xmlrpc with avrorpc
Mentors Comments
Week 3
Review of Previous Actions
Started studying the XMLRPC uses inside the project and created XMLRPC uses to document it. Continued work on Replace XMLRPC with Avro inside the crawler module.
Created the following JIRA tasks to track progress:
Jira | ||||
---|---|---|---|---|
|
Jira | ||||
---|---|---|---|---|
|
Objectives
- Finish studying the XMLRPC uses inside the project.
- Replace XMLRPC with Avro inside the crawler module.
Future Actions
Finish studying the XMLRPC uses inside the project. Replace XMLRPC with Avro inside the crawler module. After that create a proposal for the abstraction of XMLRPC/Avro after I get more experience with AvroRPC.
Also, currently XMLRPC is used as for serialization and also for the Server and Client creation. It is not clear how if in Avro this can also be assumes. And if Avro also has its own Server implementation, maybe these two things can be separated.
Mentors Comments
Week 4
Review of Previous Actions
I studied the XMLRPC uses in the project and posted the details in XMLRPC uses.
Objectives
- Replace XMLRPC with Avro inside the crawler module (continued).
Jira server ASF JIRA key OODT-852 - Come up with a way to abstract the use of XMLRPC in the existing modules based on the results in XMLRPC uses.
Jira server ASF JIRA key OODT-851
Future Actions
Replace XMLRPC with Avro inside the crawler module. After that create a proposal for the abstraction of XMLRPC/Avro after I get more experience with AvroRPC.
Mentors Comments
Week 5
Review of Previous Actions
- Replace XMLRPC with Avro inside the crawler module (done).
Jira server ASF JIRA key OODT-852 - Come up with a way to abstract the use of XMLRPC in the existing modules based on the results in XMLRPC uses.
Jira server ASF JIRA key OODT-851
https://github.com/Cyan3/oodt/commit/a54f05c9111ee8d2bc95e1831a1bcc8fe7073268
Objectives
- Implement AvroRPC with String parameters in the catalog module:
Jira server ASF JIRA key OODT-855 - separate xmlrpc logic in filemrg module.
Jira server ASF JIRA key OODT-856
Future Actions
Replace XMLRPC with Avro inside the crawler module. After that create a proposal for the abstraction of XMLRPC/Avro after I get more experience with AvroRPC.
Week 6
Review of Previous Actions
- Implement AvroRPC with String parameters in the catalog module(done)
- separate xmlrpc logic in filemrg module.(done)
Objectives
- Implement AvroRPC logic in filemgr module
- separate and Implement xmlrpc logic implement AvroRPC logic in commons module
Future Actions
Replace XMLRPC inside filemgr
Week 7
Review of Previous Actions
- Implement AvroRPC logic in filemgr module (partially done). Find some problem with AvroRPC imports for schemas, took some time to get it right.
- Separate and implement XmlRPC logic, implement AvroRPC logic in commons module(not done). I dropped this objective because work on filemgr module is a lot more that I thought.
Objectives
- Modify filemgr module so that he may choose to use avro-ipc or xmlrpc logic for interprocess communication
- Implement AvroTypeFactory for transforming from Avro types and versa.
- Create tests for AvroStructType.
Future Actions
Replace XMLRPC
Week 8
Review of Previous Actions
- Modify filemgr module so that he may choose to use avro-ipc or xmlrpc logic for interprocess communication (done)
- Implement AvroRPC logic in filemgr module (partially done). I have a bug that wont let me progress.
- Implement AvroTypeFactory for transforming from Avro types and versa. (partially done).
- Create tests for AvroTypeFactory class(done)
Objectives
- Polish AvroTypeFactory class.
- Finish Implementation AvroRPC logic in filemgr
Future Actions
Finish AvroRPC implementation in filemgr
Week 9
Review of Previous Actions
- Finish Implementation AvroRPC logic infile mgr (mostly done). Minor bugs from test.
- Polish AvroStructType(done)
Objectives
- Polish avro rpc implementation in filemgr
- Separate and Implement xmlrpc logic implement AvroRPC logic in workflow module
- Create protocol and avro type schemas for workflow module
Future Actions
Start implementing AvroRPC in workflow module
Week 10
Review of Previous Actions
- Polish avro rpc implementation in filemgr(done)
- Create protocol and avro type schemas for workflow module (done)
- Separate and Implement xmlrpc logic implement AvroRPC logic in workflow module (working)
Objectives
- Implement AvroRPC logic in workflow module
- Create tests for AvroTypeFactory.
Future Actions
Create tests for AvroTypeFactory, and finish implementation of AvroRPC.
Week 11
Review of Previous Actions
Objectives
Future Actions
Week 12
Review of Previous Actions
Objectives
Future Actions
Week 13
Review of Previous Actions
Objectives
Future Actions
About me
My name is Radu Manole and I am a student at the Alexandru Ioan Cuza Univerisy University of IASI, Romania. I am very interested in web services and distributed computing (although I am just a beginner) and I would like to expand my knowledge while contributing to an open source project.