Sqoop2 Intermediate representation

Intermediate representation

In sqoop2 connectors will supply their own map phase that will import data into HDFS. Because this piece of code will be fully under connector maintenance, we need to agree on common intermediate (map output) form for all connectors and all cases. This page goal is to do comparison of different intermediate representation, so that we can pick up the appropriate one for sqoop 2.

Goals 

• Simple
• Fast (no necessary parsing, encoding, ...)

Ideas

List of ideas that we've explored.

mysqldump format 

List<Text>