Sqoop2 Intermediate representation

Intermediate representation

In sqoop2 connectors will supply their own map phase that will import data into HDFS. Because this piece of code will be fully under connector maintenance, we need to agree on common intermediate (map output) form for all connectors and all cases. This page goal is to do comparison of different intermediate representation, so that we can pick up the appropriate one for sqoop 2.

Goals 

• Simple
• Fast (no necessary parsing, encoding, ...)

Ideas

List of ideas that we've explored.

mysqldump format 

Comma separated list of values present in one single Text instance. Strings and binary values are wrapped with simple quotation. For example:

0,'Hello world','Jarcec\'s notes'

Inside string and binary fields all bytes are printed directly with following exceptions when one byte is encoded into two different bytes:

List<Text> 

avro