LZO Compression
Table of Contents |
---|
General LZO Concepts
LZO is a lossless data compression library that favors speed over compression ratio. See http://www.oberhumer.com/opensource/lzo and http://www.lzop.org for general information.
...
Let's populate a data file containing 4 records:
No Format |
---|
19630001 john lennon
19630002 paul mccartney
19630003 george harrison
19630004 ringo starr
|
...
Next we run the command to create an LZO index file:
No Format |
---|
hadoop jar /path/to/jar/hadoop-lzo-cdh4-0.4.15-gplextras.jar com.hadoop.compression.lzo.LzoIndexer /path/to/HDFS/dir/containing/lzo/files
|
...
The following hive -e
command creates an LZO-compressed external table:
No Format |
---|
hive -e "CREATE EXTERNAL TABLE IF NOT EXISTS hive_table_name (column_1 datatype_1......column_N datatype_N)
PARTITIONED BY (partition_col_1 datatype_1 ....col_P datatype_P)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT \"com.hadoop.mapred.DeprecatedLzoTextInputFormat\"
OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\";
|
...
Hive Query Parameters
No Format |
---|
SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec
SET hive.exec.compress.output=true
SET mapreduce.output.fileoutputformat.compress=true
|
For example:
No Format |
---|
hive -e "SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodecLzoCodec; SET hive.exec.compress.output=true;SET mapreduce.output.fileoutputformat.compress=true; <query-string>" |
Note: If the data sets are large or number of output files are large , then this option does not work.
...
Prefix the query string with these parameters:
No Format |
---|
SET hive.exec.compress.output=false
SET mapreduce.output.fileoutputformat.compress=false
|
For example:
No Format |
---|
hive -e "SET hive.exec.compress.output=false;SET mapreduce.output.fileoutputformat.compress=false;<query-string>"
|