THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
LZO Compression
| Table of Contents |
|---|
General LZO Concepts
LZO is a lossless data compression library that favors speed over compression ratio. See http://www.oberhumer.com/opensource/lzo and http://www.lzop.org for general information about LZO and see Compressed Data Storage for information about compression in Hive.
Imagine a simple data file that has three columns
...
Let's populate a data file containing 4 records:
| No Format |
|---|
19630001 john lennon
19630002 paul mccartney
19630003 george harrison
19630004 ringo starr
|
...
Next we run the command to create an LZO index file:
| No Format |
|---|
hadoop jar /path/to/jar/hadoop-lzo-cdh4-0.4.15-gplextras.jar com.hadoop.compression.lzo.LzoIndexer /path/to/HDFS/dir/containing/lzo/files
|
...
The following hive -e command creates an LZO-compressed external table:
| No Format |
|---|
hive -e "CREATE EXTERNAL TABLE IF NOT EXISTS hive_table_name (column_1 datatype_1......column_N datatype_N)
PARTITIONED BY (partition_col_1 datatype_1 ....col_P datatype_P)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT \"com.hadoop.mapred.DeprecatedLzoTextInputFormat\"
OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\";
|
...
Hive Query Parameters
| No Format |
|---|
SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodecLzoCodec SET hive.exec.compress.output=true SET mapreduce.output.fileoutputformat.compress=true |
For example:
| No Format |
|---|
hive -e "SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodecLzoCodec; SET hive.exec.compress.output=true;SET mapreduce.output.fileoutputformat.compress=true; <query-string>" |
Note: If the data sets are large or number of output files are large , then this option does not work.
Option 2: Write Custom Java to Create LZO Files
...
Prefix the query string with these parameters:
| No Format |
|---|
SET hive.exec.compress.output=false
SET mapreduce.output.fileoutputformat.compress=false
|
For example:
| No Format |
|---|
hive -e "SET hive.exec.compress.output=false;SET mapreduce.output.fileoutputformat.compress=false;<query-string>"
|