Versions Compared

Old Version 2

changes.mady.by.user Lefty Leverenz

Saved on

compared with

New Version 3

changes.mady.by.user Unknown User (sanjaysubramanian@yahoo.com)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added note "If the data sets are large or number of output files are large , then the option of outputting LZO files directly from Hive query does not work"

...

No Format
hive -e "CREATE EXTERNAL TABLE IF NOT EXISTS hive_table_name (column_1  datatype_1......column_N datatype_N) 
         PARTITIONED BY (partition_col_1 datatype_1 ....col_P  datatype_P) 
         ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' 
         STORED AS INPUTFORMAT  \"com.hadoop.mapred.DeprecatedLzoTextInputFormat\"   
                   OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\";

...

No Format
hive -e "SET mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec;SET hive.exec.compress.output=true;SET mapreduce.output.fileoutputformat.compress=true;<query-string>"

     Note: If the data sets are large or number of output files are large , then this option does not work.

Option 2: Write Custom Java to Create LZO Files

...