Hadoop配置LZO压缩

Hadoop支持LZO

0. 环境准备

maven（下载安装，配置环境变量，修改sitting.xml加阿里云镜像）

gcc-c++

zlib-devel

autoconf

automake

libtool

通过yum安装即可，yum -y install gcc-c++ lzo-devel zlib-devel autoconf automake libtool

1. 下载、安装并编译LZO

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz

tar -zxvf lzo-2.10.tar.gz

cd lzo-2.10

./configure -prefix=/usr/local/hadoop/lzo/

make -j 20

make install

2. 编译hadoop-lzo源码

2.1 下载hadoop-lzo的源码，下载地址：https://github.com/twitter/hadoop-lzo/archive/master.zip

2.2 解压之后，修改pom.xml

<hadoop.current.version>3.1.3</hadoop.current.version>

2.3 声明两个临时环境变量

export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include

export LIBRARY_PATH=/usr/local/hadoop/lzo/lib

2.4 编译

进入hadoop-lzo-master，执行maven编译命令

mvn package -Dmaven.test.skip=true

2.5 进入target，hadoop-lzo-0.4.21-SNAPSHOT.jar 即编译成功的hadoop-lzo组件

<property>

<name>io.compression.codecs</name>

<value>

org.apache.hadoop.io.compress.GzipCodec,

org.apache.hadoop.io.compress.DefaultCodec,

org.apache.hadoop.io.compress.BZip2Codec,

org.apache.hadoop.io.compress.SnappyCodec,

com.hadoop.compression.lzo.LzoCodec,

com.hadoop.compression.lzo.LzopCodec

</value>

</property>

<property>

<name>io.compression.codec.lzo.class</name>

<value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

[root@emr1 opt]# pdfs dfs -mkdir /input

[root@emr1 opt]# hdfs dfs -put test.txt /input

[root@emr1 opt]# hadoop jar /opt/dtstack/Hadoop/hadoop_pkg/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar wordcount -Dmapreduce.output.fileoutputformat.compress=true -Dmapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec  /input/test.txt /output