Hadoop3.2.4纠删码基础操作（二）
1、纠删码基础操作
纠删码策略是与具体的路径（path）相关联的。也就是说，如果我们要使用纠删码，则要给一个具体的路径设置纠删码策略，后续，所有往此目录下存储的文件，都会执行此策略。
1.首先我们在HDFS中建立一个目录。
2.使用命令获取该目录的纠删码策略，提示未指定，说明新建目录不会被默认增加纠删码策略属性。
[root@hd2 hadoop]# hdfs ec -getPolicy -path /ec_xor
3.我们先启用XOR-2-1-1024k策略。
[root@hd2 hadoop]# hdfs ec -enablePolicy -policy XOR-2-1-1024k SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.4/share/hadoop/common/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] Erasure coding policy XOR-2-1-1024k is enabled
4.给目录设置纠删码策略
[root@hd2 hadoop]# hdfs ec -setPolicy -path /ec_xor -policy XOR-2-1-1024k SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.4/share/hadoop/common/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] Set XOR-2-1-1024k erasure coding policy on /ec_xor [root@hd2 hadoop]#
[root@hd2 hadoop]# hdfs ec -getPolicy -path /ec_xor
5.这里再测试一下给目录设置另外一些没启用的纠删码策略
[root@hd2 hadoop]# hdfs ec -setPolicy -path /ec_xor -policy RS-10-4-1024k SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.4/share/hadoop/common/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] RemoteException: Policy 'RS-10-4-1024k' does not match any enabled erasure coding policies: [RS-3-2-1024k, XOR-2-1-1024k]. An erasure coding policy can be enabled by enableErasureCodingPolicy API.
报错，说明在使用某个纠删码策略是，需要先启用。
6.我们启用策略RS-6-3-1024k，然后再给/ec_xor目录set策略。同时往这个目录写入数据。
[root@hd2 hadoop]# hdfs ec -enablePolicy -policy RS-6-3-1024k SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.4/share/hadoop/common/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] Erasure coding policy RS-6-3-1024k is enabled
[root@hd2 hadoop]# hdfs ec -setPolicy -path /ec_xor -policy RS-6-3-1024k SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.4/share/hadoop/common/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] Set RS-6-3-1024k erasure coding policy on /ec_xor
[root@hd2 mnt]# hadoop fs -put ec_test.txt /ec_xor SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.4/share/hadoop/common/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/apache-tez-0.10.2-bin/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] put: File /ec_xor/ec_test.txt._COPYING_ could only be written to 3 of the 6 required nodes for RS-6-3-1024k. There are 3 datanode(s) running and 3 node(s) are excluded in this operation. [root@hd2 mnt]#
发现启用纠删码策略，设置目录策略都可以成功，但是在往目录丢数据的时候会报错，这是因为RS（6,3）策略需要6份原始数据，3份校验校验，至少需要9台DataNode，但是测试环境只有3个数据节点。所以接下来会基于XOR-2-1-1024k策略来进行实操，因为XOR(2,1)只需要2份原始数据，1份校验数据，3台DataNode可以进行操作。
