开源大数据集群部署（二十二）Flink on yarn部署
flink on yarn部署（每个节点）
cd /root/bigdata/ tar -xzvf flink-1.14.6-bin-scala_2.12.tgz -C /opt/ ln -s /opt/flink-1.14.6 /opt/flink
修改环境变量添加内容
vi /etc/profile.d/bigdata.sh export HADOOP_CLASSPATH=`hadoop classpath`
引用变量
source /etc/profile
生成keytab
bash /data/kerberos/getkeytabs.sh /etc/security/keytab/flink.keytab flink
3.1.2 配置修改
c
cd /opt/flink cat conf/flink-conf.yaml blob.server.port: 6124 jobmanager.memory.process.size: 1024m jobmanager.rpc.address: localhost jobmanager.rpc.port: 6123 query.server.port: 6125 rest.port: 8081 taskmanager.memory.process.size: 1024m taskmanager.numberOfTaskSlots: 1 taskmanager.rpc.port: 6122 state.backend: rocksdb state.checkpoints.dir: hdfs:///flink/flink/flink-checkpoints state.savepoints.dir: hdfs:///flink/flink/fink-savepoints jobmanager.archive.fs.dir: hdfs:///flink/flink/completed-jobs historyserver.archive.fs.dir: hdfs:///flink/flink/completed-jobs historyserver.archive.fs.refresh-interval: 1000 security.kerberos.login.keytab: /etc/security/keytab/flink.keytab security.kerberos.login.principal: flink/hd1.dtstack.com@DTSTACK.COM high-availability: zookeeper high-availability.zookeeper.quorum:hd1:2181,hd2:2181,hd3:2181 high-availability.zookeeper.path.root: /flink high-availability.cluster-id: /flink-ha high-availability.storageDir: hdfs:///flink/flink/recovery zookeeper.sasl.login-context-name: Client zookeeper.sasl.service-name: zookeeper yarn.application-attempts: 10 classloader.check-leaked-classloader: false
创建目录
hdfs dfs -mkdir -p /flink/flink/completed-jobs hdfs dfs -mkdir -p /flink/flink/flink-checkpoints hdfs dfs -mkdir -p hdfs:///flink/flink/fink-savepoints hdfs dfs -mkdir /user/flink hdfs dfs -chown -R flink /flink/flink/recovery/flink-ha hdfs dfs -chown flink /user/flink
3.1.3 分发安装包
scp /opt/flink-1.14.6 root@hd2:/opt/ scp /opt/flink-1.14.6 root@hd3:/opt/
3.1.4 提交测试任务
Per_job 任务
./bin/flink run -m yarn-cluster -yjm 1024 -ytm 1024 examples/batch/WordCount.jar
提交session 模式
./bin/yarn-session.sh -n 3 -s 3 -nm flink-sessiontest -d -q
./bin/flink run -m hd2.dtstack.com:8081 -p 2 examples/batch/WordCount.jar
访问8081的页面
