Seatunel 集群部署
1、基础环境准备
java 1.8 并配置java home
操作系统：centos7.9
下载安装包：https://www.apache.org/dyn/closer.lua/seatunnel/2.3.3/apache-seatunnel-2.3.3-bin.tar.gz
2、安装包部署
集群中的每个节点都按照以下方式进行部署
配置环境变量
echo "export SEATUNNEL_HOME=/opt/seatunnel" >> /etc/profile echo "export PATH=$PATH:$SEATUNNEL_HOME/bin" >> /etc/profile
添加用户
useradd seatunnel
解压安装包，配置软链
tar -xzvf apache-seatunnel-2.3.3-bin.tar.gz -C /opt ln -s /opt/apache-seatunnel-2.3.3 /opt/seatunnel
3、集群配置
配置seatunnel服务。切换到/opt/seatunnel/config目录
编辑调整seatunel.yaml配置文件
vim seatunnel.yaml seatunnel: engine: history-job-expire-minutes: 1440 backup-count: 1 queue-type: blockingqueue print-execution-info-interval: 60 print-job-metrics-info-interval: 60 slot-service: dynamic-slot: true checkpoint: interval: 10000 timeout: 60000 storage: type: hdfs max-retained: 3 plugin-config: namespace: /tmp/seatunnel/checkpoint_snapshot storage.type: hdfs fs.defaultFS: hdfs://mycluster seatunnel.hadoop.dfs.nameservices: mycluster seatunnel.hadoop.dfs.ha.namenodes.mycluster: nn1,nn2 seatunnel.hadoop.dfs.namenode.rpc-address.mycluster.nn2: hadoop002:8020 seatunnel.hadoop.dfs.namenode.rpc-address.mycluster.nn1: hadoop001:8020
checkpoint 配置使用hdfs存储，并通过nameservice进行操作。通过查看core-site.xml和hdfs-site.xml查看nameservice相关配置
配置高可用配置
编辑调整hazelcast.yaml 配置文件
vim hazelcast.yaml hazelcast: cluster-name: seatunnel network: rest-api: enabled: true endpoint-groups: CLUSTER_WRITE: enabled: true DATA: enabled: true join: tcp-ip: enabled: true member-list: - hadoop001 - hadoop002 - hadoop003 port: auto-increment: false port: 5801 properties: hazelcast.invocation.max.retry.count: 20 hazelcast.tcp.join.port.try.count: 30 hazelcast.logging.type: log4j2 hazelcast.operation.generic.thread.count: 50
member-list 集群节点列表
cluster-name: seatunnel 集群名称
4、启动集群
启动集群
#切换到seatunnel用户 su - seatunnel #启动seatunnel cd /opt/seatunnel #创建日志目录 mkdir -p /opt/seatunnel/logs /bin/seatunnel-cluster.sh -d #停止seatunnel ./bin/stop-seatunnel-cluster.sh
检测是否启动成功
ss -tunlp | grep 5801
5、测试服务
配置客户端配置
vim /opt/seatunnel/config/hazelcast-client.yaml hazelcast-client: cluster-name: seatunnel properties: hazelcast.logging.type: log4j2 network: cluster-members: - hadoop001:5801 - hadoop002:5801 - hadoop003:5801
cluster-name 需要和集群配置文件中一致
cluster-members 集群清单需要和集群节点中一致
运行测试命令
cd /opt/seatunnel ./bin/seatunnel.sh --config config/v2.batch.config.template
不出现报错则表示成功
