kafka部署
安装前准备
修改环境变量
vi /etc/profile
export KAFKA_HOME=/opt/kafka
export PATH=$PATH:$KAFKA_HOME/bin
创建用户及用户组
groupadd kafka
useradd -g kafka kafka
格式化磁盘并挂载
1. 格式化磁盘
mkfs.xfs /dev/sd{b,c,d,e,f}
2. 创建挂载目录并挂载
mkdir -p /data{1,2,3,4,5}
mount /dev/sdb /data1
mount /dev/sdc /data2
mount /dev/sdd /data3
mount /dev/sde /data4
mount /dev/sdf /data5
mkdir -p /data{1,2,3,4,5}/kafka
3. 写入fstab文件
vi /etc/fatab
/dev/sdb /data{1,2,3,4,5} xfs defaults 0 0
安装kafka
1. 上传安装包并解压
tar -zxf kafka_2.12-2.8.2.tgz -C /opt/
ln -s kafka_2.12-2.8.2 /opt/kafka
2. 复制目录
scp -rp /opt/kafka_2.12-2.8.2 DPS2-kafka-qas2:/opt
scp -rp /opt/kafka_2.12-2.8.2 DPS2-kafka-qas3:/opt
3. 修改权限
mkdir -p /log/kafka
chown -R kafka:kafka /opt/kafka/
for i in {1..5}; do chown -R kafka:kafka /data$i/kafka/;done
chown -R kafka:kafka /log/kafka
配置kafka
修改配置文件
vi /opt/kafka/config/server.properties
broker.id=0 #每个节点的id不一样
listeners = PLAINTEXT://172.16.xxx.xxx:9092 #IP为每个节点的ip
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data1/kafka,/data2/kafka,/data3/kafka,/data4/kafka,/data5/kafka
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=172.16.xxx.xxx:2181,172.16.xxx.xxx:2181,172.16.xxx.xxx:2181/kafka
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
修改脚本
vi /opt/kafka/bin/kafka-run-class.sh(208行)
LOG_DIR="$base_dir/logs" 改为LOG_DIR="/log/kafka"
启动kafka
1. 切换用户
苏- kafka
2. 启动服务
cd /opt/kafka
bin/kafka-server-start.sh -daemon config/server.properties
