CDH-Kafka节点迁移
1、节点迁移
先添加节点，将需要换掉的节点关闭，等待选举leader
启动新添加的节点
关闭旧的节点
等待选举leader
出现离线分区。需要将分区分配到新的节点
2、重分配分区
kafak kerberos环境下使用
参考文档：https://docs.cloudera.com/documentation/enterprise/latest/topics/kafka_security.html
1、如果使用kerberos cache。创建jaas.conf,然后使用kinit进行认证
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};2、如果使用kerberos keytab,则创建jaas.conf
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/mykafkaclient.keytab"
principal="mykafkaclient/clients.hostname.com@EXAMPLE.COM";
};3、创建client.properties
security.protocol=SASL_PLAINTEXT sasl.kerberos.service.name=kafka
4、验证topic是否存在
kafka-topics --list --zookeeper cdp01:2181/kafka
5、使用jaas.conf文件
export KAFKA_OPTS="-Djava.security.auth.login.config=/home/user/jaas.conf"
6、使用jaas.conf进行查看topic
#路径按照实际情况配置 export KAFKA_OPTS="-Djava.security.auth.login.config=/home/user/jaas.conf" kafka-topics --bootstrap-server cdp01:9092,cdp03:9092,cdp05:9092 --list --command-config client.properties
7、生成topic-gen.json
{
"topics": [
{
"topic": "__consumer_offsets"
},
{
"topic": "__smm-app-consumer-metrics-keys-index-changelog"
},
{
"topic": "__smm-app-producer-metrics-keys-index-changelog"
},
{
"topic": "__smm-app-smm-consumer-table-15m-changelog"
},
{
"topic": "__smm-app-smm-consumer-table-15m-repartition"
},
{
"topic": "__smm-app-smm-consumer-table-30s-changelog"
},
{
"topic": "__smm-app-smm-consumer-table-30s-repartition"
},
{
"topic": "__smm-app-smm-producer-table-15m-changelog"
},
{
"topic": "__smm-app-smm-producer-table-15m-repartition"
},
{
"topic": "__smm-app-smm-producer-table-30s-changelog"
},
{
"topic": "__smm-app-smm-producer-table-30s-repartition"
},
{
"topic": "__smm_alert_notifications"
},
{
"topic": "__smm_consumer_metrics"
},
{
"topic": "__smm_producer_metrics"
},
{
"topic": "connect-configs"
},
{
"topic": "connect-offsets"
},
{
"topic": "connect-status"
},
{
"topic": "test_producer_perf"
}
],
"version": 1
}通过--topics-to-move-json-file 和--generate来分配分区策略
kafka-reassign-partitions --zookeeper cdp01:2181/kafka --broker-list "87,215,202" --topics-to-move-json-file topic-gen.json --generate --command-config client.properties
--broker-list 的值是要分配的kafka broker id，中间逗号隔开
会输出当前分配策略和期望分配策略
当前分配策略
期望分配策略
将当前分配策略备份。使用期望分配策略进行分区分配
通过--reassignment-json-file 参数和--execute执行分配策略
kafka-reassign-partitions --zookeeper cdp01:2181/kafka --reassignment-json-file topic_move.json --execute
通过-reassignment-json-file 和--verify 验证分配是否成功
全部成功后，删除旧的kafka broker
重启kafka，恢复正常
