迁移Cloudera Manager节点

耀灵11个月前技术文章382


1.概述

1.CDH环境已搭建并正常运行

2.旧Cloudera Manager节点包含Cloudera Manager Server(即cloudera-scm-server)服务和Cloudera Management Service服务(Alert Publisher/Event Server/Host Monitor/Reports Manager/Service Monitor)

2.准备Cloudera Manager新节点

2.1新CM主机前置条件

  1. 操作系统版本与集群操作系统版本一致

  2. 关闭防火墙

  3. 配置时钟同步,根据当前集群时钟同步服务进行配置

  4. swap已设置为1

  5. 关闭透明大页面

  6. 关闭SElinux

  7. 配置/etc/hosts文件

  8. 配置cmosyum

  9. 创建mysql驱动的软链接

2.2新主机信息

新主机IP地址:172.16.121.239

新Hostname:cdh05

1.主机操作系统版本

1.png

2.防火墙

2.png

3.时钟同步

3.png

4.swap信息

4.png

5.透明大页面

5.png

6.SElinux信息

6.png

7.host信息

7.png

8.Cloudera Manager和OS的yum 源

8.png

9.在/usr/share/java目录下创建mysql驱动包软链

9.png

2.3安装Cloudera Manager服务

[root@cdh05 java]# yum  -y install cloudera-manager-server cloudera-manager-agent

10.png

安装完成Cloudera Manager后暂时不要启动服务。

注意:新节点的Cloudera Manager版本必须与原Cloudera Manager版本一致;节点上不要安装CDH的其它组件;

2.4安装MariaDB数据库

由于原CM节点安装有MariaDB数据库,所以在新的CM节点也安装MariaDB数据库进行数据迁移(如果不考虑数据库迁移则可以不用安装

[root@cdh05 java]# yum -y install mariadb-server mariadb-devel

11.png

初始化MariaDB数据库

[root@cdh05 java]# systemctl enable mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[root@cdh05 java]# systemctl start mariadb
[root@cdh05 java]# /usr/bin/mysql_secure_installation

12.png

3.MariaDB数据库迁移

如果不做数据库迁移可不跳过该章节

3.1备份原MariaDB数据

将需要迁移的mysql整库导出(可根据需要导出需要的库信息)

[root@cdh01 mnt]# mysqldump -u root -p -A >oldmysql.dump
Enter password: 
[root@cdh01 mnt]#

13.png

3.2导入备份数据至新库

1.将备份文件拷贝至新mysql服务上,进行数据导入

14.png

新数据库导入数据:

[root@cdh05 mnt]# mysql -u root -p < oldmysql.dump

15.png

16.png

4.将原CM节点数据迁移至新节点

4.1备份原CM节点数据

主要备份CM的监控数据和管理信息,数据目录包括:

/var/lib/cloudera-host-monitor

/var/lib/cloudera-service-monitor

/var/lib/cloudera-scm-server

/var/lib/cloudera-scm-eventserver

/var/lib/cloudera-scm-headlamp

17.png


4.2修改CM的数据库配置信息

修改新CM的数据库配置文件/etc/cloudera-scm-server/db.properties,配置文件内容

"/etc/cloudera-scm-server/db.properties" 22L, 507C                                                                                                                                      1,1           All
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=localhost

# The database name
com.cloudera.cmf.db.name=cm

# The database user
com.cloudera.cmf.db.user=cm

# The database user's password
com.cloudera.cmf.db.password=123456

18.png

4.3CM备份数据导入新节点

将原CM上备份的数据拷贝到新CM节点上

19.png

通过如下命令将备份数据还原至对应目录

tar xf cloudera-host-monitor.tar.gz -C /var/lib/
tar xf cloudera-scm-eventserver.tar.gz -C /var/lib/
tar xf cloudera-scm-headlamp.tar.gz -C /var/lib/
tar xf cloudera-scm-server.tar.gz -C /var/lib/
tar xf cloudera-service-monitor.tar.gz -C /var/lib/


4.4更新集群所有节点的CM Server指向

修改集群所有节点上/etc/cloudera-scm-agent/config.ini文件中server_host值为新CM节点的hostname

1.png

4.5将原CM节点的Cloudera Management Service角色迁移至新节点

启动新CM节点的cloudera-scm-server和cloudera-scm-agent服务

注意:

yum install -y oracle-j2sdk1.7-1.7.0+update67-1.x86_64  ##不装的话,cloudera-scm-server起不来,报错Failed to start LSB: Cloudera SCM Server
systemctl start cloudera-scm-server
systemctl start cloudera-scm-agent

3.png

注意:在新CM节点上启动cloudera-scm-agent服务后,会将CM节点的信息添加到cm库的HOSTS表中,查看新CM节点对应的HOSTS_ID

登录mysql数据库将,查看cm.HOSTS表中的Cloudera Manager的host信息

4.png

未迁移前可以通过CM管理界面看到新CM节点是没有任何角色

5.png


MariaDB [cm]> select NAME,ROLE_TYPE,ROLE_ID from ROLES where HOST_ID=1;
+----------------------------------------------------------+-------------------+---------+
| NAME                                                     | ROLE_TYPE         | ROLE_ID |
+----------------------------------------------------------+-------------------+---------+
| mgmt-HOSTMONITOR-1d0cb071c173f9f51c97dd4377113072        | HOSTMONITOR       |       9 |
| mgmt-EVENTSERVER-1d0cb071c173f9f51c97dd4377113072        | EVENTSERVER       |      10 |
| mgmt-ACTIVITYMONITOR-1d0cb071c173f9f51c97dd4377113072    | ACTIVITYMONITOR   |      11 |
| mgmt-SERVICEMONITOR-1d0cb071c173f9f51c97dd4377113072     | SERVICEMONITOR    |      12 |
| mgmt-ALERTPUBLISHER-1d0cb071c173f9f51c97dd4377113072     | ALERTPUBLISHER    |      13 |
| zookeeper-SERVER-1d0cb071c173f9f51c97dd4377113072        | SERVER            |      15 |
| yarn-JOBHISTORY-1d0cb071c173f9f51c97dd4377113072         | JOBHISTORY        |      34 |
| yarn-RESOURCEMANAGER-1d0cb071c173f9f51c97dd4377113072    | RESOURCEMANAGER   |      38 |
| hive-HIVEMETASTORE-1d0cb071c173f9f51c97dd4377113072      | HIVEMETASTORE     |      54 |
| hive-HIVESERVER2-1d0cb071c173f9f51c97dd4377113072        | HIVESERVER2       |      56 |
| hive-GATEWAY-1d0cb071c173f9f51c97dd4377113072            | GATEWAY           |      60 |
| oozie-OOZIE_SERVER-1d0cb071c173f9f51c97dd4377113072      | OOZIE_SERVER      |      64 |
| impala-CATALOGSERVER-1d0cb071c173f9f51c97dd4377113072    | CATALOGSERVER     |      72 |
| impala-STATESTORE-1d0cb071c173f9f51c97dd4377113072       | STATESTORE        |      76 |
| hbase-MASTER-1d0cb071c173f9f51c97dd4377113072            | MASTER            |      87 |
| hbase-HBASETHRIFTSERVER-1d0cb071c173f9f51c97dd4377113072 | HBASETHRIFTSERVER |      89 |
| hbase-HBASERESTSERVER-1d0cb071c173f9f51c97dd4377113072   | HBASERESTSERVER   |      90 |
+----------------------------------------------------------+-------------------+---------+
17 rows in set (0.00 sec)

MariaDB [cm]>

通过如下命令,将旧CM的角色迁移至新CM节点上

update ROLES set HOST_ID=5 where NAME like 'mgmt%';

首先查询cdh01节点的角色:

7.png


确认cdh05节点没有角色:

8.png


##将HOST_ID为1的cdh01上的CM角色迁移到HOST_ID为5的cdh05节点

9.png

注意:

使用sql迁移角色只针对CM相关的服务,涉及组件的角色迁移使用页面上的 "迁移角色"功能


MariaDB [cm]> select NAME,ROLE_TYPE,ROLE_ID from ROLES where HOST_ID=1;
+----------------------------------------------------------+-------------------+---------+
| NAME                                                     | ROLE_TYPE         | ROLE_ID |
+----------------------------------------------------------+-------------------+---------+
| zookeeper-SERVER-1d0cb071c173f9f51c97dd4377113072        | SERVER            |      15 |
| yarn-JOBHISTORY-1d0cb071c173f9f51c97dd4377113072         | JOBHISTORY        |      34 |
| yarn-RESOURCEMANAGER-1d0cb071c173f9f51c97dd4377113072    | RESOURCEMANAGER   |      38 |
| hive-HIVEMETASTORE-1d0cb071c173f9f51c97dd4377113072      | HIVEMETASTORE     |      54 |
| hive-HIVESERVER2-1d0cb071c173f9f51c97dd4377113072        | HIVESERVER2       |      56 |
| hive-GATEWAY-1d0cb071c173f9f51c97dd4377113072            | GATEWAY           |      60 |
| oozie-OOZIE_SERVER-1d0cb071c173f9f51c97dd4377113072      | OOZIE_SERVER      |      64 |
| impala-CATALOGSERVER-1d0cb071c173f9f51c97dd4377113072    | CATALOGSERVER     |      72 |
| impala-STATESTORE-1d0cb071c173f9f51c97dd4377113072       | STATESTORE        |      76 |
| hbase-MASTER-1d0cb071c173f9f51c97dd4377113072            | MASTER            |      87 |
| hbase-HBASETHRIFTSERVER-1d0cb071c173f9f51c97dd4377113072 | HBASETHRIFTSERVER |      89 |
| hbase-HBASERESTSERVER-1d0cb071c173f9f51c97dd4377113072   | HBASERESTSERVER   |      90 |
+----------------------------------------------------------+-------------------+---------+
12 rows in set (0.00 sec)

MariaDB [cm]> 
MariaDB [cm]> 
MariaDB [cm]> 
MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'zookeeper-SERVER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.43 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> select NAME,ROLE_TYPE,ROLE_ID from ROLES where HOST_ID=1;
+----------------------------------------------------------+-------------------+---------+
| NAME                                                     | ROLE_TYPE         | ROLE_ID |
+----------------------------------------------------------+-------------------+---------+
| yarn-JOBHISTORY-1d0cb071c173f9f51c97dd4377113072         | JOBHISTORY        |      34 |
| yarn-RESOURCEMANAGER-1d0cb071c173f9f51c97dd4377113072    | RESOURCEMANAGER   |      38 |
| hive-HIVEMETASTORE-1d0cb071c173f9f51c97dd4377113072      | HIVEMETASTORE     |      54 |
| hive-HIVESERVER2-1d0cb071c173f9f51c97dd4377113072        | HIVESERVER2       |      56 |
| hive-GATEWAY-1d0cb071c173f9f51c97dd4377113072            | GATEWAY           |      60 |
| oozie-OOZIE_SERVER-1d0cb071c173f9f51c97dd4377113072      | OOZIE_SERVER      |      64 |
| impala-CATALOGSERVER-1d0cb071c173f9f51c97dd4377113072    | CATALOGSERVER     |      72 |
| impala-STATESTORE-1d0cb071c173f9f51c97dd4377113072       | STATESTORE        |      76 |
| hbase-MASTER-1d0cb071c173f9f51c97dd4377113072            | MASTER            |      87 |
| hbase-HBASETHRIFTSERVER-1d0cb071c173f9f51c97dd4377113072 | HBASETHRIFTSERVER |      89 |
| hbase-HBASERESTSERVER-1d0cb071c173f9f51c97dd4377113072   | HBASERESTSERVER   |      90 |
+----------------------------------------------------------+-------------------+---------+
11 rows in set (0.00 sec)

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'yarn-JOBHISTORY-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'yarn-RESOURCEMANAGER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'hive-HIVEMETASTORE-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'hive-HIVESERVER2-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> 
MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'hive-GATEWAY-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'oozie-OOZIE_SERVER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'impala-CATALOGSERVER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'impala-STATESTORE-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'hbase-MASTER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'hbase-HBASETHRIFTSERVER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]> update ROLES set HOST_ID=5 where NAME like 'hbase-HBASERESTSERVER-1d0cb071c173f9f51c97dd4377113072';
Query OK, 1 row affected (0.16 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [cm]>

4.6执行操作后,原CM节点的角色被迁移至新CM节点

10.png


4.7通过CM管理界面将原CM节点从集群删除

11.png

12.png

4.8将原CM节点删除

13.png

通过CM管理界面启动Cloudera Management Service

14.png

由于做了数据库迁移,所以需要修改activity monitor/hive/hue/oozie对应的数据库配置(未做数据库迁移可跳过此步)

hive:

15.png


至此CM迁移完成,CM服务都正常运行




相关文章

CDH实操--集群卸载

CDH实操--集群卸载

1、停止正在运行的服务a、控制台停止集群服务b、控制台停止Cloudera Management Servicec、命令行停止cm服务systemctl stop cloudera-scm-agent...

ElasticSearch开启xpack

ElasticSearch开启xpack

ES开启xpack1、生成ca证书(用户名和密码不用设置,一路回车,生成证书文件elastic-stack-ca.p12,生成kibana证书的时候也需要该ca证书)/opt/dtstack/es-6...

oradebug收集事件10046

oradebug 查看10046这是SQL*Plus的工具,需要提供OSPID或者Oracle PID。--1、跟踪当前会话--设置追踪会话oradebug setmypid; --设置trace大小...

CDH实操--CDH集成flink 1.13.6(二)

CDH实操--CDH集成flink 1.13.6(二)

一、编译flink1、下载flink1.13.6源码包wget https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.13.6/flink...

MySQL DBA 常用工具 SQL

MySQL DBA 常用工具 SQL

【前言】本篇文章介绍一些 MySQL 管理的实用语句及适用的场景。SQL 基于 MySQL 5.7 版本。1. 长事务事务长时间未提交,即使状态为Sleep也可能造成一些锁等待的问题,使用该查询可以查...

CDP实操--配置RangerKMS 并基于Navigator Trustee Server存储KMS密钥

CDP实操--配置RangerKMS 并基于Navigator Trustee Server存储KMS密钥

1.1添加用于部署KMS的服务器到集群从集群host页面里添加两台服务器用于部署rangerkms选择kms-1和kms-2两台服务器等待parcel分发到新加服务器上并自动完成激活 等待host i...

发表评论    

◎欢迎参与讨论,请在这里发表您的看法、交流您的观点。