现象

zabbix告警服务器存在僵尸进程

排查步骤

1、登陆服务器使用top命令，确认下僵尸进程数量。

2、使用ps -aux ｜ grep Z命令，过滤出僵尸进程PID。

3、根据查到的PID，借助ps命令，逐级查找父进程。

4、借助pstree命令，查找该父进程下所有子进程情况，并且和步骤2中的僵尸PID对比，确认吻合。

[root@prod-master2 ~]# pstree -p 9324 -c
containerd-shim(9324)─┬─agent(31058)─┬─git-remote-http(1974)
                      │              ├─git-remote-http(3241)
                      │              ├─git-remote-http(6837)
                      │              ├─git-remote-http(8297)
                      │              ├─git-remote-http(9137)
                      │              ├─git-remote-http(12016)
                      │              ├─git-remote-http(12133)
                      │              ├─git-remote-http(13406)
                      │              ├─git-remote-http(16526)
                      │              ├─git-remote-http(18119)
                      │              ├─git-remote-http(18253)
                      │              ├─git-remote-http(22260)
                      │              ├─git-remote-http(22561)
                      │              ├─git-remote-http(24067)
                      │              ├─git-remote-http(24146)
                      │              ├─git-remote-http(24183)
                      │              ├─git-remote-http(24326)
                      │              ├─git-remote-http(25585)
                      │              ├─git-remote-http(26807)
                      │              ├─git-remote-http(26912)
                      │              ├─git-remote-http(27199)
                      │              ├─git-remote-http(27284)
                      │              ├─git-remote-http(30193)
                      │              ├─git-remote-http(30320)
                      │              ├─git-remote-http(31092)
                      │              ├─{agent}(31136)
                      │              ├─{agent}(31137)
                      │              ├─{agent}(31138)

确认了父进程PID之后，进一步确认该进程对应的进程名，借助ps -ef命令

[root@prod-master2 ~]# ps -ef | grep 9324
root      9324     1  0  2021 ?        07:11:07 /var/lib/rancher/k3s/data/a6857be08414815b83ca6b960373efd98879a0b286fb24cb62b1c5fdbf3a8cb5/bin/containerd-shim-runc-v2 -namespace k8s.io -id fed3749e162f2b4efd44175fb2930873788f4f484f0a749cffb591ccf5ba28db -address /run/k3s/containerd/containerd.sock

从查询结果可以看到当前进程为容器进程，已知当前容器为，根据该pod id：“fed3749e162f2b4efd44175fb2930873788f4f484f0a749cffb591ccf5ba28db”，结合ctrictl ps命令，可以定位到具体的容器

[root@prod-master2 ~]# crictl ps | grep fed3749e1
68154f8ee3507       46177de0d5d32       3 weeks ago         Running             cluster-register            3356                fed3749e162f2

根据容器信息进一步定位具体的pod

[root@prod-master2 ~]# kubectl describe pods -n cattle-system   cattle-cluster-agent-59c76f7858-cmbfb | grep 68154f8ee3507
    Container ID:   containerd://68154f8ee350711173d41cd2b0aec3f9be3a5f47136c63bab1b84e03748f0ea5

可以看到是 cattle-cluster-agent pod导致产生的僵尸进程。查找了相关资料，说是rancher的bug，说在2.5.x的版本中已经修复，但是当前环境rancher版本已经是2.5.5，不知道为何还是出现该问题。并且短期内不会再操作升级rancher版本，考虑到截止到目前产生的僵尸进程数量不多，所以就决定使用临时解决方式处理。

处理方式

针对僵尸进程的解决方法通常是：找到父进程然后重启该进程释放僵尸进程，以上僵尸进程的父进程是：cattle-cluster-agent，接入rancher集群的agent进程。

临时解决方案为：scale the cattle-cluster-agent deployment down and then back up。

sudo k3s kubectl scale deployment -n cattle-system cattle-cluster-agent --replicas=0
sudo k3s kubectl scale deployment -n cattle-system cattle-cluster-agent --replicas=1

参考文档

https://github.com/rancher/rancher/issues/30172