kubelet.service:服务延期超时,计划重新启动

时间:2018-12-14 09:32:22

标签: kubernetes upgrade kubectl kubeadm kubelet

上下文

我们目前正在使用一些带有v1.8.7的集群(该集群是几个月前由当前不可用的开发人员创建的),并正在尝试升级到更高版本。 但是,我们希望在用于实验和POC的群集上尝试相同的操作。

我们尝试了什么

这样做,我们尝试在一个主节点上运行一些kubeadm命令,但未找到kubeadm。

因此,我们尝试使用命令安装相同的内容-

apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

发生了什么

但是,现在该节点的状态为“未就绪”并且kubelet服务出现故障

有关如何解决此问题以及我们应该做什么的任何指示?

root@k8s-master-dev-0:/home/azureuser# kubectl get nodes
NAME                     STATUS     ROLES     AGE       VERSION
k8s-master-dev-0         NotReady   master    118d      v1.8.7
k8s-master-dev-1         Ready      master    118d      v1.8.7
k8s-master-dev-2         Ready      master    163d      v1.8.7
k8s-agents-dev-0         Ready      agent     163d      v1.8.7
k8s-agents-dev-1         Ready      agent     163d      v1.8.7
k8s-agents-dev-2         Ready      agent     163d      v1.8.7

root@k8s-master-dev-0:/home/azureuser# systemctl status kubelet.service
● kubelet.service - Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: failed (Result: start-limit-hit) since Thu 2018-12-13 14:33:25 UTC; 18h ago

Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Control process exited, code=exited status=2
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: Failed to start Kubelet.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Unit entered failed state.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: Stopped Kubelet.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Start request repeated too quickly.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: Failed to start Kubelet.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Unit entered failed state.
Dec 13 14:33:25 k8s-master-dev-0 systemd[1]: kubelet.service: Failed with result 'start-limit-hit'.

3 个答案:

答案 0 :(得分:1)

您的kubelet进入不良状态的原因是,您必须更新kubelet的升级后的kubelet软件包和服务文件,如果您之前做过某些更改,则必须将其丢失。

您可以尝试以下操作:

  1. 禁用交换内存:swapoff -a
  2. 检查您的kubelet服务文件,找到kubeadm,文件位于/etc/systemd/system/kubelet.service.d/10-kubeadm.conf,并检查值--cgroup-driver,如果值为systemd,则将其命名为cgroupfs,然后:

重新加载守护程序并重新启动kubelet:

systemctl daemon-reload
systemctl restart kubelet

现在检查您的kubelet是否启动。

PS:应该仔细进行kubeadm控制平面的实时升级,请检查我有关如何升级kubeadm的答案

how to upgrade kubernetes from v1.10.0 to v1.10.11

答案 1 :(得分:0)

它是干净的Kubernetes集群吗?

我认为您应该在 LIVE Kubernetes集群中安装kubelet kubeadm kubectl时要小心。

在这里您可以找到有关在活动集群上安装kubelet的更多信息。 https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

您能告诉我您的输出吗?

kubectl get all --namespace kube-system

答案 2 :(得分:0)

@wrogrammer

root@k8s-master-dev-0:/var/log/apt# kubectl get all --namespace kube-system
NAME            DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
ds/kube-proxy   6         6         5         6            5           beta.kubernetes.io/os=linux   164d

NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/heapster               1         1         1            1           164d
deploy/kube-dns-v20           2         2         2            2           164d
deploy/kubernetes-dashboard   1         1         1            1           164d
deploy/tiller-deploy          1         1         1            1           164d

NAME                                 DESIRED   CURRENT   READY     AGE
rs/heapster-75f8df9884               1         1         1         164d
rs/heapster-7d6ffbf65                0         0         0         164d
rs/kube-dns-v20-5d9fdc7448           2         2         2         164d
rs/kubernetes-dashboard-8555bd85db   1         1         1         164d
rs/tiller-deploy-6677dc8d46          1         1         1         163d
rs/tiller-deploy-86d6cf59b           0         0         0         164d

NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/heapster               1         1         1            1           164d
deploy/kube-dns-v20           2         2         2            2           164d
deploy/kubernetes-dashboard   1         1         1            1           164d
deploy/tiller-deploy          1         1         1            1           164d

NAME            DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
ds/kube-proxy   6         6         5         6            5           beta.kubernetes.io/os=linux   164d

NAME                                 DESIRED   CURRENT   READY     AGE
rs/heapster-75f8df9884               1         1         1         164d
rs/heapster-7d6ffbf65                0         0         0         164d
rs/kube-dns-v20-5d9fdc7448           2         2         2         164d
rs/kubernetes-dashboard-8555bd85db   1         1         1         164d
rs/tiller-deploy-6677dc8d46          1         1         1         163d
rs/tiller-deploy-86d6cf59b           0         0         0         164d

NAME                                          READY     STATUS     RESTARTS   AGE
po/heapster-75f8df9884-nxn2z                  2/2       Running    0          37d
po/kube-addon-manager-k8s-master-dev-0        1/1       Unknown    4          30d
po/kube-addon-manager-k8s-master-dev-1        1/1       Running    4          118d
po/kube-addon-manager-k8s-master-dev-2        1/1       Running    2          164d
po/kube-apiserver-k8s-master-dev-0            1/1       Unknown    4          30d
po/kube-apiserver-k8s-master-dev-1            1/1       Running    4          118d
po/kube-apiserver-k8s-master-dev-2            1/1       Running    2          164d
po/kube-controller-manager-k8s-master-dev-0   1/1       Unknown    6          30d
po/kube-controller-manager-k8s-master-dev-1   1/1       Running    4          118d
po/kube-controller-manager-k8s-master-dev-2   1/1       Running    4          164d
po/kube-dns-v20-5d9fdc7448-smf9s              3/3       Running    0          37d
po/kube-dns-v20-5d9fdc7448-vtjh4              3/3       Running    0          37d
po/kube-proxy-cklcx                           1/1       Running    1          118d
po/kube-proxy-dldnd                           1/1       Running    4          164d
po/kube-proxy-gg89s                           1/1       NodeLost   3          163d

po/kube-proxy-mrkqf                           1/1       Running    4          143d
po/kube-proxy-s95mm                           1/1       Running    10         164d
po/kube-proxy-zxnb7                           1/1       Running    2          164d
po/kube-scheduler-k8s-master-dev-0            1/1       Unknown    6          30d
po/kube-scheduler-k8s-master-dev-1            1/1       Running    6          118d
po/kube-scheduler-k8s-master-dev-2            1/1       Running    4          164d
po/kubernetes-dashboard-8555bd85db-4txtm      1/1       Running    0          37d
po/tiller-deploy-6677dc8d46-5n5cp             1/1       Running    0          37d

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
svc/heapster               ClusterIP   XX Redacted XX   <none>        80/TCP          164d
svc/kube-dns               ClusterIP   XX Redacted XX   <none>        53/UDP,53/TCP   164d
svc/kubernetes-dashboard   NodePort    XX Redacted XX   <none>        80:31279/TCP    164d
svc/tiller-deploy          ClusterIP   XX Redacted XX   <none>        44134/TCP       164d

相关问题