Recover from scheduler failure

It is possible that the schedulers themselves can get into a state that will not allow them to be scheduled onto the kubernetes cluster. If this happens you can recover the scheduler manually.

The following script expects that you have configured kubectl to point at the cluster where the schedulers are in the unschedulable state.

You should be able to see:

$ kubectl get po --namespace=kube-system -l k8s-app=kube-scheduler
NAME READY STATUS RESTARTS AGE
kube-scheduler-3027616201-38xmk 1/1 Running 0 7d
kube-scheduler-3027616201-947lc 1/1 Running 0 7d
kube-scheduler-3027616201-q4knz 1/1 Running 0 7d

$ kubectl get po --namespace=kube-system -l k8s-app=kube-controller-manager
NAME READY STATUS RESTARTS AGE
kube-controller-manager-3114643686-4qfjs 1/1 Running 0 7d
kube-controller-manager-3114643686-529ds 1/1 Running 0 7d
kube-controller-manager-3114643686-n0bwv 1/1 Running 0 7d

Further, it expects that the api servers and the etcd cluster are in a healthy state.

The script will grab the content of the deployment known in etcd for the kube-scheduler. It will then create a pod definition based on that. We also derive from your cluster the first node that has the master label set. The output is then sent to kubectl convert for validation and to convert it to yaml.

The resulting file will be output to current working directory and it will be called kube-scheduler-rescue.yaml

At this point you can use kubectl to create the rescue scheduler pod.

kubectl create -f kube-scheduler-rescue.yaml

You should see the schedulers and controller-manager pods startup.

rescue-pod.sh