Kubelet failure to start. Docker daemon logs "Error starting daemon: layer does not exist"

Container Linux Version: 1409.5.0

Tectonic/Kubernetes: 1.6.2

The failure seems to be centered around the fact that the docker daemon on the node is failing. There are several error messages like below in docker.service journal:

dockerd[1209]:  level=info msg="[graphdriver] using prior storage driver \"overlay\""
dockerd[1209]:  level=fatal msg="Error starting daemon: layer does not exist"

There are a few very recent issues that seem to explain this behavior. Such as moby/moby#32170 and coreos/bugs#1808. It appears that this can be remedied by clearing out the /var/lib/docker directory.

sudo rm -rf /var/lib/docker

Our OS team has tentatively plans to backport the suggested docker/moby team fix for this issue. I will request that this receive a higher priority.

Action Items:

Stop the kubelet.service

systemctl stop kubelet.service

Clear /var/lib/docker:

sudo rm -rf /var/lib/docker

Check docker.service health:

systemctl status docker.service

Restart the kubelet.service unit:

systemctl start kubelet.service