This story starts when I noticed that nodes where going into a ‘NotReady’ status in a cluster.  One ‘NotReady’ node was drained and rebooted only for another node to later go ‘NotReady’ as seen in the output when running kubectl get nodes.

This became a juggling act with other issues being experienced along with nodes going into ‘NotReady’ state. One example was pods stopped responding via HTTP on nodes flapping between ‘Ready’ and ‘NotReady’ and slow node kube-kubelet responses could be seen.

Let’s Begin

I focused on a node that was going into the ‘NotReady’ state then I jumped on that node via SSH and right off the bat I noticed that connecting to SSH was very slow or hanging. I was able to connect the SSH session after the node was drained of all pods with command kubectl drain <IP>. This made the node instantly go into a ‘Ready’ state again.

My thought here is that we have a resource exhaustion type failure because we see failures in everything from hosted apps, control plane components and SSH this day. This idea was then reinforced once another node then became ‘NotReady’ soon after my target node was drained.

Was a pod drained from this node causing issues now on another node after being rescheduled?

Troubleshooting The Kubernetes Cluster

I started by getting the overall cluster resources by running kubectl top node as seen in tmux pane labeled 3 in the image above. Here you see the ‘NotReady’ node is using 93% memory, so lets focus in on this node.

Note: kubectl top node/pods doesn’t work unless you have added metric-server to the cluster beforehand. If its not installed then just jump ahead.

I logged into the now ‘NotReady’ node and ran ‘top’  then I pressed ‘t’, ‘l’, and ‘m‘ in order to display human readable graphs where I found out that memory on the machine was pegged out by the Prometheus process on memory. See my screenshoot in pane labeled 2.

In pane number 1 I ran docker stats on the node to see the exact container that was causing issues. This is proving even more that the container is pegged out. For extra credit I run free -g to see the free memory which is at 0 in pane number 4.

My checks here are overkill, but I really just want you to know that you can do a lot of different things to troubleshoot this.

The Prometheus pod is what caused the node to respond with the ‘NotReady’ status due to resource exhaustion on that node.

To fix this a node node affinity and toleration was configured so just the Prometheus pod used it without other pods which solved the issue for the day.

Later we split the Prometheus deployments in 2 with each on their own node . They could now sit through out the cluster spreading the usage across the clusters.