In the early days of running Kubernetes on premise we would get a user who would complain about ‘intermittent kubectl errors’. Typically this will be a Dev busy deploying his wares for the day.

We found out later that during this time the scripted bad Auth attempts of another developers automation would overrun the API process. This would cause the API process to randomly fail each time on a random master where kube-apiserver was hosted.

Depending on your API server failure you could get a “Error from the server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request” for a overloaded Api server , but you could get an error about escalation failure with something like kubectl get pods -n=namespace even though you have escalated the user. This is just to name a few I have seen in the wild.

Here I will give you a few techniques and shell scripts that I used for pin pointing this intermittent API issue.

More about the process responsible

In my example cluster here there is a load balancer that sits in front of a number of master nodes which are each running the API Server process or ‘kube-apiserver‘. See the diagram below.

If just one of the clusters master nodes kube-apiserver process out of the three goes in to a bad state then the failures are happening intermittently as seen from the dev because the load balancer (In Blue ) which the dev is sending traffic through takes a turn sending requests to each master.

Each master getting a turn to respond in this fashion is a mode called ’round robin’ which just means each master (in red on the diagram) hosting the kube-apiserver service behind the the load balancer gets a turn in responding and one is broken so the failure will seem to come and go to the dev.

If you want to reproduce this and the exact same issue is happenng to your cluster then just run the following command on Mac/Linux.

watch ‘kubectl get pods -A‘

Every 2 seconds you will see the response or error. If there are 3 masters and 1 out of 3 requests returns an error then one master is causing you trouble.

Two Different Ways To Fail. Check Both Of Them.

Keep in mind depending on permissions using ‘get pods‘ can end up acting wacky even though ‘get nodes‘ returns fine depending on your cluster. Make sure you are testing for timeouts and escalation errors by using the right command kubectl get pods in our case.

Pin Pointing The Bad Kubernetes Api Server

Using the kubectl command line tool you can actually pinpoint a master nodes API Server endpoint behind the load balancers IP by using the –server option with kubectl and inputing the master nodes IP address. Here is an example.

kubectl get pods –all-namespaces –server=https://10.2.2.3:6443

Looking at the diagram above, this means that instead of testing the load balancers IP of 192.128.2.2 that is set in your kube config you would check the masters IPs of 10.2.2.3, 10.2.2.8, and 10.2.2.10 one by one. Which ever masters fail this test with the error you see at the LB IP must be looked at closer.

Of course, we work in tech so nothing you do more then once should be done by hand. This is where my script comes in.

#!/bin/bash

if [ "$1" != "" ]; then
echo "You wanna search for masters in host file  $1"
else
echo "use: ./api_tester.sh hosts.txt"
echo "------------------------------------------------------------------------------------------"
echo "This will parse out the masters IPs and then test each master IP kube-apiserver process individually " 
echo "If the output gets stuck on one node and then times out then jump into that machine and run 'sudo systemctl restart kube-apiserver'"
echo "------------------------------------------------------------------------------------------"

fi

IP=$1
while true;do
while IFS= read -r line;
    do for ip in $line
        do
        echo
        date
        echo "working on $ip"
        echo "**Showing just 5 Pods Only**"
        time kubectl get pods --all-namespaces --server=https://$ip:6443 | tail -n 5
    done
    sleep 2s
done<$IP
done

For using this script you can just make a text file of master IPs with a IP on each line and feed them into the script.

If you don’t have a source of truth then using the script above create a host.txt file with the masters IP and run the following

./api_tester.sh hosts.txt

Don’t Compile Master Lists if you have a SOT like Vaquero. Read ahead.

Use this next script to avoid feeding in a IP list by hand if you use a SOT yaml like Vaquero IPXE configs.

Just use a single command with the configs path to parse out the 3 IPs at the top of the file. These should be your masters.

In the image below you can see a sample of a few pods being returned from the API Server testing script and the exact master that fails to return a response.

From here you can go and SSH to the master that is failing and check the logs with ‘sudo journalctl -u kube-apiserver‘ , you can look for resource exhaustion with top, check the access logs for a Dev script DOS issue, check that supporting processes are running or any number of things. After you grab what you need a simple sudo systemctl restart kube-apiserver will restart the service if its needed.