What is the CrashLoopBackOff status?

A CrashLoopBackoff status within Kubernetes just means that a pod isn’t able to start or is failing over and over after Kubernetes restarts it. You will need to make some changes to get your Pod running correctly.

Reasons for the CrashLoopBackoff can include missing environment variables, errors, misconfigured applications, or maybe that you didn’t fire off the app on container start.

If you are here then you are most likely deploying a pod to your Kubernetes cluster and getting the status CrashLoopBackOff in your output when listing pods with the command kubectl get pods -n mynamespace . Your Pod should have a status of ‘running’, so this isn’t the ideal situation.

Let’s get into finding a fix!

If you where to run kubectl run pod –image=busybox without a command you would get a CrashLoopBackOff status. The pod would restart over and over and finally land on that status because there is no command to run.

When we launch our busybox image in the above command we can watch each and every status playout in front of us using a simple command kubectl get pod -n namespace -w . Every change in status will be printed on the screen. Here you can see RunContainerError, a CrashLoopBackOff while pod restarts are happening before they stop. this is typical of the CrashLoopBackOff error.

How do you avoid this? Simply feed it a command with kubectl run crashtest -n crashtestdummy –image=busybox — sleep infinity

When watching output with kubectl get pod -n namespace -w my healthly Pod should print every status to the screen and look just like this.

Ok, But what’s wrong with my Deployment?

Ok , so you just deployed and you are having problems. First things first we want to see if there are any obvious issues with our deployment

Here you could run a kubectl describe pod mypod -n mynamespace to read the events that happened right before your pod went into the CrashLoopBackOff status.

The events will be labeled all the way at the bottom of the output..

Try kubectl get events -n namespace to get all the events in your namespace as an alternative method of reading events.

The above image is an example from a multiple container pod. The last container “mysql” tries to kick off and then we see the event “Back-off restarting failed container” following it. In this case I choose to focus in on that container being that it is the last one mentioned before things go south.

From here I want to drill down into the pod and the container to pull logs. Keep in mind this Pod has multiple containers, so we need to target the right container. In this case since the pod is called mysql-0 and the failing container written in the events output above is called mysql , so your command will look like kubectl logs mysql-0 mysql -n namespace to get log output from the failing container to help shine a light on your errors.

In this case it looks like environment variables are not being set correctly.

What if there where no errors? Make sure that you check the other containers within the Pod to make sure there isn’t some useful errors with kubectl logs pod -n namespace. Many people start here with pods with only one container, but I wanted to go over this in depth for better understanding.

It isn’t always the case that you will get errors. What if the app didn’t have logging or output?

No logs. What else can I try?

Look at my deployment below to see how to add a blocking command like “sleep infinity” in order to freeze your pod from restarting / CrashLoopBackoff to further troubleshoot things before redeploying.

The YAML below is just an example of adding the sleep command to your deployment. Add the needed lines to your deployment.

Keep in mind that your deployments image: setting could be a container that doesn’t include a shell. If you would like to use something like the busybox image posted in the below example you can use a shell for troubleshooting and just switch your image back when done.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tester
  namespace: mynamespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-test-pod
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: my-test-pod
    spec:
      containers:
        - image: busybox:latest
          name: busybox
          command: ["sleep", "infinity"]

Next use the command kubectl exec -it deploy/my-test-pod -n mynamespace — /bin/bash to jump into your pod with an interactive shell.

From here you will check that everything that your app depends on to run is there.

• Run env to check that all of your environment variables are there
• Using curl check that any endpoints you need are accessible inside and outside the cluster.
• Check that your configuration files are where they should be

Is there not curl or other commands inside your striped down container that you need to troubleshoot?

Try to install them , but first make sure to run apt-get update or yum update before installing the utilities with commands like apt-get curl / yum curl .

I hope this helps someone out there figure out how to get there app running . Good luck!