Useful tools and commands to quickly debug a Kubernetes environment

Debugging a Kubernetes environment can get tedious if you do not know the right commands to quickly deploy the tools you need at the right time. The purpose of this article is to provide some commands, tools, and practices that can help you debug faster in a Kubernetes context.

Looking at Kubernetes error messages

You can start debugging by looking for error messages. In Kubernetes, they will often take the form of a resource called "event" that you can list with the command `kubectl get events`. It provides a centralized view of events in kubernetes like pods entering a “Crashloop Backoff” state or “OOM killed”.

Looking at Kubernetes pods logs

Then you can check Kubernetes pods logs running `kubectl logs <pod-name> -f` but it forces you to put the exact pod name and you can not aggregate logs from several pods.

Stern allows you to aggregate logs of all pods and uses a regular expression to filter them `stern <expression>`

Debug flux opening

You might want to quickly check if a route is opened. To do so, run and ssh in a busybox minimal bare operating system in a single command `kubectl run --generator=run-pod/v1 -i --tty busybox --image=busybox --restart=Never -- sh`. It contains several useful tools for debugging.

Debug services connections

For an http call the same command as above with an image provided with curl will do `kubectl run --generator=run-pod/v1 -i --tty busybox --image=radial/busyboxplus:curl --restart=Never -- sh`

Debug databases connections

You might want to quickly test the connection to a database. Each database has its client and each client works differently but you can always run them quickly. For instance, get a mysql client in your kubernetes cluster with the following command

`kubectl run mysql -it --rm --image=mysql -- mysql  -h <ip> -P <port> -u <user> -p<password>`

Similar commands exist for postgres, sqlserver, oracle ...

Low level debug with kube-debug

Finally, for debugging lower level Linux stuff in a kubernetes pod, you can use kubectl-debug. It will inject a container that will share pid, network, user and ipc with other containers of the pod. Also, adding the `--fork` option allows you to fork pods that are in crashloop backoff so that you can still debug them. It is pretty neat!

 

Here are the tools and commands we use to debug in a Kubernetes environment, tell us yours in the comment section! Also check our other articles on Kubernetes.

Emmanuel Lilette

Emmanuel Lilette

Emmanuel is a Site Reliability Engineer (SRE) at Padok. He is one of our architect specialized in cloud (AWS and GCP) and load testing using Gatling tool.

What do you think? Leave your comments here !