Kubernetes troubleshooting patterns

Most Kubernetes faults announce themselves the same few ways. Each pattern here names the signals you will see, the order to check things in, and the lessons and break-fix drills where you practise it on a cluster.

Node NotReady: read the conditions, then the kubelet

When to use it: A node shows NotReady or Unknown, and the pods on it stop answering or are being replaced elsewhere.

Signals

  • kubectl get nodes shows NotReady, or Unknown when the kubelet has stopped reporting at all
  • The node carries the node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taint
  • Pods on the node are evicted once their toleration for that taint runs out

Steps

  1. Run kubectl describe node and read the Conditions table: which condition changed, its reason, and when the last heartbeat arrived.
  2. Cordon the node so nothing new is scheduled onto it while you work.
  3. On the node, check the kubelet with systemctl status kubelet and read journalctl -u kubelet from the first error, not the last.
  4. Check what the kubelet depends on: the container runtime, its client certificate and CA file, and free disk and memory.
  5. Fix the cause, restart the kubelet, wait for Ready, then uncordon the node.

Where it is taught

CrashLoopBackOff: read what the container said before it died

When to use it: A pod keeps restarting and its status alternates between Running, Error and CrashLoopBackOff.

Signals

  • The RESTARTS count keeps climbing and the status shows CrashLoopBackOff
  • kubectl describe pod shows a Last State of Terminated with an exit code
  • Back-off events in the pod's event list, with the delay between restarts growing

Steps

  1. Read Last State in kubectl describe pod: the reason and exit code say whether the process exited with an error of its own (such as 1) or was killed (137; the reason OOMKilled means it went over its memory limit).
  2. Read the logs of the run that crashed with kubectl logs --previous, because the current container may not have logged anything yet.
  3. Compare what the container expects with what exists: environment variable names, ConfigMap and Secret keys, mounted files and the command.
  4. Check the probes: a liveness probe that fires before the app is ready will kill a healthy process.
  5. Fix the Deployment, not the pod, and watch the rollout until the new pods stay up.

Where it is taught

A Service with nobody behind it: follow the selector

When to use it: Calls to a Service time out or are refused although its pods are running.

Signals

  • Requests to the Service's ClusterIP or name time out or are refused
  • kubectl get endpointslices for the Service lists no ready addresses
  • The pods are Running, but their labels changed in a recent rollout or rollback

Steps

  1. Read the Service's selector and ports with kubectl get service -o wide and kubectl describe service.
  2. List the pods that selector matches with kubectl get pods -l and the selector; no pods means the labels and the selector disagree.
  3. Check the EndpointSlice: pods that match but are not ready are left out until their readiness probe passes.
  4. Check that the Service's targetPort is the port the container actually listens on.
  5. Fix the label, selector or port, then test from a pod inside the cluster.

Where it is taught

Pending pod: let the scheduler's event name the constraint

When to use it: A pod stays Pending and never gets a node.

Signals

  • The pod's status is Pending and its NODE column is empty
  • A FailedScheduling event such as "0/3 nodes are available", followed by the reasons per node
  • A PersistentVolumeClaim the pod uses is itself Pending

Steps

  1. Run kubectl describe pod and read the FailedScheduling event: it names each node's reason.
  2. Match the reason to its constraint: insufficient CPU or memory, a taint the pod does not tolerate, node affinity or a selector, or an unbound claim.
  3. For resources, compare the pod's requests with each node's allocatable capacity and what is already requested on it.
  4. Change the smallest thing that removes the constraint: the request, a toleration, a label or the claim's storage class.
  5. Confirm the pod reached a node and became Ready.

Where it is taught

A name that will not resolve: expand the name the pod asked for

When to use it: An application inside the cluster cannot reach another Service by name.

Signals

  • Errors such as "could not resolve host" or NXDOMAIN from inside a pod
  • The full name service.namespace.svc.cluster.local works but the short name does not
  • Calls to a Service in another namespace fail while calls inside the namespace work

Steps

  1. Run the lookup from a pod in the same namespace as the caller, not from your laptop.
  2. Read the pod's /etc/resolv.conf: the search list decides how a short name is expanded.
  3. Use service.namespace for a Service in another namespace.
  4. Check that CoreDNS pods are running and read the Corefile if every lookup fails.
  5. Check that the Service exists and has endpoints: a name can resolve and still lead nowhere.

Where it is taught

Certification courses