Kubernetes troubleshooting patterns
Most Kubernetes faults announce themselves the same few ways. Each pattern here names the signals you will see, the order to check things in, and the lessons and break-fix drills where you practise it on a cluster.
Node NotReady: read the conditions, then the kubelet
When to use it: A node shows NotReady or Unknown, and the pods on it stop answering or are being replaced elsewhere.
Signals
- kubectl get nodes shows NotReady, or Unknown when the kubelet has stopped reporting at all
- The node carries the node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taint
- Pods on the node are evicted once their toleration for that taint runs out
Steps
- Run kubectl describe node and read the Conditions table: which condition changed, its reason, and when the last heartbeat arrived.
- Cordon the node so nothing new is scheduled onto it while you work.
- On the node, check the kubelet with systemctl status kubelet and read journalctl -u kubelet from the first error, not the last.
- Check what the kubelet depends on: the container runtime, its client certificate and CA file, and free disk and memory.
- Fix the cause, restart the kubelet, wait for Ready, then uncordon the node.
Where it is taught
- A Node Goes NotReady Forty Minutes In — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Node NotReady: Kubelet, Runtime And Certificates — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Drill: node02 went NotReady and took cache with it — Lab, Certified Kubernetes Administrator (CKA) Exam Prep
- Troubleshooting: NotReady Nodes And kubelet Logs — Lesson, Kubernetes: Production-Grade Container Orchestration
- Read node conditions and events — Lab, Kubernetes: Production-Grade Container Orchestration
CrashLoopBackOff: read what the container said before it died
When to use it: A pod keeps restarting and its status alternates between Running, Error and CrashLoopBackOff.
Signals
- The RESTARTS count keeps climbing and the status shows CrashLoopBackOff
- kubectl describe pod shows a Last State of Terminated with an exit code
- Back-off events in the pod's event list, with the delay between restarts growing
Steps
- Read Last State in kubectl describe pod: the reason and exit code say whether the process exited with an error of its own (such as 1) or was killed (137; the reason OOMKilled means it went over its memory limit).
- Read the logs of the run that crashed with kubectl logs --previous, because the current container may not have logged anything yet.
- Compare what the container expects with what exists: environment variable names, ConfigMap and Secret keys, mounted files and the command.
- Check the probes: a liveness probe that fires before the app is ready will kill a healthy process.
- Fix the Deployment, not the pod, and watch the rollout until the new pods stay up.
Where it is taught
- What A Container Said Before It Died — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Drill: orders-db crashes on every start — Lab, Certified Kubernetes Administrator (CKA) Exam Prep
- Troubleshooting: Probes, Logs, exec And CrashLoopBackOff — Lesson, Kubernetes: Production-Grade Container Orchestration
- Collect evidence before changing code — Lab, Kubernetes: Production-Grade Container Orchestration
- A probe that kills a healthy server — Lab, Certified Kubernetes Administrator (CKA) Exam Prep
A Service with nobody behind it: follow the selector
When to use it: Calls to a Service time out or are refused although its pods are running.
Signals
- Requests to the Service's ClusterIP or name time out or are refused
- kubectl get endpointslices for the Service lists no ready addresses
- The pods are Running, but their labels changed in a recent rollout or rollback
Steps
- Read the Service's selector and ports with kubectl get service -o wide and kubectl describe service.
- List the pods that selector matches with kubectl get pods -l and the selector; no pods means the labels and the selector disagree.
- Check the EndpointSlice: pods that match but are not ready are left out until their readiness probe passes.
- Check that the Service's targetPort is the port the container actually listens on.
- Fix the label, selector or port, then test from a pod inside the cluster.
Where it is taught
- A Service With Nobody Behind It — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Drill: checkout answers nobody after a rollback — Lab, Certified Kubernetes Administrator (CKA) Exam Prep
- Troubleshooting: DNS, Empty Endpoints And Broken Routes — Lesson, Kubernetes: Production-Grade Container Orchestration
- An empty EndpointSlice explains the timeout — Lab, Kubernetes: Production-Grade Container Orchestration
- Troubleshooting: describe, Events And A First Smoke Test — Lesson, Kubernetes: Production-Grade Container Orchestration
Pending pod: let the scheduler's event name the constraint
When to use it: A pod stays Pending and never gets a node.
Signals
- The pod's status is Pending and its NODE column is empty
- A FailedScheduling event such as "0/3 nodes are available", followed by the reasons per node
- A PersistentVolumeClaim the pod uses is itself Pending
Steps
- Run kubectl describe pod and read the FailedScheduling event: it names each node's reason.
- Match the reason to its constraint: insufficient CPU or memory, a taint the pod does not tolerate, node affinity or a selector, or an unbound claim.
- For resources, compare the pod's requests with each node's allocatable capacity and what is already requested on it.
- Change the smallest thing that removes the constraint: the request, a toleration, a label or the claim's storage class.
- Confirm the pod reached a node and became Ready.
Where it is taught
- Troubleshooting: Pending Pod Triage — Lesson, Kubernetes: Production-Grade Container Orchestration
- Events identify the blocking constraint — Lab, Kubernetes: Production-Grade Container Orchestration
- Usage, Requests And The Pod That Will Not Fit — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Make the third replica fit — Lab, Certified Kubernetes Administrator (CKA) Exam Prep
- Troubleshooting: Pending Claims And Failed Mounts — Lesson, Kubernetes: Production-Grade Container Orchestration
A name that will not resolve: expand the name the pod asked for
When to use it: An application inside the cluster cannot reach another Service by name.
Signals
- Errors such as "could not resolve host" or NXDOMAIN from inside a pod
- The full name service.namespace.svc.cluster.local works but the short name does not
- Calls to a Service in another namespace fail while calls inside the namespace work
Steps
- Run the lookup from a pod in the same namespace as the caller, not from your laptop.
- Read the pod's /etc/resolv.conf: the search list decides how a short name is expanded.
- Use service.namespace for a Service in another namespace.
- Check that CoreDNS pods are running and read the Corefile if every lookup fails.
- Check that the Service exists and has endpoints: a name can resolve and still lead nowhere.
Where it is taught
- The Name That Will Not Resolve — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Fix the cross-namespace name — Lab, Certified Kubernetes Administrator (CKA) Exam Prep
- CoreDNS: The Corefile Every Name Depends On — Lesson, Certified Kubernetes Administrator (CKA) Exam Prep
- Check DNS and route status — Lab, Kubernetes: Production-Grade Container Orchestration