Linux troubleshooting patterns
A Linux fault usually leaves its evidence in a predictable place: the file system's counters, the journal, a file's mode, a socket table or the process list. Each pattern names the signals, the order to look in, and the lessons and labs where you practise it.
Full disk: df for the file system, du for the directory, and inodes too
When to use it: Writes fail with "No space left on device", or a service stops because it cannot write its logs or data.
Signals
- "No space left on device" in an application log or a shell
- df -h shows a file system at or near 100 percent
- df -h shows free space but writes still fail, because df -i shows no free inodes
Steps
- Run df -h to find which file system is full, and df -i to check its inodes.
- Walk down from that mount point with du -xh --max-depth=1 | sort -h, staying on the one file system.
- If df and du disagree, look for deleted files a process still holds open with lsof +L1: their space comes back only when the process closes them.
- Remove or rotate what is safe to remove, or restart the process holding the deleted file.
- Confirm the space is back with df, and fix whatever filled it so it does not fill again.
Where it is taught
- Running Out Of Space: df, du And Inodes — Lesson, Linux Fundamentals & Systems Administration
- Read filesystem capacity — Lab, Linux Fundamentals & Systems Administration
- Find the directory pressure — Lab, Linux Fundamentals & Systems Administration
- Inodes, Hard Links And What rm Removes — Lesson, Linux Fundamentals & Systems Administration
A service that will not start: systemctl status, then the journal
When to use it: A systemd service is failed, or keeps restarting, after a deploy, a reboot or a configuration change.
Signals
- systemctl status shows failed, or activating (auto-restart) over and over
- An exit status such as 203/EXEC, which means systemd could not run the ExecStart program at all
- The service works when started by hand from a shell but not as a unit
Steps
- Run systemctl status for the unit and read its state, its exit status and the last log lines.
- Read the full story with journalctl -u and the unit name, from this boot, starting at the first error.
- Read the unit as systemd sees it with systemctl cat: ExecStart's path, User, WorkingDirectory and Environment.
- Fix the unit or the program, run systemctl daemon-reload if the unit file changed, then restart it.
- Check it stays active, and that systemctl is-enabled says it will start at boot.
Where it is taught
- systemd: Units, Targets And The Boot Graph — Lesson, Linux Fundamentals & Systems Administration
- Your Own Unit, And Reading journald — Lesson, Linux Fundamentals & Systems Administration
- Query the journal — Lab, Linux Fundamentals & Systems Administration
- Separate state from enablement — Lab, Linux Fundamentals & Systems Administration
Permission denied: identity first, then the mode, then every directory on the path
When to use it: A user, a service or SSH is refused access to a file that seems to be there.
Signals
- "Permission denied" from a program, a script or the shell
- The same command works as root or as the file's owner, but not as the account that needs it
- SSH refuses a private key because its permissions are too open
Steps
- Check who is asking with id: the user, its groups, and whether a new group membership needs a fresh login.
- Read the file's owner, group and mode with ls -l, and decide which of owner, group or other applies to that identity.
- Check every directory on the path, because each one needs the execute bit for that identity (namei -l lists them all).
- Make the narrowest change: add the user to the group, or change one class's bits, rather than chmod 777.
- For SSH private keys, keep them owned by the user and readable only by them (mode 600).
Where it is taught
- Reading And Setting A File Mode — Lesson, Linux Fundamentals & Systems Administration
- Change only the intended class — Lab, Linux Fundamentals & Systems Administration
- Groups, And How Shared Access Is Granted — Lesson, Linux Fundamentals & Systems Administration
- Why a directory needs the execute bit — Lab, Bash Scripting & Command Line Automation
Cannot reach it: climb the diagnostic ladder one rung at a time
When to use it: A client cannot reach a service on another machine, and you do not yet know which layer is at fault.
Signals
- Connection refused: the host answered, but nothing listens on that port
- Connection timed out: nothing answered at all, often a route or a firewall
- Name or service not known: the failure happened before any packet was sent
Steps
- Is it up: check the process is running on the server.
- Is it named: resolve the name the client uses, with getent hosts or dig.
- Is it routed: check the client has a route to that address with ip route get.
- Is it listening: check with ss -ltnp that the process listens on that port and on the right address, not only 127.0.0.1.
- Is it answering and arriving: test with curl -v, and check firewalls on both sides. Stop at the first rung that fails and fix that.
Where it is taught
- The Diagnostic Ladder: Is It Up, Is It Named, Is It Routed — Lesson, Linux Fundamentals & Systems Administration
- The Diagnostic Ladder: Is It Listening, Answering, Arriving — Lesson, Linux Fundamentals & Systems Administration
- Locate the missing leg — Lab, Linux Fundamentals & Systems Administration
- Refused, Timed Out, Reset: Reading A Failure — Lesson, Linux Fundamentals & Systems Administration
- Classify the result — Lab, Linux Fundamentals & Systems Administration
A stuck or runaway process: identify it, then signal it deliberately
When to use it: A process uses all of a CPU, ignores Ctrl-C, will not exit, or leaves zombies behind.
Signals
- High load, with one process at the top of top or ps
- A process that ignores Ctrl-C or kill, or sits in state D
- Processes in state Z (zombies) that never go away
Steps
- Find the process ID and its state with ps or top, or in /proc/<pid>/status: R is running, S is sleeping, D is waiting on I/O, Z is a zombie.
- Ask it to stop first with SIGTERM, which a program can catch to clean up, and give it time.
- Send SIGKILL only if it will not stop; it cannot be caught, so nothing gets cleaned up.
- A process in state D is waiting on a device or a network file system: signals wait too, so fix the I/O it is stuck on.
- A zombie has already exited; its parent has not collected it, so fix or restart the parent.
Where it is taught
- What A Process Is — Lesson, Linux Fundamentals & Systems Administration
- Signals, And What kill Really Sends — Lesson, Linux Fundamentals & Systems Administration
- Escalate deliberately — Lab, Linux Fundamentals & Systems Administration
- Everything Is A File: proc, sys And dev — Lesson, Linux Fundamentals & Systems Administration
- Watching a signal arrive — Lab, Bash Scripting & Command Line Automation