Linux troubleshooting patterns

A Linux fault usually leaves its evidence in a predictable place: the file system's counters, the journal, a file's mode, a socket table or the process list. Each pattern names the signals, the order to look in, and the lessons and labs where you practise it.

Full disk: df for the file system, du for the directory, and inodes too

When to use it: Writes fail with "No space left on device", or a service stops because it cannot write its logs or data.

Signals

  • "No space left on device" in an application log or a shell
  • df -h shows a file system at or near 100 percent
  • df -h shows free space but writes still fail, because df -i shows no free inodes

Steps

  1. Run df -h to find which file system is full, and df -i to check its inodes.
  2. Walk down from that mount point with du -xh --max-depth=1 | sort -h, staying on the one file system.
  3. If df and du disagree, look for deleted files a process still holds open with lsof +L1: their space comes back only when the process closes them.
  4. Remove or rotate what is safe to remove, or restart the process holding the deleted file.
  5. Confirm the space is back with df, and fix whatever filled it so it does not fill again.

Where it is taught

A service that will not start: systemctl status, then the journal

When to use it: A systemd service is failed, or keeps restarting, after a deploy, a reboot or a configuration change.

Signals

  • systemctl status shows failed, or activating (auto-restart) over and over
  • An exit status such as 203/EXEC, which means systemd could not run the ExecStart program at all
  • The service works when started by hand from a shell but not as a unit

Steps

  1. Run systemctl status for the unit and read its state, its exit status and the last log lines.
  2. Read the full story with journalctl -u and the unit name, from this boot, starting at the first error.
  3. Read the unit as systemd sees it with systemctl cat: ExecStart's path, User, WorkingDirectory and Environment.
  4. Fix the unit or the program, run systemctl daemon-reload if the unit file changed, then restart it.
  5. Check it stays active, and that systemctl is-enabled says it will start at boot.

Where it is taught

Permission denied: identity first, then the mode, then every directory on the path

When to use it: A user, a service or SSH is refused access to a file that seems to be there.

Signals

  • "Permission denied" from a program, a script or the shell
  • The same command works as root or as the file's owner, but not as the account that needs it
  • SSH refuses a private key because its permissions are too open

Steps

  1. Check who is asking with id: the user, its groups, and whether a new group membership needs a fresh login.
  2. Read the file's owner, group and mode with ls -l, and decide which of owner, group or other applies to that identity.
  3. Check every directory on the path, because each one needs the execute bit for that identity (namei -l lists them all).
  4. Make the narrowest change: add the user to the group, or change one class's bits, rather than chmod 777.
  5. For SSH private keys, keep them owned by the user and readable only by them (mode 600).

Where it is taught

Cannot reach it: climb the diagnostic ladder one rung at a time

When to use it: A client cannot reach a service on another machine, and you do not yet know which layer is at fault.

Signals

  • Connection refused: the host answered, but nothing listens on that port
  • Connection timed out: nothing answered at all, often a route or a firewall
  • Name or service not known: the failure happened before any packet was sent

Steps

  1. Is it up: check the process is running on the server.
  2. Is it named: resolve the name the client uses, with getent hosts or dig.
  3. Is it routed: check the client has a route to that address with ip route get.
  4. Is it listening: check with ss -ltnp that the process listens on that port and on the right address, not only 127.0.0.1.
  5. Is it answering and arriving: test with curl -v, and check firewalls on both sides. Stop at the first rung that fails and fix that.

Where it is taught

A stuck or runaway process: identify it, then signal it deliberately

When to use it: A process uses all of a CPU, ignores Ctrl-C, will not exit, or leaves zombies behind.

Signals

  • High load, with one process at the top of top or ps
  • A process that ignores Ctrl-C or kill, or sits in state D
  • Processes in state Z (zombies) that never go away

Steps

  1. Find the process ID and its state with ps or top, or in /proc/<pid>/status: R is running, S is sleeping, D is waiting on I/O, Z is a zombie.
  2. Ask it to stop first with SIGTERM, which a program can catch to clean up, and give it time.
  3. Send SIGKILL only if it will not stop; it cannot be caught, so nothing gets cleaned up.
  4. A process in state D is waiting on a device or a network file system: signals wait too, so fix the I/O it is stuck on.
  5. A zombie has already exited; its parent has not collected it, so fix or restart the parent.

Where it is taught