Don't Panic — Linux Kernel Debugging

Linux kernel debugging is the work of explaining a fault below user space, where the machine has acquired a complaint but not necessarily a helpful autobiography. A panic, oops, lockup, or corrupted result is evidence. It is not a signed confession from the component that caused it.

The useful shape is a timeline. Something triggers a state change. A warning may arrive later. Recovery may make the scene look tidier or worse. Then the visible failure appears, wearing the expression of someone who has been here all along. Preserve the log, kernel build, loaded modules, hardware context, workload, and timestamps before a reboot clears the room.

tracefs is the file-system interface for controlling and reading kernel tracing data. Trace events give ordered, structured observations. ftrace, the kernel tracing framework, can follow selected functions. Both are more helpful when they answer one question in a short window than when they record the entire life story of a busy host. The ring buffer is bounded; your attention should be too.

Addresses need a translator. Matching debug information maps an address to functions, files, and lines, while the uncompressed vmlinux image supplies the symbol-aware kernel image used for that work. A stack is still a snapshot, not a causal timeline. It can show where the kernel noticed trouble, which is a useful clue and a terrible alibi.

When logs and traces cannot expose the state you need, kdb is the shell-style inspector on a stopped kernel, and kgdb connects GDB on a development host to the target for source-level work. They are valuable because they stop the system. That is also their principal social defect. A controlled virtual machine is a laboratory; production is usually an evidence source.

Read the Intro for the investigation loop and the boundary between kernel, module, firmware, hardware, and user-space faults. Use Slides for the evidence ladder and tool choices. Keep the Cheatsheet nearby while collecting artifacts, then use the Practice reference and exercise to make one small, bounded capture before the real failure chooses a much less convenient time to be interesting.