4.6.4. Using Kdb/Kgdb to Debug the Kernel
4.6.4.1. Overview of Kdb/Kgdb
Kdb is a debugging tool for the Linux kernel, designed to help developers debug directly in kernel mode without relying on a complete operating system. It is typically used to handle situations after a kernel crash or during runtime kernel issues. Below is the development history of Kdb:
2001: The initial version of Kdb was developed by Kurt Garloff. Kdb is a simple kernel debugging tool for Linux, aiming to provide a debugging method independent of external systems, enabling direct interaction with the kernel via the console.
Early 2000s: Kdb became an important debugging tool for Linux kernel developers, helping diagnose and fix problems when system crashes or kernel issues occurred. Running in kernel mode, Kdb allows developers to debug via serial port or console even if the operating system has crashed or become unresponsive.
Mid-2000s: As kernel functionality expanded, Kdb began to be integrated into more kernel versions. It could be enabled at kernel boot time, offering stronger debugging support.
Present: Kdb remains a commonly used tool in kernel development, especially for low-level troubleshooting and post-crash analysis. It is often used for simple, immediate kernel debugging, but due to its simpler functionality compared to GDB, developers often combine it with other tools for more complex debugging tasks.
Kgdb is designed as a source-level debugger for the Linux kernel. It works with GDB to debug the Linux kernel, allowing GDB to “intrude” into the kernel to inspect memory, variables, and call stack information—similar to how application developers use GDB to debug applications. Breakpoints can be set in kernel code, and limited execution steps can be performed. Below is the development history of Kgdb:
2001: The initial work on KGDB was proposed and implemented by Jason Wessel. KGDB is based on GDB and communicates with the target kernel via serial ports, Ethernet, or other communication interfaces, enabling developers to debug the kernel as they would debug user-space programs.
2002: KGDB began to attract widespread attention in the Linux 2.6 kernel. It allowed developers to use GDB in user space to debug kernel-space code, greatly improving debugging flexibility. Compared to KDB, KGDB supports more complex debugging operations such as setting breakpoints and single-stepping, similar to debugging user-space programs.
Late 2000s: KGDB’s supported features gradually increased, including support for multiple hardware platforms (such as x86, ARM) and more efficient remote debugging capabilities.
2010s: KGDB was used in conjunction with other debugging tools (such as QEMU), supporting kernel debugging through virtualization platforms. Its debugging methods became more flexible, enabling complex debugging using remote debugging and hardware emulation tools.
Present: KGDB continues to be used by kernel developers and debuggers, especially during kernel development. Used in conjunction with GDB and KDB, it provides powerful debugging capabilities. Although KGDB configuration is relatively complex, it remains one of the preferred tools for Linux kernel developers performing kernel-level debugging.
4.6.4.2. Introduction to Kdb/Kgdb Features
Introduction to Kdb Features
Kdb is a built-in debugger in the Linux kernel that allows developers to debug while the kernel is running, typically used for low-level debugging such as crash analysis, memory inspection, and viewing kernel state.
Features and Characteristics of Kdb:
Kernel-Mode Debugging: KDB runs directly in kernel mode without requiring external debugging tools or environments. It is embedded in the kernel, so even during a kernel crash, developers can still interact with and debug the kernel via console, serial port, etc.
Trigger-Based Debugging: KDB can automatically start when specific events occur in the kernel, such as system crashes, exceptions, or error detection. Developers can set breakpoints and inspect the kernel state under specific conditions.
Real-Time Debugging: KDB does not rely on external hardware or debuggers. Developers can interact with the kernel in real time, inspecting kernel variables, stacks, memory, etc.
Support for Multiple Interfaces: KDB supports accessing and debugging the kernel through console, serial port, or even directly from the debugging command line, which is very useful in environments without a full graphical interface, especially in embedded development.
Debugging Kernel Data Structures: KDB can directly access and view kernel data structures such as process scheduling, memory management, and file systems, helping to identify low-level issues.
Independence from External Tools: Unlike external tools such as GDB, KDB does not require additional configuration. As part of the kernel, it is suitable for embedded systems or scenarios where external debugging tools cannot be connected.
Main Uses of Kdb:
Debugging During Kernel Crashes: When a kernel crash occurs, KDB can automatically trigger and start a debugging session, allowing developers to inspect the crash context, memory, registers, call stacks, etc.
Low-Level Debugging: KDB can directly access kernel data structures, making it suitable for low-level debugging, especially when external debuggers cannot be used.
Real-Time System Debugging: In real-time operating systems, KDB can respond immediately to system states and perform debugging, helping to quickly locate problems.
Introduction to Kgdb Features
Kgdb is a tool that allows developers to debug kernel code using GDB. It enables the use of GDB to debug the kernel, especially when the kernel is running on a target device, allowing developers to debug the target device via serial port or network. Essentially, Kgdb is the kernel’s support for GDB, serving as a bridge between GDB and the kernel.
Features and Characteristics of Kgdb:
Debugging the Kernel via GDB:
Kgdb allows developers to use GDB in user space to debug kernel code. This means you can use GDB’s powerful debugging features (such as breakpoints, single-stepping, stack tracing, etc.) to debug kernel code.
Integration with GDB:
Kgdb integrates with GDB, connecting the debugging session to the kernel debugging target through serial ports, networks, or other communication methods. Users can use GDB commands to set breakpoints, single-step, inspect variables, etc., on the kernel.
Remote Debugging:
Kgdb allows remote kernel debugging. The kernel communicates with the GDB host via serial port, network, or other channels, which is very useful for remotely debugging kernels on hardware devices.
Support for Multiple Debugging Features:
When using GDB, developers can set breakpoints, single-step, inspect memory, view register contents, debug kernel modules, etc. GDB provides very powerful debugging features, making it easy to perform complex kernel debugging.
Debugging Kernel Crashes:
Kgdb can also be used to debug kernel crashes. Developers can perform remote debugging via GDB at the time of a crash, inspecting the crash context, viewing call stacks, printing memory contents, etc.
Cooperation with KDB:
Kgdb and KDB are not mutually exclusive; in fact, they can be used together. KDB allows developers to debug the kernel without relying on external tools, while Kgdb allows for more detailed and complex debugging via GDB.
Main Uses of Kgdb:
Remote Debugging: Through communication methods such as serial ports and Ethernet, Kgdb can be used to remotely debug kernels on embedded devices or other target machines. This is very helpful for developing and maintaining embedded systems or other special hardware platforms.
Complex Kernel Debugging: When kernel code requires complex debugging, developers can leverage GDB’s powerful features, such as multi-threaded debugging and memory analysis, to locate and resolve issues.
Kernel Module Debugging: Developers can use GDB to debug kernel modules, analyzing problems during module loading and execution.
Comparison Between Kdb and Kgdb
| Feature/Function | KDB | KGDB |
|---|---|---|
| Debugging Method | Built-in kernel debugging tool, interacts directly during kernel runtime | Debugs the kernel via external GDB debugger, typically using serial or network connections |
| Use Case | Suitable for simple, quick kernel debugging, especially for starting debugging after a system crash | Suitable for complex kernel debugging, especially for remote debugging or when GDB features are needed |
| Debugging Features | Provides basic debugging commands such as viewing stack, registers, memory, etc. | Provides all GDB debugging features such as breakpoints, single-stepping, memory analysis, etc. |
| Debugging Complexity | Simpler functionality, suitable for low-level debugging and basic issue diagnosis | Powerful functionality, suitable for complex debugging tasks |
| Debugging Devices | No external devices required; interacts with the kernel only via console, serial port, etc. | Requires GDB and external communication devices (serial port, network, etc.) |
| Debugging Real-Time Performance | Provides immediate debugging, especially fast response during system crashes | May have slight delays due to reliance on external debuggers |
| Applicability | Suitable for environments without external debugging tools or hardware, such as embedded devices | Suitable for environments requiring powerful debugging features and remote debugging |
| Remote Debugging Support | Does not support remote debugging | Supports remote debugging via serial port, network, etc. |
| Cooperation with KDB | Can be used with KGDB for simple debugging | Can be used with KDB for more complex debugging |
4.6.4.3. Specific Usage Methods of Kdb/Kgdb
Enabling Kdb/Kgdb
By default, the X5 kernel does not support Kdb/Kgdb and requires modifications to the kernel.
Run ./bd.sh boot menuconfig and modify the kgdb-related configuration items as follows:
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
CONFIG_KGDB_KDB=y
CONFIG_KDB_DEFAULT_ENABLE=0x1
After saving the changes, open kernel/arch/arm64/configs/hobot_x5_soc_defconfig to ensure that the kgdb-related configuration items have been correctly set.
Starting Kdb Debugging on the Board
There are two ways to start kdb debugging on the board: you can configure it during the U-Boot stage or start it after entering the kernel.
Starting Kdb During U-Boot Stage
Modify and save bootargs in U-Boot to change the kernel’s command line options, then boot with (e.g., using ttyS0):
Hobot# setenv bootargs kgdboc=ttyS0,115200 kgdbwait
Hobot# run bootcmd
After the kernel starts, the following log will appear, indicating entry into kgdb:
Starting kernel ...
[ 0.000000] Linux version 6.1.83-DR-PL5.1_V1.0.14 (sxq@DESKTOP-6VORLA0) (aarch64-none-linux-gnu-gcc (Arm GNU Toolchain 11.3.Rel1) 11.3.1 20220712, GNU ld (Arm GNU Toolchain 11.3.Rel1) 2.38.20220708) #1 SMP PREEMPT Wed Dec 4 20:10:42 CST 2024
[ 0.000000] Kernel command line: console=ttyS0,115200n8 root=/dev/mmcblk0p9 ro rootwait hobotboot.slot_suffix=_a hobotboot.reason=COLD_BOOT hobotboot.medium=MMC hobotboot.mode=normal hobotboot.ab_switch_reason=normal hobotboot.pmic_type=single-pmic kgdboc=ttyS0,115200 kgdbwait
[ 0.111978] audit: type=2000 audit(0.090:1): state=initialized audit_enabled=0 res=1
[ 0.144479] (NULL device *): no horizon,gpio-banks in node /soc/disp_apb/disp_iomuxc@3e0a0054
[ 0.182704] SCSI subsystem initialized
[ 0.262379] Initialise system trusted keyrings
[ 0.294398] Key type asymmetric registered
[ 0.294409] Asymmetric key parser 'x509' registered
[ 0.393114] KGDB: Waiting for connection from remote gdb...
Entering kdb (current=0xffff0000058a0000, pid 1) on processor 7 due to Keyboard Entry
[7]kdb>
Starting Kdb After Kernel Boot
To start kgdb after entering the kernel:
# Unload the watchdog driver first (if present)
rmmod hobot_watchdog
echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
echo g > /proc/sysrq-trigger
Log output:
root@buildroot:~# echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
root@buildroot:~# echo g > /proc/sysrq-trigger
Entering kdb (current=0xffff0000cca42b80, pid 1003) on processor 4 due to Keyboard Entry
[4]kdb>
Note: In the debug version of the X5 BSP, the Watchdog device is registered by default, but the watchdog timer is not enabled, so there is no need to unload the Watchdog.
Introduction to Kdb Debugging Commands
Enter the help command in the Kdb command terminal to display the list of Kdb commands:
[7]kdb> help
Command Usage Description
----------------------------------------------------------
md <vaddr> Display Memory Contents, also mdWcN, e.g. md8c1
mdr <vaddr> <bytes> Display Raw Memory
mdp <paddr> <bytes> Display Physical Memory
mds <vaddr> Display Memory Symbolically
mm <vaddr> <contents> Modify Memory Contents
go [<vaddr>] Continue Execution
rd Display Registers
rm <reg> <contents> Modify Registers
ef <vaddr> Display exception frame
bt [<vaddr>] Stack traceback
btp <pid> Display stack for process <pid>
bta [<state_chars>|A] Backtrace all processes whose state matches
btc Backtrace current process on each cpu
btt <vaddr> Backtrace process given its struct task address
env Show environment variables
set Set environment variables
help Display Help Message
? Display Help Message
cpu <cpunum> Switch to new cpu
kgdb Enter kgdb mode
ps [<state_chars>|A] Display active task list
pid <pidnum> Switch to another task
reboot Reboot the machine immediately
lsmod List loaded kernel modules
sr <key> Magic SysRq key
dmesg [lines] Display syslog buffer
defcmd name "usage" "help" Define a set of commands, down to endefcmd
kill <-signal> <pid> Send a signal to a process
summary Summarize the system
per_cpu <sym> [<bytes>] [<cpu>]
Display per_cpu variables
grephelp Display help on | grep
bp [<vaddr>] Set/Display breakpoints
bl [<vaddr>] Display breakpoints
bc <bpnum> Clear Breakpoint
be <bpnum> Enable Breakpoint
bd <bpnum> Disable Breakpoint
ss Single Step
dumpcommon Common kdb debugging
dumpall First line debugging
dumpcpu Same as dumpall but only tasks on cpus
ftdump [skip#entries] [cpu]
Dump ftrace log; -skip dumps last #entries
Below is a brief introduction to these commands:
Memory-Related Commands
| Command | Parameter | Description |
|---|---|---|
md |
<vaddr> |
Displays memory contents; <vaddr> is the virtual address. Supports WcN parameters to control display format and block size (e.g., md8c1). |
mdr |
<vaddr> <bytes> |
Displays <bytes> bytes of raw memory starting from <vaddr>. |
mdp |
<paddr> <bytes> |
Displays physical memory contents starting from <paddr> for the specified number of bytes. |
mds |
<vaddr> |
Symbolically displays memory contents starting from virtual address <vaddr>. |
mm |
<vaddr> <contents> |
Modifies memory contents by writing <contents> to the memory at virtual address <vaddr>. |
Execution Control Commands
| Command | Parameter | Description |
|---|---|---|
go |
[<vaddr>] |
Continues program execution. If a virtual address <vaddr> is specified, execution starts from that address. Also used to exit the kdb interface and return to the kernel command line. |
bt |
[<vaddr>] |
Displays stack backtrace. An optional <vaddr> can be specified as the starting address. |
bta |
[<state_chars>\|A] |
Backtraces stacks of all processes filtered by process state <state_chars>; A means all processes. |
btc |
Backtraces the stack of the current process on all CPUs. | |
btt |
<vaddr> |
Backtraces the process stack starting from the given task structure address <vaddr>. |
ss |
Performs single-step debugging. | |
kill |
<-signal> <pid> |
Sends a signal <signal> to the specified process <pid>. |
pid |
<pidnum> |
Switches debugging context to the process with ID <pidnum>. |
sr |
<key> |
Triggers the magic SysRq key, usually used for emergency operations like forced reboot. |
Register-Related Commands
| Command | Parameter | Description |
|---|---|---|
rd |
Displays the contents of registers on the current CPU. | |
rm |
<reg> <contents> |
Modifies the contents of register <reg> to the new value <contents>. |
Environment Variable Commands
| Command | Parameter | Description |
|---|---|---|
env |
Displays current environment variables. | |
set |
Sets the value of an environment variable. |
Process-Related Commands
| Command | Parameter | Description |
|---|---|---|
ps |
[<state_chars>\|A] |
Displays the list of active tasks, optionally filtered by process state <state_chars>; A means all tasks. |
lsmod |
Lists currently loaded kernel modules. | |
pid |
<pidnum> |
Switches debugging context to the process with ID <pidnum>. |
btp |
<pid> |
Displays stack information for the specified process ID (pid). |
Kernel Debugging Commands
| Command | Parameter | Description |
|---|---|---|
dmesg |
[lines] |
Displays contents of the kernel log buffer; optionally specifies the number of lines lines. |
dumpcommon |
Performs common kernel debugging dumps. | |
dumpall |
Performs full memory dump debugging. | |
dumpcpu |
Dumps only task-related information on CPUs. |
Custom and Help Commands
| Command | Parameter | Description |
|---|---|---|
defcmd |
name "usage" "help" |
Defines a set of custom commands with usage and help documentation. |
grephelp |
Displays help information, which can be filtered using pipe (\|). |
Other Commands
| Command | Parameter | Description |
|---|---|---|
kgdb |
Enters KGDB mode for low-level debugging. | |
summary |
Displays a brief summary of the system. | |
per_cpu |
<sym> [<bytes>] [<cpu>] |
Displays per-CPU variables for symbol <sym>, optionally specifying byte count <bytes> and CPU <cpu>. |
bl |
Displays current breakpoint information. | |
bp |
<vaddr> |
Sets a breakpoint to halt execution at the specified virtual address. |
bc |
<bpnum> |
Clears the specified breakpoint. |
be |
<bpnum> |
Enables the specified breakpoint. |
bd |
<bpnum> |
Disables the specified breakpoint. |
Below are execution logs of some Kdb commands:
# Display stack backtrace
[7]kdb> bt
Stack traceback for pid 999
0xffff0000c60f0000 999 1 1 7 R 0xffff0000c60f09f0 *bash
CPU: 7 PID: 999 Comm: bash Tainted: P C O 6.1.83-DR-PL5.1_V1.0.14 #11
Hardware name: D-Robotics X5 EVB LP4 1_B board (DT)
Call trace:
dump_backtrace+0xd8/0x130
show_stack+0x18/0x30
dump_stack_lvl+0x68/0x84
dump_stack+0x18/0x34
kdb_dump_stack_on_cpu+0x88/0x90
kdb_show_stack+0x90/0xa0
kdb_bt1+0xc4/0x140
kdb_bt+0x328/0x37c
kdb_parse+0x2c4/0x63c
kdb_main_loop+0x434/0x7b4
kdb_stub+0x270/0x444
kgdb_cpu_enter+0x168/0x66c
kgdb_handle_exception+0xcc/0x120
kgdb_compiled_brk_fn+0x28/0x40
call_break_hook+0x68/0x7c
brk_handler+0x1c/0x60
# Display contents of specified virtual memory; below shows the full kernel version string
[0]kdb> md linux_banner
0xffff800008baae48 65762078756e694c 2e36206e6f697372 Linux version 6.
0xffff800008baae58 2d52442d33382e31 31565f312e354c50 1.83-DR-PL5.1_V1
0xffff800008baae68 78732820302e312e 4f544b5345444071 .1.0 (sxq@DESKTO
0xffff800008baae78 414c524f56362d50 6372616128202930 P-6VORLA0) (aarc
0xffff800008baae88 656e6f6e2d343668 672d78756e696c2d h64-none-linux-g
0xffff800008baae98 28206363672d756e 20554e47206d7241 nu-gcc (Arm GNU
0xffff800008baaea8 696168636c6f6f54 522e332e3131206e Toolchain 11.3.R
0xffff800008baaeb8 2e31312029316c65 3232303220312e33 el1) 11.3.1 2022
# Display current CPU register contents
[7]kdb> rd
x0: ffff800009268000 x1: 0000000000000001 x2: ffff800009268558
x3: 0000000000000000 x4: ffff0000ff746b60 x5: ffff0000ff746b60
x6: 0000000000000000 x7: ffff800009184748 x8: 00000000ffffefff
x9: ffff80000912c748 x10: ffff800009184748 x11: 00000000000002fa
x12: 00000000000008ee x13: ffff80000912c748 x14: 0000000000000000
x15: fffffffffffed7d8 x16: 0000000000000000 x17: 0000000000000000
x18: 0000000000000018 x19: 0000000000000067 x20: ffff80000912c000
x21: ffff80000911a000 x22: 0000000000000006 x23: 0000000000000000
x24: 0000000000000000 x25: ffff800008af8f40 x26: 0000000000000000
x27: 0000000000000000 x28: ffff0000c60f0000 x29: ffff800019563c80
x30: ffff800008116184 sp: ffff800019563c80 pc: ffff8000081160ec
pstate: 60400009 v0: ?? v1: ?? v2: ?? v3: ?? v4: ?? v5: ?? v6: ?? v7: ??
v8: ?? v9: ?? v10: ?? v11: ?? v12: ?? v13: ?? v14: ?? v15: ?? v16: ??
v17: ?? v18: ?? v19: ?? v20: ?? v21: ?? v22: ?? v23: ?? v24: ?? v25: ??
v26: ?? v27: ?? v28: ?? v29: ?? v30: ?? v31: ?? fpsr: 00000000
fpcr: 00000000
Remote Connection to Board for Kgdb Debugging
After entering the Kdb interface on the board, execute the kgdb command to wait for the host GDB to connect remotely:
kdb> kgdb
Entering please attach debugger or use $D#44+ or $3#33
When prompted to wait for connection, close the serial terminal (to avoid port occupation). Then, on the host, use the compilation toolchain used by the X5 BSP: opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gdb, and load the kernel vmlinux (path in X5 BSP source: out/build/kernel/vmlinux). For convenience, it is recommended to create an alias in .bashrc:
alias arm_gdb='/opt/arm-gnu-toolchain-11.3.rel1-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gdb'
Then, connect to the target machine via the host serial port (e.g., /dev/ttyUSB0) (add sudo if device permissions require it):
arm_gdb out/build/kernel/vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
Once connected, you can use GDB commands for debugging.
For more usage information on Kdb/Kgdb, refer to Using kgdb, kdb and the kernel debugger internals.
4.6.4.4. Common Issues
Common Kdb Issues
Unable to Connect to Debug Terminal: If Kdb interacts externally via serial port or other terminal devices, incorrect device configuration (e.g., wrong baud rate) may prevent connection.
Incomplete Debug Information: Kdb provides relatively limited debug information and may not offer sufficient context, especially during complex crashes. In such cases, other debugging tools like kgdb or crash may need to be used in conjunction.
Kdb Not Responding to Input: When the system crashes or hangs, Kdb may fail to respond to input. This could be due to console or serial device configuration issues, or because the kernel failed to properly initialize the debug port before crashing.
Common Kgdb Issues
GDB Unable to Connect to Kernel: This may be due to the kernel not being properly configured to enable Kgdb, or unstable external device connections (e.g., serial or network). Ensure that CONFIG_KGDB is enabled in the kernel and that communication ports and GDB configurations are correct.
Incomplete Kernel Debug Information: Debug symbols may not be included in the kernel image, resulting in insufficient debug information. Ensure that the kernel is configured with debug symbols (e.g., CONFIG_DEBUG_INFO).
GDB Version Mismatch with Kernel: The GDB version may be incompatible with the kernel’s debug interface, causing crashes or connection issues during debugging. Use a GDB version compatible with the kernel.
Communication Delay Between Debugger and Kernel: When Kgdb performs remote debugging via serial port or network, significant delays may occur, especially in high-load systems. Slow response or timeouts may occur during debugging.