GDB: Tips for Low-Level Debugging

Gene Cooperman

There are many good tutorials on the basics of debugging in gdb. The purpose of this document is to discuss lower level issues: debugging in assembly, loading the right symbols from a library, debugging into libc.so or libpthread.so, etc.

common gdb commands: gdb --args, finish, where, frame 0, list, break ..., info breakpoints, del <BREAK_NUM>, break ... if <COND> (break if <COND> is true), until, <TAB> for auto-completion

debugging fork syscall: set follow-fork-mode child/parent

debugging exec, excvp, etc.: Add delay loop to second program (into which we will exec): {static int xxx=1; while(xxx >= 1); x++;} to target source code. Then run the original program. After it execs into the second program, use gdb attach: gdb <PROGRAM> `pgrep -n <PROGRAM>`

debugging threads: info threads, thread <THREAD_NUM>, thread apply all where, break ... thread <THREAD_NUM>, set scheduler-locking on/off (default: 'off'; 'on' means that single-stepping steps only the current thread, while other threads remain stopped; in some unusual circumstances, this can lead to deadlock)

WILL FILL IN MORE

Debugging at the Assembly Level

p $pc, p $sp, info registers, x/10i $pc, x/10i $pc-12, x/10i main b *0x4001f000, x/20w array, stepi (si), nexti (ni),

Note: display/5i $pc WILL FILL IN

Library Symbols and GDB

WILL FILL IN

Debugging inside run-time libraries such as libc.so

You can find out if your system has debug symbols for the various libraries by looking in: /usr/lib/debug/lib or possibly: /usr/lib/debug/*/lib

The gdb debugger has been set up to automatically detect debug libraries if they are present. If they are present, you can immediately skip to setting up the gdb source directory to point to the source. (cd /tmp/gene; tar xf /usr/src/glibc/eglibc-2.13.tar.xz ) Create the following program as tmp.c: 

int main() {
  sleep(2);
  return 0;
}
Then compile it with debugging: 
gcc -g -O0 tmp.c
If you don't have a debug library for libc, you will see the following. (This is using gdb-7.3. Note that 'step' does not allow you to step inside the 'sleep' function in libc.so.) 
gdb a.out
...
(gdb) break main
(gdb) run
Starting program: /tmp/gene/a.out 

Breakpoint 1, main () at tmp.c:2
warning: Source file is more recent than executable.
2	  sleep(2);
(gdb) step
3	  return 0;
Next, install the debug version of libc: 
sudo apt-get install libc6-dbg
Now, we can step inside the 'sleep' function and see the line number. But we can't yet see the source code of sleep.s. 
(gdb) where
#0  __sleep (seconds=2) at ../sysdeps/unix/sysv/linux/sleep.c:43
#1  0x0000000000400507 in main () at tmp.c:2
(gdb) frame 0
#0  __sleep (seconds=2) at ../sysdeps/unix/sysv/linux/sleep.c:43
43	in ../sysdeps/unix/sysv/linux/sleep.c
(gdb) list
38	../sysdeps/unix/sysv/linux/sleep.c: No such file or directory.
	in ../sysdeps/unix/sysv/linux/sleep.c
The Debian package eglibc-source contains the source code for the glibc used in Debian (libc.so.6). Install that first. (For other libraries, there are ways to get the source code, as modified by Debian.) Next, unpack it. I'll unpack it into /tmp/gene. 
sudo apt-get install eglibc-source
(cd /tmp/gene; tar xf /usr/src/glibc/eglibc-2.13.tar.xz )
So, we set the gdb search path for directoriss containing source code. Note that we are looking for the relative path ../sysdeps/unix/sysv/linux/sleep.c. So, the search path must contain a prefix consistent with ../sysdeps/...: 
(gdb) dir /tmp/gene/eglibc-2.13/sysdeps/
Source directories searched: /tmp/gene/eglibc-2.13/sysdeps:/tmp/gene/eglibc-2.13:$cdir:$cwd
(gdb) frame 0
#0  __sleep (seconds=2) at ../sysdeps/unix/sysv/linux/sleep.c:43
43	{
(gdb) list
38	/* We are going to use the `nanosleep' syscall of the kernel.  But the
39	   kernel does not implement the stupid SysV SIGCHLD vs. SIG_IGN
40	   behaviour for this syscall.  Therefore we have to emulate it here.  */
41	unsigned int
42	__sleep (unsigned int seconds)
43	{
44	  const unsigned int max
45	    = (unsigned int) (((unsigned long int) (~((time_t) 0))) >> 1);
46	  struct timespec ts;
47	  sigset_t set, oset;
Note that in the dir command above, tab completion works as you type the directory pathname.

Further Information

If you need to debug inside a general Debian package (not necessarily a library), you can do it as described here. (But don't do this for a common library like libc or libpthread. You could corrupt your system. Use the prepackaged debug libraries as discussed above.)

For further information, try: