How to Use GDB
What GDB is and why you need to use it
GDB is a debugger. The fundamental point of a debugger is to stop and inspect the state of a running program. This helps you analyze the behavior of your program, and thus helps diagnose bugs or mistakes. With GDB you can (in general) do the following things:
- Control aspects of the environment that your program will run in.
- Start your program or connect to an already-running program.
- Make your program stop for inspection.
- Step through your program one line at a time or one machine instruction at a time.
- Inspect the state of your program once it has stopped.
- Change the state of your program and then allow it to resume execution.
In your previous programming experience, you may have managed without using a debugger. You might have been able to find the mistakes in your programs by printing things on the screen or simply reading through your code. Beware, however, that OS/161 is a large and complex body of code, much more so than you may have worked on in the past. To make matters worse, you didn't write most of it. A debugger is an essential tool in this environment. We would not lie if we said that there has never been a student developing in OS161 who has survived the class without using GDB. You should, therefore, take the time to learn GDB and make it your best friend. (Or rather, your second best friend; your best friend should be your partner.) This guide will explain to you how to get started debugging OS/161, describe the most common GDB commands, and suggest some helpful debugging techniques.
First, make sure you are able to start the debugger, by following these steps.
At this point, if everything worked properly, GDB will connect to
your kernel and tell you that the target program is stopped in
A first GDB run
This first time, you aren't hunting a specific bug, you're just
trying things out; so a good place to put a breakpoint is
(gdb) break kmain
Now continue:
(gdb) c
and GDB will stop again in
(gdb) s
This will take you to the first line of boot, which is a call
to You'll notice that even though there are several calls to When you step over Now tell GDB to execute through to the end of boot:
It will come to a stop again on the second line of kmain, which
calls the kernel menu code. You can step into this to see how the menu
works, but it's actually not all that interesting. So instead, put a
breakpoint on the Now the kernel will print the menu prompt waiting for input. Run the poweroff utility, by typing (in the other window):
If you step through this you'll see that after attending to various
shutdown tasks, it calls While you often know that you want to run in the debugger when
starting the kernel, in which case you use the method above, this
isn't always the case. You can attach GDB to System/161 at any time by
simply running it and entering the target remote line. This connects
to System/161's debugger port, and that causes System/161 to stop.
System/161 will itself also sometimes stop and wait for a debugger
connection. This will happen on various hardware-level conditions that
are not normally supposed to happen; it will also happen if the kernel
executes a breakpoint instruction or requests the debugger explicitly
via the ltrace device. And it will happen if you turn System/161's
progress monitoring on (see the System/161 manual for an explanation)
and the kernel fails to make progress in the specified amount of
time. In these cases you attach the debugger in the same way: by
running it and entering the target remote line.
Importantly, you can also make System/161 stop and wait for the
debugger by pressing On startup, GDB reads the file Note that the first time you add something
to It is also often useful to put common breakpoint settings
in For reasons we don't yet understand (if we understood them, we have
have fixed this), sometimes connecting the debugger while System/161
is running (rather than waiting for a debugger connection) doesn't
work. Also sometimes, but not usually the same times, pressing Also note that if GDB is already connected, and you tell it to
continue and the kernel doesn't stop again when you expected, getting
it to stop so you can do more with GDB can be annoyingly
difficult. Theoretically at this point pressing When you are done debugging, you can disconnect the debugger from System/161 (and thus the running kernel) in two ways:
will unhook the debugger and leave the kernel still running, whereas
will unceremoniously nuke System/161, much as if you'd gone to its
window and typed Here are the most common and often-used commands for controlling the execution of the kernel in GDB:
will execute the next line of code. If the next line is a function
call, the debugger will step into this function.
to just continue executing. Execution continues until something
happens that stops it, such as hitting a breakpoint, or if you type ^G
into System/161. (See above for some notes on ^G.)
means that your program will stop every time it executes a
statement on line 18. As with the "list" command, you can specify
break on a function, e.g.:
you will delete the breakpoint numbered 1. GDB displays the number
of a breakpoint when you set that breakpoint. Typing "d" without
arguments deletes all breakpoints.
which makes breakpoint 3 only happen when the expression The chief purpose of a debugger is to inspect the target
program. Here are the most common/useful commands for doing that.
will display line 101 in Note: If you need to debug in start.S (or any other assembler file)
GDB can't find the sources by default. This is because of a glitch in
the toolchain: the paths to the assembler source files are relative,
not absolute, so they only work by default in the kernel build
directory. To work around it, you can tell GDB to start looking for
the sources in the kernel build directory:
for whatever compilation directory was used for your current kernel
(it's the same as the name of the config string you supplied when
configuring the kernel). Tip: you can put this in to print the value of name. You can use arbitrarily complex C
expressions; it is often useful for example to include
typecasts. Normally the value will be printed according to its type;
you can insert
insert Nowadays GDB understands the concept of multiple threads; you can
switch from one thread to another while debugging in order to inspect
all the available execution contexts. The good news is: this now works
with System/161. The not so good news is: GDB threads map to CPUs, not
to OS/161 kernel threads. This means that while you can inspect all
the CPUs that are executing, you still can't easily inspect sleeping
threads. Unfortunately, GDB's thread model combined with the fact that
the debugger talks transparently to System/161 makes this mapping more
or less the only choice.
When stopped in GDB, list the threads like this:
The leftmost column is the thread number that you need to use while
talking to GDB, and the CPU number listed is the System/161 CPU
number. Unfortunately the GDB thread numbers are offset by 1 at the
start and may diverge further over time at GDB's whim. (The "Thread
13" number is the number used in the communications channel between
GDB and System/161; you don't need to care about it. It's offset from
the CPU number by 10 because if you use 0 GDB dumps core.)
To switch to another CPU, use the "thread" command with the number from the leftmost "Id" column:
GDB has a curses-based "text user interface" (TUI) that gives you a
slightly less 1980 experience when debugging. This gives you a
split-screen window with the current source file in the top part and
the GDB prompt in the bottom part. Type ^X a (that's control-X
followed by the letter a) after starting GDB to enable it, and type
that again to make it go away.
In TUI mode most GDB commands work as before; there are just a couple of keystrokes you need to know about:
^L - redraw screen (like in many programs).
^X o - switch to other panel in the GDB window (like in
Emacs) By default the current panel is the source listing, but typing
goes into the GDB prompt panel anyway; however, to use the arrow keys
for input editing you need to switch to the GDB prompt panel.
You can get an even nicer gdb interface by running it inside emacs
(a text editor). To bring it up, start sys161 waiting for the debugger
connection, as usual, then, in another window run emacs by typing:
or
depending on whether you have graphics working in your environment. Then, type:
You will then see the following line at the bottom of the emacs
screen (in the control pane): If you press Voila! You are running os161-gdb within emacs. From then on, you
use gdb as you normally would. For example, try typing the The installation is done!
If you get an error executing the above commands, close the
terminal and reopen it again. Then change to the os161 root directory
as indicated above.
with the following:
Remember the db rule in:
A new buffer window will appear. You can switch between buffer
windows using where So how do you actually approach debugging? When you have a bug in a
program, it means that you have a particular belief about how your
program should behave, and somewhere in the program this belief is
violated. For example, you may believe that a certain variable should
always be 0 when you start a "for" loop, or a particular pointer can
never be NULL in a certain "if statement". To check such beliefs, set
a breakpoint in the debugger at a line where you can check the
validity of your belief. And when your program hits the breakpoint,
ask the debugger to display the value of the variable in question.
If you have a situation where a variable does not have the value you
expect, and you want to find a place where it is modified, instead of
walking through the entire program line by line, you can check the
value of the variable at several points in the program and narrow down
the location of the misbehaving code.
Steve Maguire (the author of Writing Solid Code) recommends using the
debugger to step through every new line of code you write, at least
once, in order to understand exactly what your code is doing. It helps
you visually verify that your program is behaving more or less as
intended. With judicious use, the step, next and finish commands can
help you trace through complex code quickly and make it possible to
examine key data structures as they are built.
We highly encourage you to use GDB in a graphical mode, i.e., within the emacs editor as described above, because this will make your debugging experience a lot more efficient!
A lot of programmers like to find mistakes in their programs by
inserting "printf" statements that display the values of the
variables. If you decide to resort to this technique, you have to keep
in mind two things: First, because adding printfs requires a
recompile, printf debugging may take longer overall than using a
debugger.
More subtly, if you are debugging a multi-threaded program, such as
a kernel, the order in which the instructions are executed depends on
how your threads are scheduled, and some bugs may or may not manifest
themselves under a particular execution scenario. Because printf
outputs to the console, and the console in System/161 is a serial
device that isn't extraordinarily fast, an extra call to printf may
alter the timing and scheduling considerably. This can make bugs hide
or appear to come and go, which makes your debugging job much more
difficult.
A deadlock occurs when two or more threads are all waiting for the
others to proceed. (We'll talk more about this in lecture.) In a
simple deadlock there'll be one thread T1 that already holds a lock
L1, and another thread T2 that already holds a lock L2. T1 is waiting
to get L2, so it can't proceed until T2 runs, and T2 is waiting to get
L1, so it can't proceed until T1 runs. The goal of debugging a
deadlock is to find out the identities of T1 and T2, and of L1 and L2,
and then figure out what combination of circumstances and code paths
allowed them to get into this state.
To do this, you generally want to begin by finding one of the
threads, since there are usually a lot more locks in the system than
threads. Each thread that's blocked reports what it's blocked on in
the field t_wchan_name. The basic procedure is to locate a thread (the
first ones to start with are the ones currently on the CPUs; after
that look in your process table that you wrote for assignment 2),
check what it's waiting for, then find who's holding that. If it's
waiting for a lock, hopefully the locks you wrote record who's holding
them. That gives you another thread to examine; repeat until you find
a loop. If it's waiting on a CV, it gets a bit more interesting
because you need to figure out who was supposed to signal that CV and
didn't... or if they did, why the waiting thread didn't receive the
signal. For deadlocks involving spinlocks, remember that in OS/161
spinlocks are held by CPUs, not threads.
Once you have the participants identified, the stack traces from
those threads will usually (but, alas, not always) give you enough
information to work out what happened.
It is often helpful to set up a global table of all locks and/or
CVs (OS/161 already comes with a global table of all wchans) to make
this process easier. It is also possible, and potentially worthwhile,
to hack more state information into the thread structure. One can even
write an automatic deadlock detector, although this probably has a bad
cost/benefit ratio and won't help you much when CVs are involved.
When GDB stops in an assembly source file (a .S file) various
special considerations apply. GDB is meant to be a source-level
debugger for high level languages and isn't very good at debugging
assembly. So various tricks are required to get adequate results. The OS/161 toolchain tells the assembler to emit line number
information for assembly files, but as noted above (under "list") you
need to tell GDB how to find them. Do that. Then you can at least see
the file you're working on.
It is also sometimes helpful to disassemble the kernel; type
To single step through assembler, use the The command Print register values with The trace161 program is the same as sys161 but includes additional
support for various kinds of tracing and debugging operations. You can
have System/161 report disk I/O, interrupts, exceptions, or
whatever. See
the System/161
documentation for more information.
One of the perhaps more interesting trace options is to have
System/161 report every machine instruction that is executed, either
at user level, at kernel level, or both. Because this setting
generates vast volumes of output, it's generally not a good idea to
turn it on from the command line. (It is sometimes useful, however, in
the early stages of debugging assignment 2 or 3, to log all user-mode
instructions.) However, the trace options can be turned on and off
under software control using the System/161 trace control device. It
can be extremely useful to turn instruction logging on for short
intervals in places you suspect something strange is happening. See
If you have a When you get a stack backtrace and it reaches an exception frame,
GDB can sometimes now trace through the exception frame, but it
doesn't always work very well. Sometimes it only gets one function
past the exception, and sometimes it skips one function. (This is a
result of properties of the MIPS architecture and the way GDB is
implemented and doesn't appear readily fixable.) Always check the
(gdb) n
to execute the
(gdb) list
(gdb) finish
(gdb) break sys_reboot
(gdb) c
OS/161 kernel [? for menu]: p /sbin/poweroff
Connecting and disconnecting GDB
define db
target remote unix:.sockets/gdb
end
(gdb) db
(gdb) detach
(gdb) kill
Executing under GDB control
(gdb) s
(gdb) c
(gdb) b 18
(gdb) b main
will stop when execution encounters the main function.
(gdb) d 1
(gdb) cond 3 count > 1000
(gdb) command 2
> printf "theString = %s\n", theString
> print /x x > end
Inspecting stuff with GDB
(gdb) l file.c:101
(gdb) directory ~/os161/src/kern/compile/DUMBVM
(gdb) p name
When you print something, you'll get a result line like this:
$1 = 0x80063d80 ""
The $1 is a convenience variable, a scratch variable created and
remembered by GDB for your subsequent use. You can use $1 as a value
in subsequent expressions and it will retrieve the value that was
printed; this saves having to retype it or cut and paste it. (Note
though that it saves the value, not the expression; if you execute for
a while and then use $1 again it will have the same value regardless
of what's happened in the meantime to the values in the expression
that generated it.
(gdb) printf "X = %d, Y = %d\n",X,Y
(gdb) display abc
If there is no abc in scope when the program stops, the display won't
appear. This command is otherwise the same as "print", including the
support for type modifiers.
(gdb) set variable x = 15
will change the value of "x" to 15.
Other GDB commands to note
Note that several of the commands listed above (e.g. "info
breakpoints") are specific instances of more general GDB commands,
like "info" and "set".
(gdb) help
GDB threads support
(gdb) info threads
This will give a display something like this:
Id Target Id Frame
4 Thread 13 (CPU 3) wait () at ../../arch/mips/thread/cpu.c:279
3 Thread 12 (CPU 2) wait () at ../../arch/mips/thread/cpu.c:279
2 Thread 11 (CPU 1) wait () at ../../arch/mips/thread/cpu.c:279
* 1 Thread 10 (CPU 0) runprogram (progname=0x800f3f40 "/bin/sh") at ../../syscall/runprogram.c:492
(gdb) thread 1
The GDB "text user interface"
The GDB emacs user interface
emacs
emacs -nw
M-x gdb
Run gdb (like this): os161-gdb -i=mi kernel
The GDB vi or vim user interface
You can also run GDB with vim by
installing Conque-GDB
plugin:
wget "http://www.vim.org/scripts/download_script.php?src_id=22163" -O conque_gdb.vmb
:so %
:q
jharvard@appliance (~/os161/root): vim
:ConqueGdbExe os161-gdb
:ConqueGdb
set shell=/bin/sh
if !exists('g:ConqueGdb_GdbExe')
let g:ConqueGdb_GdbExe = ''
endif
if !exists('g:ConqueGdb_GdbExe')
let g:ConqueGdb_GdbExe = 'os161-gdb'
endif
:ConqueGdb
file kernel
db
jharvard@appliance (~/os161/root): cat ~/.gdbinit
set auto-load safe-path /
define db
directory ~/os161/src/kern/compile/DUMBVM
target remote unix:.sockets/gdb
end
jharvard@appliance (~/os161/root): sys161 -w kernel
jharvard@appliance (~/os161/root): grep "mapleader" ~/.vimrc
let mapleader = ","
Debugging tips
Tip #1: Check your beliefs about the program
Tip #2: Narrow down your search
Tip #3: Walk through your code
Tip #4: Use good tools
Tip #5: Beware of printfs!
Tip #6: Debugging deadlocks
Tip #7: Debugging assembly
% os161-objdump -d kernel | less
in another window and page or search through it as needed.
Tip #8: trace161
Tip #9: casting void pointers
Tip #10: tracing through exceptions
Where to go for help
For help on GDB commands, type "help" from inside GDB. You can find
the GDB
manual here. Your
friendly TAs are always there to help, but please be sure that you
tried running the commands and applying the tips presented in this
document before enlisting TA help!