Custom kernels

Debug options

FreeBSD is a very stable operating system. No software is perfect, however, and sometimes it crashes. When it does, it provides a number of facilities to help fix the problem. Some of these are dependent on kernel build options.

Even if you have no intention of debugging a kernel problem yourself, you should set debug symbols when you build a kernel. They cost nothing except disk space, and if you are short on disk space, you can remove most of the files after the build.

To set the debug symbols, remove the leading # mark from this line in the configuration file:

makeoptions  DEBUG=-g   # Build kernel with gdb(1) debug symbols

Under normal circumstances this makes no difference: the build process still installs the kernel without the debug symbols, and it has no effect on performance. If, however, you have a crash, the kernel with debug symbols is available in the kernel build directory, in this case /usr/src/sys/i386/compile/FREEBIE/kernel.debug, to assist analysis of the problem. Without this file it will be very difficult to find it.

So why is it commented out? Without debug symbols, your build directory will take about 50 MB of disk space. With debug symbols, it will be about 250 MB. The FreeBSD Project couldn't agree to change it.

Other debugging options

If you run into trouble with your system, there are a number of other debugging options that you can use. The following are the more important ones:

options   DDB
options   BREAK_TO_DEBUGGER
options   DDB_UNATTENDED      # Don't drop into DDB for a panic
options   GDB_REMOTE_CHAT     #Use gdb remote debugging protocol
options   KTRACE
options   DIAGNOSTIC
options   INVARIANTS
options   INVARIANT_SUPPORT
options   WITNESS           #Enable checks to detect deadlocks and cycles
options   WITNESS_SKIPSPIN     #Don't run witness on spinlocks for speed
These options provide support for various debugging features.

DDB

Specify DDB to include the kernel debugger, ddb. If you set this option, you might also want to set the BREAK_TO_DEBUGGER option,

BREAK_TO_DEBUGGER

Use the option BREAK_TO_DEBUGGER if you have installed the kernel debugger and you have the system console on a serial line.

DDB_UNATTENDED

If you have a panic on a system with ddb, it will not reboot automatically. Instead, it will enter ddb and give you a chance to examine the remains of the system before rebooting. This can be a disadvantage on systems that run unattended: after a panic, they would wait until somebody comes past before rebooting. Use the DDB_UNATTENDED option to cause a system with ddb to reboot automatically on panic.

GDB_REMOTE_CHAT

ddb supports remote debugging from another FreeBSD machine via a serial connection. See the online handbook for more details. To use this feature, set the option GDB_REMOTE_CHAT.

KTRACE

Set KTRACE if you want to use the system call trace program ktrace .

DIAGNOSTIC

A number of source files use the DIAGNOSTIC option to enable extra sanity checking of internal structures. This support is not enabled by default because of the extra time it would take to check for these conditions, which can only occur as a result of programming errors.

INVARIANTS and INVARIANT_SUPPORT

INVARIANTS is used in a number of source files to enable extra sanity checking of internal structures. This support is not enabled by default because of the extra time it would take to check for these conditions, which can only occur as a result of programming errors.

INVARIANT_SUPPORT option compiles in support for verifying some of the internal structures. It is a prerequisite for INVARIANTS. The intent is that you can set INVARIANTS for single source files (by changing the source file or specifying it on the command line) if you have INVARIANT_SUPPORT enabled.

WITNESS and WITNESS_SKIPSPIN

One of the big changes in FreeBSD Release 5 relates to the manner in which the kernel performs resource locking. As a result, the danger exists of deadlocks, locks that can'tbe undone without rebooting the machine. WITNESS checks for the danger of deadlocks and warns if it finds a potential deadlock ("lock order reversal"). This is a very expensive debugging option: it can slow the machine down by an order of magnitude, so don't use it unless you have to.

A compromise that doesn't use quite so much processor power is the combination of WITNESS with WITNESS_SKIPSPIN, which avoids spin locks. It can still catch most problems.

Preparing for upgrades

When changing the configuration file, consider that it probably won't be the only time you make these changes. At some time in the future, you'll upgrade the system, and you'll have to build your custom kernel all over again. But the GENERIC kernel configuration file will have changed as well. You have two choices: incorporate the modifications to GENERIC into your configuration file, or incorporate your modifications to the old GENERIC file into the new GENERIC configuration file. It turns out that the latter path is easier.

To prepare for this approach, try to change as little as possible in the body of the configuration file. Instead, add all your changes to the end, along with a comment so that you can readily recognize the changes. For example, you might append:

# Added by grog, 24 October 2002
# Comment out WITNESS, WITNESS_SKIPSPIN and SCSI_DELAY above
options    BREAK_TO_DEBUGGER
options    SCSI_DELAY=3000    #Delay (in ms) before probing SCSI
options    CAMDEBUG
options    MSGBUF_SIZE=81920
#options   TIMEREQUESTS      #watch for delays
device     snp               #Snoop device - to look at pty/vty/etc.

That won't be all, of course. Look at that option SCSI_DELAY.That option already exists in the configuration file (with a value of 15 seconds instead of 3). If you leave both, config will issue a warning. You need to comment out the first occurrence, as the comment at the top indicates.