FreeBSD Operating System

Custom kernels

Разбить на страницы
Показывать лекцию целиком

So far, everything we've done has been with the standard GENERIC kernel distributed with FreeBSD. You may find it an advantage to install a custom kernel:

  • As we saw in Chapter 2, GENERIC doesn't support everything that FreeBSD knows about. For example, if you want to install a Yoyodyne frobulator, you'll need to install special support for it In fact, the developer working on the Yoyodyne has defected to the Free Software Foundation. See the GNU General Public License for further details .
  • It will take less time to boot because it does not have to spend time probing for hardware that you do not have.
  • A custom kernel often uses less memory. The kernel is the one system component that must always be present in memory, so unused code ties up memory that would otherwise be available to the virtual memory system. On a system with limited RAM, you can save some memory by building a custom kernel, but don't overestimate the savings: a minimal kernel might save 500 kB over the GENERIC kernel supplied with the system.
  • In addition, there are several kernel options that you can tune to fit your needs.
  • Finally, on page 622 we'll look at things to think about if you want to run the -CURRENT version of FreeBSD.
  • In older releases of BSD, you needed to build a new kernel for just about anything you wanted to change, even things as simple as a different IRQ for a device. FreeBSD has evolved quite a bit since then, and it's becoming increasingly less necessary to build a custom kernel. You will certainly need to do so if you want to enable kernel-wide options, such as extra consistency checking, but in many cases you have more flexible alternatives:

  • If you just need to add device support, you may be able to load a Kernel Loadable Module, or kld. See page 620 for more information on klds.
  • If you want to change ISA parameters such as I/O address, IRQ or DMA settings, you no longer need to build a new kernel: the kernel configuration file no longer knows about such parameters. Instead, they're in the file /boot/device.hints, that we'll look at below.
  • A number of kernel options have been replaced by the sysctl interface. For example, the GENERIC kernel does not perform packet routing by default. In older releases of FreeBSD, you had to build a new kernel with the option GATEWAY. Nowadays you can turn this feature on and off at will with a sysctl command. We'll look at sysctls on page
  • Configuring a kernel has changed a lot since the early days of FreeBSD, but it's not done yet. The information in this chapter represents a snapshot in the evolution of building kernels. The goal of these changes is to make it unnecessary to build a kernel at all except to upgrade to a new release.

    Building a new kernel

    FreeBSD is distributed in source, and building a kernel primarily involves compiling the source files needed for the kernel. To build a kernel, you perform the following steps:

  • Install the system source, if you haven't already done so. We looked at that in Chapter 31.
  • Define your kernel configuration in a kernel configuration file. This file defines parameters to use during the build process. We'll look at how to do this starting on page .
  • Change to the directory /usr/src and run make kernel. This builds and installs the kernel and all modules. We'll look at it in more detail on page 617, where we'll also see alternatives that give more control over the process.
  • Configuring I/O devices

    A lot of the configuration file relates to the I/O devices that you may connect to your machine. In older releases of FreeBSD, it was often necessary to specify some of the IRQ, DMA channel, board memory, and I/O addresses for the devices you configure, particularly for ISA boards. Since Release 5 of FreeBSD, this is no longer the case. Instead, you modify the file boot/device.hints, which we looked at on page 575.

    The kernel installation does not install /boot/device.hints automatically. If it doesn't exist, copy it from the configuration directory:

    # cp -p /usr/src/sys/i386/conf/GENERIC.h±nts /boot/device.hints
    

    The kernel build directory

    The kernel sources are kept in the directory /usr/src/sys. The symbolic link /sys also points to this directory. There are a number of subdirectories of /usr/src/sys that represent different parts of the kernel, but for our purposes, the most important of them are the architecture dependent directories such as /usr/src/sys/i386/conf (for the i386 architecture), /usr/src/sys/alpha/conf (for the Alpha architecture), or /usr/src/sys/sparc64/conf (for the SPARC64 architecture) where you edit your custom kernel configuration. In addition, the old style kernel build described below builds the kernel in the directory /usr/src/sys/i386/compile, /usr/src/sys/alpha/compile or /usr/src/sys/sparc64/compile respectively. Notice the logical organization of the directory tree: each supported device, file system, and option has its own subdirectory. In the rest of this chapter, we'll look at the i386 architecture. Most of this applies to other architectures as well.

    If your system doesn't have the directory /usr/src/sys, the kernel source has not been installed. If you have a CD-ROM, the sources are on the first CD-ROM in the directory /src. To install from the CD-ROM, perform the following steps:

    # mkdir -p /usr/src/sys
    # ln -s /usr/src/sys /sys
    # cd /
    # cat /cdrom/src/ssys.[a-h]* | tar xzvf -
    

    The symbolic link /sys for /usr/src/sys is not strictly necessary, but it's a good idea: some software uses it, and otherwise you may end up with two different copies of the sources.

    By definition, the files on CD-ROM are out of date. See Chapter 31 for details of how to get the current, up-to-date sources.

    Next, move to the directory i386/conf and copy the GENERIC configuration file to the name you want to give your kernel. For example:

    # cd /usr/src/sys/i386/conf
    # cp GENERIC FREEBIE
    

    Traditionally, this name is in all capital letters and, if you are maintaining multiple FreeBSD machines with different hardware, it's a good idea to name it after your machine's hostname. In this example we call it FREEBIE.

    Now, edit FREEBIE with your favourite text editor. Change the comment lines at the top to reflect your configuration or the changes you've made to differentiate it from GENERIC:

    #
    # FREEBIE -- My personal configuration file
    #
    # For more information on this file, please read the handbook section on
    # Kernel Configuration Files:
    #
    # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
    #
    # The handbook is also available locally in /usr/share/doc/handbook
    # ifyou've installed the doc distribution, otherwise always see the
    # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
    # latest information. 
    #
    # Anexhaustive list of options and more detailed explanations of the
    # device lines is also present in the ../../conf/NOTES and NOTES files.
    # Ifyou are in doubt as to the purpose or necessity of a line, check first
    # inNOTES.
    #
    # $FreeBSD: src/sys/i386/conf/FREEBIE,v 1.369 2002/10/19 16:54:07 rwatson Exp $
    machine    "i386"
    cpu        "I486_CPU"
    cpu       "I586_CPU"
    cpu       "I686_CPU"
    ident      FREEBIE
    maxusers   0
    

    The configuration file

    The directory /sys/i386/conf contains a number of configuration files:

  • generic General-purpose configuration file
  • lint This file used to be a "complete" configuration file with comments, used for testing and documentation. Since FreeBSD Release 5, it no longer exists. Youcan create it from the files NOTES and /usr/src/sys/conf/NOTES with the command:
    $ make LIN
    
  • notes A complete pseudo-configuration file with copious comments. This file is descended from LINT, but it also includes device hints. You can't use it for building kernels. Instead, create the file LINT as described above. NOTES contains only platform-specific information. Most of the information is in the platform-independent file /usr/src/sys/conf/NOTES.
  • oldcard A configuration file for laptops that use PCCARD controllers. At the time of writing, PCCARD support has largely been rewritten, but the new code does not support some of the devices that the old code supports. This configuration file uses the old PCCARD code instead of the new code. When the new code is complete, it will go away.
  • The general format of a configuration file is quite simple. Each line contains a keyword and one or more arguments. Anything following a # is considered a comment and ignored. Keywords that contain numbers used as text must be enclosed in quotation marks.

    One of the results of this simplicity is that you can put in options that have absolutely no effect. For example, you could add a line like this:

    options APPLE_MAC_COMPATIBILITY
    

    You can build a kernel with this option. It will make no difference whatsoever. Now it's unlikely that you'll think up a non-existent option like this, but it's much more possible that you'll misspell a valid option, especially finger-twisters like SYSVSHM, with the result that you don't compile in the option you wanted. The config program warns if you use unknown options, so take these warnings seriously.

    Kernel options change from release to release, so there's no point in describing them all here. In the following sections we'll look at some of the more interesting ones; for a complete list, read LINT or the online handbook. See above for details of how to create LINT.

    Naming the kernel

    Every kernel you build requires the keywords machine, cpu, and ident. For example,

    machine  "i386"       For 386 architecture
    machine  "alpha"      For alpha architecture
    machine  "sparc64"    For PARC 64 architecture
    cpu      "I486_CPU"
    cpu      "I586_CPU"
    cpu      "I686_CPU"
    ident     FREEBIE
    

    machine

    The keyword machine describes the machine architecture for which the kernel is to be built. Currently it should be i386 for the Intel architecture, and alpha for the AXP architecture. Don't confuse this with the processor: for example, the i386 architecture refers to the Intel 80386 and all its successors, including lookalikes made by AMD, Cyrix and IBM.

    cpu cpu_type

    cpu describes which CPU chip or chips this kernel should support. For the i386 architecture, the possible values are I386_CPU, I486_CPU, I586_CPU and I686_CPU, and you can specify any combination of these values. For a custom kernel, it is best to specify only the CPU you have. If, for example, you have an Intel Pentium, use I586_CPU for cpu_type.

    If you're not sure what processor type you have, look at the output from the dmesg command when running the GENERIC kernel. For example:

    CPU: AMD Athlon(tm) XP processor 1700+ (1462.51-MHz 686-class CPU)
    Origin = "AuthenticAMD"    Id = 0x662   Stepping = 2
    Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,^OV,PAT,PS E36,MMX,FXSR,SSE>
    AMD Features=0xc0480000<<b19>,AMIE,DSP,3DNcw!>
    

    This shows that the processor, an AMD Athlon XP, is a "686 class" CPU, so to run a kernel on this processor, you must set I686_CPU in the config file.

    Since Release 5 of FreeBSD, it is no longer possible to build a single kernel with support for both the 80386 processor and later processors: the code for the later processors is optimized to use instructions that the 80386 processor does not have. Choose either I386_CPU or any combination of the others.

    ident machine_name

    ident specifies a name used to identify the kernel. In the file GENERIC it is GENERIC. Change this to whatever you named your kernel, in this example, FREEBIE. The value you put in ident will print when you boot up the kernel, so it's useful to give a kernel a different name if you want to keep it separate from your usual kernel (if you want to build an experimental kernel, for example). As with machine and cpu, enclose your kernel's name in quotation marks if it contains any numbers.

    This name is passed to the C compiler as a variable, so don't use names like DEBUG, or something that could be confused with another machine or CPU name, like vax.

    Kernel options

    There are a number of global kernel options, most of which you don't need to change. In the following section, we'll look at some of the few exceptions.

    Configuring specific I/O devices

    There are some devices that the GENERIC kernel does not support. In older releases of FreeBSD, you needed to build a new kernel to support them. This is seldom the case any more: most devices can be supported by klds. Work is under way to support the remainder. In case of doubt, look at the file HARDWARE.TXT on the installation CD-ROM.

    maxusers number

    This value sets the size of a number of important system tables. It is still included in the kernel configuration file, but you no longer need build a new kernel to change it. Instead, you can set it at boot time. For example, you might add the following line to your /boot/loader.conf file:

    maxusers="64"
    

    See 532 for more details of /boot/loader.conf.

    maxusers is intended to be roughly equal to the number of simultaneous users you expect to have on your machine, but it is only used to determine the size of some system tables. The default value 0 tells the kernel to autosize the tables depending on the amount of memory on the system. If the autosizing doesn' t work to your satisfaction, change this value. Even if you are the only person to use the machine, you shouldn't set maxusers lower than the default value 32, especially if you're using X or compiling software. The reason is that the most important table set by maxusers is the maximum number of processes, which is set to 20 + 16 * maxusers, so if you set maxusers to one, you can only have 36 simultaneous processes, including the 18 or so that the system starts up at boot time, and the 15 or so you will probably create when you start X. Even a simple task like reading a man page can start up nine processes to filter, decompress, and view it. Setting maxusers to 32 will allow you to have up to 532 simultaneous processes, which is normally ample. If, however, you see the dreaded proc table full error when trying to start another program, or are running a server with a large number of simultaneous users, you can always increase this number and reboot.

    maxusers does not limit the number of users who can log into the machine. It simply sets various table sizes to reasonable values considering the maximum number of users you will likely have on your system and how many processes each of them will be running. It's probable that this parameter will go away in future.

    Multiple processors

    FreeBSD 5.0 supports most modern multiprocessor systems. Earlier versions of the GENERIC kernel did not support them by default. Nowadays GENERIC supports multiple processors, so you don't need to do anything special. Look at the option SMP and the device apic in the configuration file if you're interested.

    Debug options

    FreeBSD is a very stable operating system. No software is perfect, however, and sometimes it crashes. When it does, it provides a number of facilities to help fix the problem. Some of these are dependent on kernel build options.

    Even if you have no intention of debugging a kernel problem yourself, you should set debug symbols when you build a kernel. They cost nothing except disk space, and if you are short on disk space, you can remove most of the files after the build.

    To set the debug symbols, remove the leading # mark from this line in the configuration file:

    makeoptions  DEBUG=-g   # Build kernel with gdb(1) debug symbols
    

    Under normal circumstances this makes no difference: the build process still installs the kernel without the debug symbols, and it has no effect on performance. If, however, you have a crash, the kernel with debug symbols is available in the kernel build directory, in this case /usr/src/sys/i386/compile/FREEBIE/kernel.debug, to assist analysis of the problem. Without this file it will be very difficult to find it.

    So why is it commented out? Without debug symbols, your build directory will take about 50 MB of disk space. With debug symbols, it will be about 250 MB. The FreeBSD Project couldn't agree to change it.

    Other debugging options

    If you run into trouble with your system, there are a number of other debugging options that you can use. The following are the more important ones:

    options   DDB
    options   BREAK_TO_DEBUGGER
    options   DDB_UNATTENDED      # Don't drop into DDB for a panic
    options   GDB_REMOTE_CHAT     #Use gdb remote debugging protocol
    options   KTRACE
    options   DIAGNOSTIC
    options   INVARIANTS
    options   INVARIANT_SUPPORT
    options   WITNESS           #Enable checks to detect deadlocks and cycles
    options   WITNESS_SKIPSPIN     #Don't run witness on spinlocks for speed
    These options provide support for various debugging features.
    

    DDB

    Specify DDB to include the kernel debugger, ddb. If you set this option, you might also want to set the BREAK_TO_DEBUGGER option,

    BREAK_TO_DEBUGGER

    Use the option BREAK_TO_DEBUGGER if you have installed the kernel debugger and you have the system console on a serial line.

    DDB_UNATTENDED

    If you have a panic on a system with ddb, it will not reboot automatically. Instead, it will enter ddb and give you a chance to examine the remains of the system before rebooting. This can be a disadvantage on systems that run unattended: after a panic, they would wait until somebody comes past before rebooting. Use the DDB_UNATTENDED option to cause a system with ddb to reboot automatically on panic.

    GDB_REMOTE_CHAT

    ddb supports remote debugging from another FreeBSD machine via a serial connection. See the online handbook for more details. To use this feature, set the option GDB_REMOTE_CHAT.

    KTRACE

    Set KTRACE if you want to use the system call trace program ktrace .

    DIAGNOSTIC

    A number of source files use the DIAGNOSTIC option to enable extra sanity checking of internal structures. This support is not enabled by default because of the extra time it would take to check for these conditions, which can only occur as a result of programming errors.

    INVARIANTS and INVARIANT_SUPPORT

    INVARIANTS is used in a number of source files to enable extra sanity checking of internal structures. This support is not enabled by default because of the extra time it would take to check for these conditions, which can only occur as a result of programming errors.

    INVARIANT_SUPPORT option compiles in support for verifying some of the internal structures. It is a prerequisite for INVARIANTS. The intent is that you can set INVARIANTS for single source files (by changing the source file or specifying it on the command line) if you have INVARIANT_SUPPORT enabled.

    WITNESS and WITNESS_SKIPSPIN

    One of the big changes in FreeBSD Release 5 relates to the manner in which the kernel performs resource locking. As a result, the danger exists of deadlocks, locks that can'tbe undone without rebooting the machine. WITNESS checks for the danger of deadlocks and warns if it finds a potential deadlock ("lock order reversal"). This is a very expensive debugging option: it can slow the machine down by an order of magnitude, so don't use it unless you have to.

    A compromise that doesn't use quite so much processor power is the combination of WITNESS with WITNESS_SKIPSPIN, which avoids spin locks. It can still catch most problems.

    Preparing for upgrades

    When changing the configuration file, consider that it probably won't be the only time you make these changes. At some time in the future, you'll upgrade the system, and you'll have to build your custom kernel all over again. But the GENERIC kernel configuration file will have changed as well. You have two choices: incorporate the modifications to GENERIC into your configuration file, or incorporate your modifications to the old GENERIC file into the new GENERIC configuration file. It turns out that the latter path is easier.

    To prepare for this approach, try to change as little as possible in the body of the configuration file. Instead, add all your changes to the end, along with a comment so that you can readily recognize the changes. For example, you might append:

    # Added by grog, 24 October 2002
    # Comment out WITNESS, WITNESS_SKIPSPIN and SCSI_DELAY above
    options    BREAK_TO_DEBUGGER
    options    SCSI_DELAY=3000    #Delay (in ms) before probing SCSI
    options    CAMDEBUG
    options    MSGBUF_SIZE=81920
    #options   TIMEREQUESTS      #watch for delays
    device     snp               #Snoop device - to look at pty/vty/etc.
    

    That won't be all, of course. Look at that option SCSI_DELAY.That option already exists in the configuration file (with a value of 15 seconds instead of 3). If you leave both, config will issue a warning. You need to comment out the first occurrence, as the comment at the top indicates.

    Building and installing the new kernel

    The traditional way to build a BSD kernel was, assuming a configuration file called FREEBIE:

    # cd /usr/src/sys/i386/conf
    # config FREEBIE
    # cd ../compile/FREEBIE
    # make depend all install
    

    At the time of writing, this still works, but it will go away at some time in the future. It has the disadvantage that you need to know your architecture, and the kernel is built in the source tree, which is not a good idea. The new kernel build method starts from /usr/src, the same directory as all other build operations, and it builds the kernel in the /usr/obj hierarchy.

    You'll need about 250 MB of free space on /usr/obj to build a kernel. If you're really tight on space, you can reduce this value to about 50 MB by omitting the makeoptions DEBUG=-g specification, but if you have problems with the system at some later stage, it will be much more difficult to find what is causing them.

    There are a number of alternative commands for building the kernel:

    # cd /usr/src
    # make kernel KERNCONF=FREEBIE               build and install kernel and klds
    # make buildkernel KERNCONF=FREEBIE          build kernel and klds
    #make installkernel KERNCONF=FREEBIE         install prebuilt kernel and klds
    #make reinstallkernel KERNCONF=FREEBIE       reinstall kernel and klds
    # make kernel KERNCONF=FREEBIE -DNO_MODULES  build and install kernel only
    

    The easiest way is make kernel, which is subdivided into the buildkernel and installkernel steps. You can perform these steps individually if you want. If you use the reinstallkernel target instead of the installkernel target, the /boot/kernel.old hierarchy remains unchanged. This is useful when a previous kernel build failed to boot.

    If you know the klds are not going to change, you can speed things up by not building them again. Use the -DNO_MODULES flag in combination with any of the other targets to inhibit building or installing the klds, as shown in the last example. Don't do this the first time you build the kernel: if you have mismatches between the kernel and the klds, you may find it impossible to start the system. On the other hand, if you find the kernel doesn't boot, and you want to change the configuration file and rebuild it from the same sources, you can save some time like this.

    The first step in building the kernel is to run the config program. You no longer have to do this yourself; the buildkernel and kernel targets do it for you. config creates a directory in which to build the kernel and fills it with the necessary infrastructure, notably the kernel Makefile and a number of header files.

    It's possible to get error messages at this stage if you have made a mistake in the config file. If the config command fails when you give it your kernel description, you've probably made a simple error somewhere. Fortunately, config will print the line number that it had trouble with, so you can quickly find it with an editor. For example:

    config: line 17: syntax error
    

    One possibility is that you have mistyped a keyword. Compare it to the entry in the GENERIC or LINT kernel definitions.

    The next step is to compile all the source files and create a kernel and a set of matching klds. It can take some time, up to an hour on a slow machine. It's also possible to have errors, and unfortunately they are usually not self-explanatory. If the make command fails, it usually signals an error in your kernel description that is not obvious enough for config to catch. A common one is when you omit an entry from the configuration file which is a prerequisite for an entry which is present. For example, if you have SCSI disks (device da), you require the scbus device as well, and if you have just about any kind of Ethernet card, you require the miibus device as well:

    device   scbus   # SCSI bus (required)
    device   da      # Direct Access (disks)
    device   miibus  # MII bus support
    device   fxp     # Intel EtherExpress PRO/100B (82557, 82558)
    

    If you leave scbus or miibus out of the configuration, config will not complain, but the kernel link phase will fail with lots of unresolved references.

    If you can't resolve the problem after comparing your configuration file with GENERIC, send mail to FreeBSD-questions@FreeBSD.org with your kernel configuration, and it should be diagnosed very quickly. A description of how to interpret these errors has been in the works for a long time, but currently it's still deep magic.

    Rebooting

    Next, shutdown the system and reboot load the new kernel:

    # shutdown -r now
    

    If the new kernel does not boot, or fails to recognize your devices, don't panic. Reset the machine, and when the boot prompt appears, press the space bar to interrupt the boot. Then boot the old kernel:

    Ok unload                        remove the kernel that the loader has loaded
    Ok load /boot/kernel.old/kernel  load the previous kernel
    Ok boot
    

    When reconfiguring a kernel, it is always a good idea to keep on hand a kernel that is known to work. There are two points here:

  • If your kernel doesn't boot, you don't want to save it to kernel.old when you build a new one. It's no use, and if the new kernel doesn't boot either, you'll be left without a runnable kernel. In this case, use the reinstallkernel target mentioned above:
    # make buildkernel KERNCONF=FREEBIE      build kernel and klds
    # make reinstallkernel KERNCONF=FREEBIE  reinstall kernel and klds
    
  • You could still make a mistake and type make install, throwing away your last good kernel. It's easy enough to end up with a completely unbootable system like this. It's a good idea to keep another kernel copy with a name like kernel.save, which the installation procedure does not touch.
  • After booting with a good kernel you can check over your configuration file and try to build it again. One helpful resource is the /var/log/messages file which records, among other things, all of the kernel messages from every successful boot. Also, the dmesg command prints the most recent kernel messages from the current boot. After some time the original messages are overwritten, so the system startup procedure saves the messages at boot time in the file /var/run/dmesg.boot. Note that most laptops maintain the previous contents of the kernel message buffer when rebooted, so the beginning of the output of dmesg may relate to an earlier boot. Check through to the end before jumping to conclusions.

    Making device nodes

    FreeBSD Release 5 comes with devfs, the device file system. One great advantage of devfs is that it automatically creates device nodes for the hardware it finds, so you no longer need to run the /dev/MAKEDEV script supplied with older releases of FreeBSD.

    Kernel loadable modules

    As we saw at the beginning of the chapter, you may not have to build a new kernel to implement the functionality you want. Instead, just load it into the running kernel with a Kernel Loadable Module (kld). The directory /boot/kernel/modules contains a number of klds. To load them, use kldload. For example, if you wanted to load SCO UNIX compatibility, you would enter:

    #  kldload ibcs2
    

    This loads the module /boot/kernel/modules/ibcs2.ko. Note that you don't need to specify the directory name, nor the .ko extension.

    To find what modules are loaded, use kldstat:

    # kldstat
    Id Refs  Address      Size    Name
    1     5  0xc0100000   1d08b0  kernel
    2     2  0xc120d000   a000    ibcs2.ko
    3     1  0xc121b000   3000    ibcs2_coff.ko
    4     1  0xc1771000   e000    linux.ko
    5     1  0xc177f000   bf000   vinum.ko
    

    You can also unload some klds, but not all of them. Use kldunload for this purpose:

    #  kldunload vinum
    

    You can't unload a kld which has active resources. In the case of vinum, for example, you can only unload it when none of its volumes are mounted.

    sysctl

    sysctl is a relatively new kernel interface that allows access to specific variables in the kernel. Some of these variables are read-only: you can look, but not touch. Others are changeable.

    sysctl variables are usually referred to simply as sysctls. Each sysctl has a name in "Management Information Base"(MIB) form, consisting of a hierarchical arrangement of names separated by periods (.). The first component of the name indicates the part of the kernel to which it relates. The following examples give you an idea of how to use the sysctl program:

    $ sysctl kern.ostype
    FreeBSD
    $ sysctl kern                      list all sysctls starting with kern
    $ sysctl -a                        list all sysctls
    # sysctl net.inet.ip.forwarding=1  turn IP forwarding on
    net.inet.ip.forwarding: 0 -> 1
    

    Some of the more interesting sysctls are:

    kern.ostype: FreeBSD kern.osrelease: 5.0-RELEASE
    kern.version: FreeBSD 5.0-RELEASE #0: Thu Jan 16 15:03:31 CST 2003
    grog@freebie.example.org:/usr/src/sys/GENERIC kern.hostname: freebie.example.org
    kern.boottime:  { sec = 1007165073, usec = 570637 } Fri Jan 17 10:34:33 2003 
    kern.bootfile: /boot/kernel/kernel
    kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall
    kern.module_path: /boot/kernel;/boot/kernel;/boot/modules;/modules 
    kern.coredump: 1 
    kern.corefile: /var/tmp/%N.core
    kern.msgbuf: nreach TCP 213.46.243.23:25370 139.130.136.138:25 in via ppp0
    net.inet.ip.fw.enable: 1 hw.machine: i386
    hw.model: Pentium II/Pentium II Xeon/Celeron
    hw.ncpu: 1
    hw.byteorder: 1234
    hw.physmem: 129949696
    hw.usermem: 100556800
    hw.pagesize: 4096
    hw.floatingpoint: 1
    hw.machine_arch: i386
    hw.ata.ata_dma: 1
    hw.ata.wc: 1
    hw.ata.tags: 0
    hw.ata.atapi_dma: 0
    compat.linux.osname: Linux
    compat.linux.osrelease: 2.2.12
    compat.linux.oss_version: 198144
    

    Many of these need no description, but some are less obvious:

  • kern.msgbuf shows the contents of the kernel message buffer, which is also listed by the dmesg program.
  • kern.corefile specifies a template for the name of the core dump file generated when a process fails. By default the core file ends up in the current working directory—whatever that might be. By specifying an absolute path name, you can ensure that any core file will go into a specific directory. The text %N is replaced by the name of the program.
  • Living with FreeBSD-CURRENT

    Keeping up with FreeBSD-CURRENT requires work on your part. You should be on the FreeBSD-current@FreeBSD.org mailing list, which you can join via majordomo. See page 17 for details of how to use majordomo.

    Build kernels with debug symbols

    FreeBSD-CURRENT is not as stable as the released releases. To prepare yourself for possible problems, you should build kernels that include debug symbols. The resultant kernel is about 30 MB in size, but it will make debugging with ddb (the kernel debugger) or gdb much easier. Even if you don't intend to do this yourself, the information will be of great use to anybody you may call in to help. We looked at how to build a debug kernel on page 614.

    Solving problems in FreeBSD-CURRENT

    You will run into problems with FreeBSD-CURRENT. When it happens, please first read the mailing list and possibly the mail archives and see if the problem has been reported. If it hasn't, try to investigate the problem yourself. Then send mail to FreeBSD-current describing the problem and what you have done to solve it.

    If you experience a panic, please don't just send a message to FreeBSD-current saying "My kernel panics when I type foo." Remember that you're asking somebody to use their spare time to look at the problem. Make it easy for them:

  • Update to the absolutely latest sources, unless emails have been warning against this.
  • If you have any local patches, back them out.
  • Recompile, from scratch, your kernel with ddb and with complete symbols (see above).
  • Report all details from the panic. At an absolute minimum, give all information from
  • show reg and trace.
  • Try to dump the system.
  • If you're successful, follow the procedure discussed in the following section to find out something about how the problem occurred.
  • If you don't do at least this, there isn't much chance that a mail message to FreeBSD-current will have much effect.

    Analyzing kernel crash dumps

    When the kernel panics, and you have dumping enabled, you'll usually see something like this on the console:

    \
    Fatal trap 9: general protection fault while in kernel mode
    instruction pointer     = 0x8:0xc01c434b
    stack pointer           = 0x10:0xc99f8d0c
    frame pointer           = 0x10:0xc99f8d28
    code segment            = base 0x0, limit 0xfffff, type 0x1b
                            = DPL 0, pres 1, def32 1, gran 1
    processor eflags        = interrupt enabled, resume, IOPL = 0
    current process         = 2638 (find)
    interrupt mask          = net tty bio cam
    trap number             = 9
    panic: general protection fault
    
    syncing disks... 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7
    giving up on 6 buffers
    Uptime: 17h53m13s
    
    dumping to dev #ad/1, offset 786560
    dump ata0: resetting devices .. done
    

    You don't need to write this information down: it is saved in the dump.

    When you reboot, the system startup scripts find that you have a dump in the designated dump device (see above) and copy it and the current kernel to /var/crash, assuming the directory exists and there's enough space for the dump. You'll see something like this in the directory:

    # cd /var/crash
    # ls -l
    -rw-r--r--   1   root   wheel           3  Dec 29   10:09  bounds
    -rw-r--r--   1   root   wheel     4333000  Dec 29   10:10  kernel.22
    -rw-r--r--   1   root   wheel           5  Sep 17    1999  minfree
    -rw-------   1   root   wheel   268369920  Dec 29   10:09  vmcore.22
    

    The important files here are kernel.22, which contains a copy of the kernel running when the crash occurred, and vmcore.22, which contains the contents of memory. The number 22 indicates that the sequence number of the dump. It's possible to have multiple dumps in /var/crash. Note that you can waste a lot of space like that.

    The file bounds contains the number of the next dump (23 in this case), and minfree specifies the minimum amount of free space (in kilobytes) to leave on the file system after you've copied the dump. If this can't be guaranteed, savecore doesn't save the dump.

    savecore copies the kernel from which you booted. As we've seen, it typically isn't a debug kernel. In the example above, we installed /usr/src/sys/i386/conf/FREEBIE/kernel, but the debug version was /usr/src/sys/i386/conf/FREEBIE/kernel.debug. This is the one you need. The easiest way to access it is to use a symbolic link:

    # ln -s /usr/src/sys/i386/conf/FREEBIE/kernel.debug .
    # ls -lL
    -rw-r--r--   1   root  wheel          3 Dec 29 10:09 bounds
    -rwxr-xr-x   1   grog  lemis   16796546 Dec 18 14:21 kernel.debug
    -rw-r--r--   1   root  wheel    4333000 Dec 29 10:10 kernel.22
    -rw-r--r--   1   root  wheel          5 Sep 17  1999 minfree
    -rw-------   1   root  wheel  268369920 Dec 29 10:09 vmcore.22
    

    As you can see, it's much larger.

    Next, run gdb against the kernel and the dump:

    # gdb -k kernel.debug vmcore.22
    

    The first thing you see is a political message from the Free Software Foundation, followed by a repeat of the crash messages, a listing of the current instruction (always the same) and a prompt:

    #0   dumpsys () at ../../kern/kern_shutdown.c:473
    473   if (dumping++) {
    (kgdb)
    

    Due to the way C, gd and FreeBSD work, the real information you're looking for is further down the stack. The first thing you need to do is to find out exactly where it happens. Do that with the backtrace command:

    (kgdb) bt
    #0   dumpsys () at ../../kern/kern_shutdown.c:473
    #1   0xc01c88bf in boot (howto=256) at ../../kern/kern_shutdown.c:313
    #2   0xc01c8ca5 in panic (fmt=0xc03a8cac "%s") at ../../kern/kern_shutdown.c:581
    #3   0xc033ab03 in trap_fatal (frame=0xc99f8ccc, eva=0)
         at ../../i386/i386/trap.c:956
    #4   0xc033a4ba in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16,
           tf_edi = -1069794208, tf_esi = -1069630360, tf_ebp = -912290520,
           tf_isp = -912290568, tf_ebx = -1069794208, tf_edx = 10, tf_ecx = 10,
           tf_eax = -1, tf_trapno = 9, tf_err = 0, tf_eip = -1071889589, tf_cs = 8,
           tf_eflags = 66182, tf_esp = 1024, tf_ss = 6864992})
         at ../../i386/i386/trap.c:618
    #5   0xc01c434b in malloc (size=1024, type=0xc03c3c60, flags=0)
         at ../../kern/kern_malloc.c:233 
    #6   0xc01f015c in allocbuf (bp=0xc3a6f7cc, size=1024)
         at ../../kern/vfs_bio.c:2380 
    #7   0xc01effa6 in getblk (vp=0xc9642f00, blkno=0, size=1024, slpflag=0,
         slptimeo=0) at ../../kern/vfs_bio.c:2271 
    #8   0xc01eded2 in bread (vp=0xc9642f00, blkno=0, size=1024, cred=0x0,
         bpp=0xc99f8e3c) at ../../kern/vfs_bio.c:504 
    #9   0xc02d0634 in ffs_read (ap=0xc99f8ea0) at ../../ufs/ufs/ufs_readwrite.c:273 
    #10 0xc02d734e in ufs_readdir (ap=0xc99f8ef0) at vnode_if.h:334
    #11 0xc02d7cd1 in ufs_vnoperate (ap=0xc99f8ef0)
        at ../../ufs/ufs/ufs_vnops.c:2382
    #12 0xc01fbc3b in getdirentries (p=0xc9a53ac0, uap=0xc99f8f80)
        at vnode_if.h:769
    #13 0xc033adb5 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
          tf_edi = 134567680, tf_esi = 134554336, tf_ebp = -1077937404,
          tf_isp = -912289836, tf_ebx = 672064612, tf_edx = 134554336,
          tf_ecx = 672137600, tf_eax = 196, tf_trapno = 7, tf_err = 2,
          tf_eip = 671767876, tf_cs = 31, tf_eflags = 582, tf_esp = -1077937448,
          tf_ss = 47}) at ../../i386/i386/trap.c:1155
    #14 0xc032b825 in Xint0x80_syscall ()
    #15 0x280a1eee in ?? ()
    #16 0x280a173a in ?? () #17 0x804969e in ?? () #18 0x804b550 in ?? () #19 0x804935d in ?? () (kgdb)
    

    The rest of this chapter is only of interest to programmers with a good understanding of C. If you're not a programmer, this is about as far as you can go. Save this information and supply it to whomever you ask for help. It's usually not enough to solve the problem, but it's a good start, and your helper will be able to tell you what to do next.

    Climbing through the stack

    The backtrace outputs information about stack frames, which are built when a function is called. They're numbered starting from the most recent frame, #0, which is seldom the one that interests us. In general, we've had a panic, the most important frame is the function that calls panic:

    #3  0xc033ab03 in trap_fatal (frame=0xc99f8ccc, eva=0)
        at ../../i386/i386/trap.c:956
    

    The information here is:

  • #3 is the frame number. This is a number allocated by gdb. You can use it to reference the frame in a number of commands.
  • 0xc033ab03 is the return address from the call to the next function up the stack (panic in this case).
  • trap_fatal is the name of the function.
  • (frame=0xc99f8ccc, eva=0) are the parameter values supplied to trap_fatal.
  • ../../i386/i386/trap.c:956 gives the name of the source file and the line number in the file. The path names are relative to the kernel build directory, so they usually start with ../../.
  • In this example, the panic comes from a user process. Starting at the bottom, depending on the processor platform, you may see the user process stack. You can recognize them on an Intel platform by the addresses below the kernel base address 0xc0000000.On other platforms, the address might be different. In general, you won't get any symbolic information for these frames, since the kernel symbol table doesn't include user symbols.

    Climbing up the stack, you'll find the system call stack frame, in this example at frames 14 and 13. This is where the process involved the kernel. The stack frame above (frame 12) generally shows the name of the system call, in this case getdirentries.To perform its function, getdirentries indirectly calls ffs_read, the function that reads from a UFS file. ffs_read calls bread, which reads into the buffer cache. To do so, it allocates a buffer with getblk and allocbuf, which calls malloc to allocate memory for buffer cache. The next thing we see is a stack frame for trap: something has gone wrong inside malloc. trap determines that the trap in unrecoverable and calls trap_fatal, which in turn calls panic. The stack frames above show how the system prepares to dump and writes to disk. They're no longer of interest.

    Finding out what really happened

    In general, you start analyzing a panic dump in the stack frame that called panic, but in the case of the fatal trap that we have here, the most important stack frame is the one below trap, in this case frame 5. That's where things went wrong. Select it with the frame command, abbreviated to f, and list the code with list (or l):

    (kgdb) f 5
    #5 0xc01c434b in malloc (size=1024, type=0xc03c3c60, flags=0)
        at ../../kern/kern_malloc.c:233
    233             va = kbp->kb_next;
    (kgdb) l
    228                }
    229                freep->next = savedlist;
    230                if(kbp->kb_last == NULL)
    231                        kbp->kb_last = (caddr_t)freep;
    232                }
    233             va = kbp->kb_next;
    234             kbp->kb_next = ((struct freelist *)va)->next;
    235    #ifdef INVARIANTS
    236             freep = (struct freelist *)va;
    237             savedtype = (const char *) freep->type->ks_shortdesc;
    (kgdb)
    

    You might want to look at the local (automatic) variables. Use info local, which you can abbreviate to I loc:

    (kgdb) I loc
    type = (struct malloc_type *) 0xc03c3c60
    kbp = (struct kmembuckets *) 0xc03ebc68
    kup = (struct kmemusage *) 0x0
    freep = (struct freelist *) 0x0
    indx = 10
    npg = -1071714292
    allocsize = -1069794208
    s=6864992
    va = 0xffffffff <Address 0xffffffff out of bounds>
    cp = 0x0 savedlist = 0x0
    ksp = (struct malloc_type *) 0xffffffff
    (kgdb)
    

    The line where the problem occurs is 233:

    233   va =kbp->kb_next;
    

    Look at the structure kbp:

    (kgdb) p*kbp $2 = {
    kb_next = 0xffffffff <Address 0xffffffff out of bounds>,
    kb_last = 0xc1a31000 "",
    kb_calls = 83299,
    kb_total = 1164,
    kb_elmpercl = 4,
    kb_totalfree = 178,
    kb_highwat = 20,
    kb couldfree = 3812
    }
    

    The problem here is that the pointer kb_next is set to Oxffffffff . It should contain a valid address, but as gdb observes, this isn't not valid.

    So far we have found that the crash is in malloc, and that it's caused by an invalid pointer in an internal data structure. malloc is a function that is used many times a second by all computers. It's unlikely that the bug is in malloc. In fact, the most likely cause is that a function that has used memory allocated by malloc has overwritten its bounds and hit malloc's data structures.

    What do we do now? To quote fortune:

    The seven eyes of Ningauble the Wizard floated back to his hood as he reported to Fafhrd: "I have seen much, yet cannot explain all. The Gray Mouser is exactly twenty-five feet below the deepest cellar in the palace of Gilpkerio Kistomerces. Even though twenty-four parts in twenty-five of him are dead, he is alive.

    "Now about Lankhmar. She's been invaded, her walls breached everywhere and desperate fighting is going on in the streets, by a fierce host which out-numbers Lankhmar's inhabitants by fifty to one -and equipped with all modern weapons. Yet you can save the city."

    "How?" demanded Fafhrd.

    Ningauble shrugged. "You're a hero. You should know."

    -- Fritz Leiber, from "The Swords of Lankhmar"

    From here on, you're on your own. If you get this far, the FreeBSD-hackers mailing list may be interested in giving suggestions.

    Страницы:

    So far, everything we've done has been with the standard GENERIC kernel distributed with FreeBSD. You may find it an advantage to install a custom kernel:

  • As we saw in Chapter 2, GENERIC doesn't support everything that FreeBSD knows about. For example, if you want to install a Yoyodyne frobulator, you'll need to install special support for it In fact, the developer working on the Yoyodyne has defected to the Free Software Foundation. See the GNU General Public License for further details .
  • It will take less time to boot because it does not have to spend time probing for hardware that you do not have.
  • A custom kernel often uses less memory. The kernel is the one system component that must always be present in memory, so unused code ties up memory that would otherwise be available to the virtual memory system. On a system with limited RAM, you can save some memory by building a custom kernel, but don't overestimate the savings: a minimal kernel might save 500 kB over the GENERIC kernel supplied with the system.
  • In addition, there are several kernel options that you can tune to fit your needs.
  • Finally, on page 622 we'll look at things to think about if you want to run the -CURRENT version of FreeBSD.
  • In older releases of BSD, you needed to build a new kernel for just about anything you wanted to change, even things as simple as a different IRQ for a device. FreeBSD has evolved quite a bit since then, and it's becoming increasingly less necessary to build a custom kernel. You will certainly need to do so if you want to enable kernel-wide options, such as extra consistency checking, but in many cases you have more flexible alternatives:

  • If you just need to add device support, you may be able to load a Kernel Loadable Module, or kld. See page 620 for more information on klds.
  • If you want to change ISA parameters such as I/O address, IRQ or DMA settings, you no longer need to build a new kernel: the kernel configuration file no longer knows about such parameters. Instead, they're in the file /boot/device.hints, that we'll look at below.
  • A number of kernel options have been replaced by the sysctl interface. For example, the GENERIC kernel does not perform packet routing by default. In older releases of FreeBSD, you had to build a new kernel with the option GATEWAY. Nowadays you can turn this feature on and off at will with a sysctl command. We'll look at sysctls on page
  • Configuring a kernel has changed a lot since the early days of FreeBSD, but it's not done yet. The information in this chapter represents a snapshot in the evolution of building kernels. The goal of these changes is to make it unnecessary to build a kernel at all except to upgrade to a new release.

    Building a new kernel

    FreeBSD is distributed in source, and building a kernel primarily involves compiling the source files needed for the kernel. To build a kernel, you perform the following steps:

  • Install the system source, if you haven't already done so. We looked at that in Chapter 31.
  • Define your kernel configuration in a kernel configuration file. This file defines parameters to use during the build process. We'll look at how to do this starting on page .
  • Change to the directory /usr/src and run make kernel. This builds and installs the kernel and all modules. We'll look at it in more detail on page 617, where we'll also see alternatives that give more control over the process.
  • Configuring I/O devices

    A lot of the configuration file relates to the I/O devices that you may connect to your machine. In older releases of FreeBSD, it was often necessary to specify some of the IRQ, DMA channel, board memory, and I/O addresses for the devices you configure, particularly for ISA boards. Since Release 5 of FreeBSD, this is no longer the case. Instead, you modify the file boot/device.hints, which we looked at on page 575.

    The kernel installation does not install /boot/device.hints automatically. If it doesn't exist, copy it from the configuration directory:

    # cp -p /usr/src/sys/i386/conf/GENERIC.h±nts /boot/device.hints
    

    The kernel build directory

    The kernel sources are kept in the directory /usr/src/sys. The symbolic link /sys also points to this directory. There are a number of subdirectories of /usr/src/sys that represent different parts of the kernel, but for our purposes, the most important of them are the architecture dependent directories such as /usr/src/sys/i386/conf (for the i386 architecture), /usr/src/sys/alpha/conf (for the Alpha architecture), or /usr/src/sys/sparc64/conf (for the SPARC64 architecture) where you edit your custom kernel configuration. In addition, the old style kernel build described below builds the kernel in the directory /usr/src/sys/i386/compile, /usr/src/sys/alpha/compile or /usr/src/sys/sparc64/compile respectively. Notice the logical organization of the directory tree: each supported device, file system, and option has its own subdirectory. In the rest of this chapter, we'll look at the i386 architecture. Most of this applies to other architectures as well.

    If your system doesn't have the directory /usr/src/sys, the kernel source has not been installed. If you have a CD-ROM, the sources are on the first CD-ROM in the directory /src. To install from the CD-ROM, perform the following steps:

    # mkdir -p /usr/src/sys
    # ln -s /usr/src/sys /sys
    # cd /
    # cat /cdrom/src/ssys.[a-h]* | tar xzvf -
    

    The symbolic link /sys for /usr/src/sys is not strictly necessary, but it's a good idea: some software uses it, and otherwise you may end up with two different copies of the sources.

    By definition, the files on CD-ROM are out of date. See Chapter 31 for details of how to get the current, up-to-date sources.

    Next, move to the directory i386/conf and copy the GENERIC configuration file to the name you want to give your kernel. For example:

    # cd /usr/src/sys/i386/conf
    # cp GENERIC FREEBIE
    

    Traditionally, this name is in all capital letters and, if you are maintaining multiple FreeBSD machines with different hardware, it's a good idea to name it after your machine's hostname. In this example we call it FREEBIE.

    Now, edit FREEBIE with your favourite text editor. Change the comment lines at the top to reflect your configuration or the changes you've made to differentiate it from GENERIC:

    #
    # FREEBIE -- My personal configuration file
    #
    # For more information on this file, please read the handbook section on
    # Kernel Configuration Files:
    #
    # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
    #
    # The handbook is also available locally in /usr/share/doc/handbook
    # ifyou've installed the doc distribution, otherwise always see the
    # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
    # latest information. 
    #
    # Anexhaustive list of options and more detailed explanations of the
    # device lines is also present in the ../../conf/NOTES and NOTES files.
    # Ifyou are in doubt as to the purpose or necessity of a line, check first
    # inNOTES.
    #
    # $FreeBSD: src/sys/i386/conf/FREEBIE,v 1.369 2002/10/19 16:54:07 rwatson Exp $
    machine    "i386"
    cpu        "I486_CPU"
    cpu       "I586_CPU"
    cpu       "I686_CPU"
    ident      FREEBIE
    maxusers   0
    

    The configuration file

    The directory /sys/i386/conf contains a number of configuration files:

  • generic General-purpose configuration file
  • lint This file used to be a "complete" configuration file with comments, used for testing and documentation. Since FreeBSD Release 5, it no longer exists. Youcan create it from the files NOTES and /usr/src/sys/conf/NOTES with the command:
    $ make LIN
    
  • notes A complete pseudo-configuration file with copious comments. This file is descended from LINT, but it also includes device hints. You can't use it for building kernels. Instead, create the file LINT as described above. NOTES contains only platform-specific information. Most of the information is in the platform-independent file /usr/src/sys/conf/NOTES.
  • oldcard A configuration file for laptops that use PCCARD controllers. At the time of writing, PCCARD support has largely been rewritten, but the new code does not support some of the devices that the old code supports. This configuration file uses the old PCCARD code instead of the new code. When the new code is complete, it will go away.
  • The general format of a configuration file is quite simple. Each line contains a keyword and one or more arguments. Anything following a # is considered a comment and ignored. Keywords that contain numbers used as text must be enclosed in quotation marks.

    One of the results of this simplicity is that you can put in options that have absolutely no effect. For example, you could add a line like this:

    options APPLE_MAC_COMPATIBILITY
    

    You can build a kernel with this option. It will make no difference whatsoever. Now it's unlikely that you'll think up a non-existent option like this, but it's much more possible that you'll misspell a valid option, especially finger-twisters like SYSVSHM, with the result that you don't compile in the option you wanted. The config program warns if you use unknown options, so take these warnings seriously.

    Kernel options change from release to release, so there's no point in describing them all here. In the following sections we'll look at some of the more interesting ones; for a complete list, read LINT or the online handbook. See above for details of how to create LINT.

    Naming the kernel

    Every kernel you build requires the keywords machine, cpu, and ident. For example,

    machine  "i386"       For 386 architecture
    machine  "alpha"      For alpha architecture
    machine  "sparc64"    For PARC 64 architecture
    cpu      "I486_CPU"
    cpu      "I586_CPU"
    cpu      "I686_CPU"
    ident     FREEBIE
    

    machine

    The keyword machine describes the machine architecture for which the kernel is to be built. Currently it should be i386 for the Intel architecture, and alpha for the AXP architecture. Don't confuse this with the processor: for example, the i386 architecture refers to the Intel 80386 and all its successors, including lookalikes made by AMD, Cyrix and IBM.

    cpu cpu_type

    cpu describes which CPU chip or chips this kernel should support. For the i386 architecture, the possible values are I386_CPU, I486_CPU, I586_CPU and I686_CPU, and you can specify any combination of these values. For a custom kernel, it is best to specify only the CPU you have. If, for example, you have an Intel Pentium, use I586_CPU for cpu_type.

    If you're not sure what processor type you have, look at the output from the dmesg command when running the GENERIC kernel. For example:

    CPU: AMD Athlon(tm) XP processor 1700+ (1462.51-MHz 686-class CPU)
    Origin = "AuthenticAMD"    Id = 0x662   Stepping = 2
    Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,^OV,PAT,PS E36,MMX,FXSR,SSE>
    AMD Features=0xc0480000<<b19>,AMIE,DSP,3DNcw!>
    

    This shows that the processor, an AMD Athlon XP, is a "686 class" CPU, so to run a kernel on this processor, you must set I686_CPU in the config file.

    Since Release 5 of FreeBSD, it is no longer possible to build a single kernel with support for both the 80386 processor and later processors: the code for the later processors is optimized to use instructions that the 80386 processor does not have. Choose either I386_CPU or any combination of the others.

    ident machine_name

    ident specifies a name used to identify the kernel. In the file GENERIC it is GENERIC. Change this to whatever you named your kernel, in this example, FREEBIE. The value you put in ident will print when you boot up the kernel, so it's useful to give a kernel a different name if you want to keep it separate from your usual kernel (if you want to build an experimental kernel, for example). As with machine and cpu, enclose your kernel's name in quotation marks if it contains any numbers.

    This name is passed to the C compiler as a variable, so don't use names like DEBUG, or something that could be confused with another machine or CPU name, like vax.

    Kernel options

    There are a number of global kernel options, most of which you don't need to change. In the following section, we'll look at some of the few exceptions.

    Configuring specific I/O devices

    There are some devices that the GENERIC kernel does not support. In older releases of FreeBSD, you needed to build a new kernel to support them. This is seldom the case any more: most devices can be supported by klds. Work is under way to support the remainder. In case of doubt, look at the file HARDWARE.TXT on the installation CD-ROM.

    maxusers number

    This value sets the size of a number of important system tables. It is still included in the kernel configuration file, but you no longer need build a new kernel to change it. Instead, you can set it at boot time. For example, you might add the following line to your /boot/loader.conf file:

    maxusers="64"
    

    See 532 for more details of /boot/loader.conf.

    maxusers is intended to be roughly equal to the number of simultaneous users you expect to have on your machine, but it is only used to determine the size of some system tables. The default value 0 tells the kernel to autosize the tables depending on the amount of memory on the system. If the autosizing doesn' t work to your satisfaction, change this value. Even if you are the only person to use the machine, you shouldn't set maxusers lower than the default value 32, especially if you're using X or compiling software. The reason is that the most important table set by maxusers is the maximum number of processes, which is set to 20 + 16 * maxusers, so if you set maxusers to one, you can only have 36 simultaneous processes, including the 18 or so that the system starts up at boot time, and the 15 or so you will probably create when you start X. Even a simple task like reading a man page can start up nine processes to filter, decompress, and view it. Setting maxusers to 32 will allow you to have up to 532 simultaneous processes, which is normally ample. If, however, you see the dreaded proc table full error when trying to start another program, or are running a server with a large number of simultaneous users, you can always increase this number and reboot.

    maxusers does not limit the number of users who can log into the machine. It simply sets various table sizes to reasonable values considering the maximum number of users you will likely have on your system and how many processes each of them will be running. It's probable that this parameter will go away in future.

    Multiple processors

    FreeBSD 5.0 supports most modern multiprocessor systems. Earlier versions of the GENERIC kernel did not support them by default. Nowadays GENERIC supports multiple processors, so you don't need to do anything special. Look at the option SMP and the device apic in the configuration file if you're interested.

    Debug options

    FreeBSD is a very stable operating system. No software is perfect, however, and sometimes it crashes. When it does, it provides a number of facilities to help fix the problem. Some of these are dependent on kernel build options.

    Even if you have no intention of debugging a kernel problem yourself, you should set debug symbols when you build a kernel. They cost nothing except disk space, and if you are short on disk space, you can remove most of the files after the build.

    To set the debug symbols, remove the leading # mark from this line in the configuration file:

    makeoptions  DEBUG=-g   # Build kernel with gdb(1) debug symbols
    

    Under normal circumstances this makes no difference: the build process still installs the kernel without the debug symbols, and it has no effect on performance. If, however, you have a crash, the kernel with debug symbols is available in the kernel build directory, in this case /usr/src/sys/i386/compile/FREEBIE/kernel.debug, to assist analysis of the problem. Without this file it will be very difficult to find it.

    So why is it commented out? Without debug symbols, your build directory will take about 50 MB of disk space. With debug symbols, it will be about 250 MB. The FreeBSD Project couldn't agree to change it.

    Other debugging options

    If you run into trouble with your system, there are a number of other debugging options that you can use. The following are the more important ones:

    options   DDB
    options   BREAK_TO_DEBUGGER
    options   DDB_UNATTENDED      # Don't drop into DDB for a panic
    options   GDB_REMOTE_CHAT     #Use gdb remote debugging protocol
    options   KTRACE
    options   DIAGNOSTIC
    options   INVARIANTS
    options   INVARIANT_SUPPORT
    options   WITNESS           #Enable checks to detect deadlocks and cycles
    options   WITNESS_SKIPSPIN     #Don't run witness on spinlocks for speed
    These options provide support for various debugging features.
    

    DDB

    Specify DDB to include the kernel debugger, ddb. If you set this option, you might also want to set the BREAK_TO_DEBUGGER option,

    BREAK_TO_DEBUGGER

    Use the option BREAK_TO_DEBUGGER if you have installed the kernel debugger and you have the system console on a serial line.

    DDB_UNATTENDED

    If you have a panic on a system with ddb, it will not reboot automatically. Instead, it will enter ddb and give you a chance to examine the remains of the system before rebooting. This can be a disadvantage on systems that run unattended: after a panic, they would wait until somebody comes past before rebooting. Use the DDB_UNATTENDED option to cause a system with ddb to reboot automatically on panic.

    GDB_REMOTE_CHAT

    ddb supports remote debugging from another FreeBSD machine via a serial connection. See the online handbook for more details. To use this feature, set the option GDB_REMOTE_CHAT.

    KTRACE

    Set KTRACE if you want to use the system call trace program ktrace .

    DIAGNOSTIC

    A number of source files use the DIAGNOSTIC option to enable extra sanity checking of internal structures. This support is not enabled by default because of the extra time it would take to check for these conditions, which can only occur as a result of programming errors.

    INVARIANTS and INVARIANT_SUPPORT

    INVARIANTS is used in a number of source files to enable extra sanity checking of internal structures. This support is not enabled by default because of the extra time it would take to check for these conditions, which can only occur as a result of programming errors.

    INVARIANT_SUPPORT option compiles in support for verifying some of the internal structures. It is a prerequisite for INVARIANTS. The intent is that you can set INVARIANTS for single source files (by changing the source file or specifying it on the command line) if you have INVARIANT_SUPPORT enabled.

    WITNESS and WITNESS_SKIPSPIN

    One of the big changes in FreeBSD Release 5 relates to the manner in which the kernel performs resource locking. As a result, the danger exists of deadlocks, locks that can'tbe undone without rebooting the machine. WITNESS checks for the danger of deadlocks and warns if it finds a potential deadlock ("lock order reversal"). This is a very expensive debugging option: it can slow the machine down by an order of magnitude, so don't use it unless you have to.

    A compromise that doesn't use quite so much processor power is the combination of WITNESS with WITNESS_SKIPSPIN, which avoids spin locks. It can still catch most problems.

    Preparing for upgrades

    When changing the configuration file, consider that it probably won't be the only time you make these changes. At some time in the future, you'll upgrade the system, and you'll have to build your custom kernel all over again. But the GENERIC kernel configuration file will have changed as well. You have two choices: incorporate the modifications to GENERIC into your configuration file, or incorporate your modifications to the old GENERIC file into the new GENERIC configuration file. It turns out that the latter path is easier.

    To prepare for this approach, try to change as little as possible in the body of the configuration file. Instead, add all your changes to the end, along with a comment so that you can readily recognize the changes. For example, you might append:

    # Added by grog, 24 October 2002
    # Comment out WITNESS, WITNESS_SKIPSPIN and SCSI_DELAY above
    options    BREAK_TO_DEBUGGER
    options    SCSI_DELAY=3000    #Delay (in ms) before probing SCSI
    options    CAMDEBUG
    options    MSGBUF_SIZE=81920
    #options   TIMEREQUESTS      #watch for delays
    device     snp               #Snoop device - to look at pty/vty/etc.
    

    That won't be all, of course. Look at that option SCSI_DELAY.That option already exists in the configuration file (with a value of 15 seconds instead of 3). If you leave both, config will issue a warning. You need to comment out the first occurrence, as the comment at the top indicates.

    Building and installing the new kernel

    The traditional way to build a BSD kernel was, assuming a configuration file called FREEBIE:

    # cd /usr/src/sys/i386/conf
    # config FREEBIE
    # cd ../compile/FREEBIE
    # make depend all install
    

    At the time of writing, this still works, but it will go away at some time in the future. It has the disadvantage that you need to know your architecture, and the kernel is built in the source tree, which is not a good idea. The new kernel build method starts from /usr/src, the same directory as all other build operations, and it builds the kernel in the /usr/obj hierarchy.

    You'll need about 250 MB of free space on /usr/obj to build a kernel. If you're really tight on space, you can reduce this value to about 50 MB by omitting the makeoptions DEBUG=-g specification, but if you have problems with the system at some later stage, it will be much more difficult to find what is causing them.

    There are a number of alternative commands for building the kernel:

    # cd /usr/src
    # make kernel KERNCONF=FREEBIE               build and install kernel and klds
    # make buildkernel KERNCONF=FREEBIE          build kernel and klds
    #make installkernel KERNCONF=FREEBIE         install prebuilt kernel and klds
    #make reinstallkernel KERNCONF=FREEBIE       reinstall kernel and klds
    # make kernel KERNCONF=FREEBIE -DNO_MODULES  build and install kernel only
    

    The easiest way is make kernel, which is subdivided into the buildkernel and installkernel steps. You can perform these steps individually if you want. If you use the reinstallkernel target instead of the installkernel target, the /boot/kernel.old hierarchy remains unchanged. This is useful when a previous kernel build failed to boot.

    If you know the klds are not going to change, you can speed things up by not building them again. Use the -DNO_MODULES flag in combination with any of the other targets to inhibit building or installing the klds, as shown in the last example. Don't do this the first time you build the kernel: if you have mismatches between the kernel and the klds, you may find it impossible to start the system. On the other hand, if you find the kernel doesn't boot, and you want to change the configuration file and rebuild it from the same sources, you can save some time like this.

    The first step in building the kernel is to run the config program. You no longer have to do this yourself; the buildkernel and kernel targets do it for you. config creates a directory in which to build the kernel and fills it with the necessary infrastructure, notably the kernel Makefile and a number of header files.

    It's possible to get error messages at this stage if you have made a mistake in the config file. If the config command fails when you give it your kernel description, you've probably made a simple error somewhere. Fortunately, config will print the line number that it had trouble with, so you can quickly find it with an editor. For example:

    config: line 17: syntax error
    

    One possibility is that you have mistyped a keyword. Compare it to the entry in the GENERIC or LINT kernel definitions.

    The next step is to compile all the source files and create a kernel and a set of matching klds. It can take some time, up to an hour on a slow machine. It's also possible to have errors, and unfortunately they are usually not self-explanatory. If the make command fails, it usually signals an error in your kernel description that is not obvious enough for config to catch. A common one is when you omit an entry from the configuration file which is a prerequisite for an entry which is present. For example, if you have SCSI disks (device da), you require the scbus device as well, and if you have just about any kind of Ethernet card, you require the miibus device as well:

    device   scbus   # SCSI bus (required)
    device   da      # Direct Access (disks)
    device   miibus  # MII bus support
    device   fxp     # Intel EtherExpress PRO/100B (82557, 82558)
    

    If you leave scbus or miibus out of the configuration, config will not complain, but the kernel link phase will fail with lots of unresolved references.

    If you can't resolve the problem after comparing your configuration file with GENERIC, send mail to FreeBSD-questions@FreeBSD.org with your kernel configuration, and it should be diagnosed very quickly. A description of how to interpret these errors has been in the works for a long time, but currently it's still deep magic.

    Rebooting

    Next, shutdown the system and reboot load the new kernel:

    # shutdown -r now
    

    If the new kernel does not boot, or fails to recognize your devices, don't panic. Reset the machine, and when the boot prompt appears, press the space bar to interrupt the boot. Then boot the old kernel:

    Ok unload                        remove the kernel that the loader has loaded
    Ok load /boot/kernel.old/kernel  load the previous kernel
    Ok boot
    

    When reconfiguring a kernel, it is always a good idea to keep on hand a kernel that is known to work. There are two points here:

  • If your kernel doesn't boot, you don't want to save it to kernel.old when you build a new one. It's no use, and if the new kernel doesn't boot either, you'll be left without a runnable kernel. In this case, use the reinstallkernel target mentioned above:
    # make buildkernel KERNCONF=FREEBIE      build kernel and klds
    # make reinstallkernel KERNCONF=FREEBIE  reinstall kernel and klds
    
  • You could still make a mistake and type make install, throwing away your last good kernel. It's easy enough to end up with a completely unbootable system like this. It's a good idea to keep another kernel copy with a name like kernel.save, which the installation procedure does not touch.
  • After booting with a good kernel you can check over your configuration file and try to build it again. One helpful resource is the /var/log/messages file which records, among other things, all of the kernel messages from every successful boot. Also, the dmesg command prints the most recent kernel messages from the current boot. After some time the original messages are overwritten, so the system startup procedure saves the messages at boot time in the file /var/run/dmesg.boot. Note that most laptops maintain the previous contents of the kernel message buffer when rebooted, so the beginning of the output of dmesg may relate to an earlier boot. Check through to the end before jumping to conclusions.

    Making device nodes

    FreeBSD Release 5 comes with devfs, the device file system. One great advantage of devfs is that it automatically creates device nodes for the hardware it finds, so you no longer need to run the /dev/MAKEDEV script supplied with older releases of FreeBSD.

    Kernel loadable modules

    As we saw at the beginning of the chapter, you may not have to build a new kernel to implement the functionality you want. Instead, just load it into the running kernel with a Kernel Loadable Module (kld). The directory /boot/kernel/modules contains a number of klds. To load them, use kldload. For example, if you wanted to load SCO UNIX compatibility, you would enter:

    #  kldload ibcs2
    

    This loads the module /boot/kernel/modules/ibcs2.ko. Note that you don't need to specify the directory name, nor the .ko extension.

    To find what modules are loaded, use kldstat:

    # kldstat
    Id Refs  Address      Size    Name
    1     5  0xc0100000   1d08b0  kernel
    2     2  0xc120d000   a000    ibcs2.ko
    3     1  0xc121b000   3000    ibcs2_coff.ko
    4     1  0xc1771000   e000    linux.ko
    5     1  0xc177f000   bf000   vinum.ko
    

    You can also unload some klds, but not all of them. Use kldunload for this purpose:

    #  kldunload vinum
    

    You can't unload a kld which has active resources. In the case of vinum, for example, you can only unload it when none of its volumes are mounted.

    sysctl

    sysctl is a relatively new kernel interface that allows access to specific variables in the kernel. Some of these variables are read-only: you can look, but not touch. Others are changeable.

    sysctl variables are usually referred to simply as sysctls. Each sysctl has a name in "Management Information Base"(MIB) form, consisting of a hierarchical arrangement of names separated by periods (.). The first component of the name indicates the part of the kernel to which it relates. The following examples give you an idea of how to use the sysctl program:

    $ sysctl kern.ostype
    FreeBSD
    $ sysctl kern                      list all sysctls starting with kern
    $ sysctl -a                        list all sysctls
    # sysctl net.inet.ip.forwarding=1  turn IP forwarding on
    net.inet.ip.forwarding: 0 -> 1
    

    Some of the more interesting sysctls are:

    kern.ostype: FreeBSD kern.osrelease: 5.0-RELEASE
    kern.version: FreeBSD 5.0-RELEASE #0: Thu Jan 16 15:03:31 CST 2003
    grog@freebie.example.org:/usr/src/sys/GENERIC kern.hostname: freebie.example.org
    kern.boottime:  { sec = 1007165073, usec = 570637 } Fri Jan 17 10:34:33 2003 
    kern.bootfile: /boot/kernel/kernel
    kern.init_path: /sbin/init:/sbin/oinit:/sbin/init.bak:/stand/sysinstall
    kern.module_path: /boot/kernel;/boot/kernel;/boot/modules;/modules 
    kern.coredump: 1 
    kern.corefile: /var/tmp/%N.core
    kern.msgbuf: nreach TCP 213.46.243.23:25370 139.130.136.138:25 in via ppp0
    net.inet.ip.fw.enable: 1 hw.machine: i386
    hw.model: Pentium II/Pentium II Xeon/Celeron
    hw.ncpu: 1
    hw.byteorder: 1234
    hw.physmem: 129949696
    hw.usermem: 100556800
    hw.pagesize: 4096
    hw.floatingpoint: 1
    hw.machine_arch: i386
    hw.ata.ata_dma: 1
    hw.ata.wc: 1
    hw.ata.tags: 0
    hw.ata.atapi_dma: 0
    compat.linux.osname: Linux
    compat.linux.osrelease: 2.2.12
    compat.linux.oss_version: 198144
    

    Many of these need no description, but some are less obvious:

  • kern.msgbuf shows the contents of the kernel message buffer, which is also listed by the dmesg program.
  • kern.corefile specifies a template for the name of the core dump file generated when a process fails. By default the core file ends up in the current working directory—whatever that might be. By specifying an absolute path name, you can ensure that any core file will go into a specific directory. The text %N is replaced by the name of the program.
  • Living with FreeBSD-CURRENT

    Keeping up with FreeBSD-CURRENT requires work on your part. You should be on the FreeBSD-current@FreeBSD.org mailing list, which you can join via majordomo. See page 17 for details of how to use majordomo.

    Build kernels with debug symbols

    FreeBSD-CURRENT is not as stable as the released releases. To prepare yourself for possible problems, you should build kernels that include debug symbols. The resultant kernel is about 30 MB in size, but it will make debugging with ddb (the kernel debugger) or gdb much easier. Even if you don't intend to do this yourself, the information will be of great use to anybody you may call in to help. We looked at how to build a debug kernel on page 614.

    Solving problems in FreeBSD-CURRENT

    You will run into problems with FreeBSD-CURRENT. When it happens, please first read the mailing list and possibly the mail archives and see if the problem has been reported. If it hasn't, try to investigate the problem yourself. Then send mail to FreeBSD-current describing the problem and what you have done to solve it.

    If you experience a panic, please don't just send a message to FreeBSD-current saying "My kernel panics when I type foo." Remember that you're asking somebody to use their spare time to look at the problem. Make it easy for them:

  • Update to the absolutely latest sources, unless emails have been warning against this.
  • If you have any local patches, back them out.
  • Recompile, from scratch, your kernel with ddb and with complete symbols (see above).
  • Report all details from the panic. At an absolute minimum, give all information from
  • show reg and trace.
  • Try to dump the system.
  • If you're successful, follow the procedure discussed in the following section to find out something about how the problem occurred.
  • If you don't do at least this, there isn't much chance that a mail message to FreeBSD-current will have much effect.

    Analyzing kernel crash dumps

    When the kernel panics, and you have dumping enabled, you'll usually see something like this on the console:

    \
    Fatal trap 9: general protection fault while in kernel mode
    instruction pointer     = 0x8:0xc01c434b
    stack pointer           = 0x10:0xc99f8d0c
    frame pointer           = 0x10:0xc99f8d28
    code segment            = base 0x0, limit 0xfffff, type 0x1b
                            = DPL 0, pres 1, def32 1, gran 1
    processor eflags        = interrupt enabled, resume, IOPL = 0
    current process         = 2638 (find)
    interrupt mask          = net tty bio cam
    trap number             = 9
    panic: general protection fault
    
    syncing disks... 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7
    giving up on 6 buffers
    Uptime: 17h53m13s
    
    dumping to dev #ad/1, offset 786560
    dump ata0: resetting devices .. done
    

    You don't need to write this information down: it is saved in the dump.

    When you reboot, the system startup scripts find that you have a dump in the designated dump device (see above) and copy it and the current kernel to /var/crash, assuming the directory exists and there's enough space for the dump. You'll see something like this in the directory:

    # cd /var/crash
    # ls -l
    -rw-r--r--   1   root   wheel           3  Dec 29   10:09  bounds
    -rw-r--r--   1   root   wheel     4333000  Dec 29   10:10  kernel.22
    -rw-r--r--   1   root   wheel           5  Sep 17    1999  minfree
    -rw-------   1   root   wheel   268369920  Dec 29   10:09  vmcore.22
    

    The important files here are kernel.22, which contains a copy of the kernel running when the crash occurred, and vmcore.22, which contains the contents of memory. The number 22 indicates that the sequence number of the dump. It's possible to have multiple dumps in /var/crash. Note that you can waste a lot of space like that.

    The file bounds contains the number of the next dump (23 in this case), and minfree specifies the minimum amount of free space (in kilobytes) to leave on the file system after you've copied the dump. If this can't be guaranteed, savecore doesn't save the dump.

    savecore copies the kernel from which you booted. As we've seen, it typically isn't a debug kernel. In the example above, we installed /usr/src/sys/i386/conf/FREEBIE/kernel, but the debug version was /usr/src/sys/i386/conf/FREEBIE/kernel.debug. This is the one you need. The easiest way to access it is to use a symbolic link:

    # ln -s /usr/src/sys/i386/conf/FREEBIE/kernel.debug .
    # ls -lL
    -rw-r--r--   1   root  wheel          3 Dec 29 10:09 bounds
    -rwxr-xr-x   1   grog  lemis   16796546 Dec 18 14:21 kernel.debug
    -rw-r--r--   1   root  wheel    4333000 Dec 29 10:10 kernel.22
    -rw-r--r--   1   root  wheel          5 Sep 17  1999 minfree
    -rw-------   1   root  wheel  268369920 Dec 29 10:09 vmcore.22
    

    As you can see, it's much larger.

    Next, run gdb against the kernel and the dump:

    # gdb -k kernel.debug vmcore.22
    

    The first thing you see is a political message from the Free Software Foundation, followed by a repeat of the crash messages, a listing of the current instruction (always the same) and a prompt:

    #0   dumpsys () at ../../kern/kern_shutdown.c:473
    473   if (dumping++) {
    (kgdb)
    

    Due to the way C, gd and FreeBSD work, the real information you're looking for is further down the stack. The first thing you need to do is to find out exactly where it happens. Do that with the backtrace command:

    (kgdb) bt
    #0   dumpsys () at ../../kern/kern_shutdown.c:473
    #1   0xc01c88bf in boot (howto=256) at ../../kern/kern_shutdown.c:313
    #2   0xc01c8ca5 in panic (fmt=0xc03a8cac "%s") at ../../kern/kern_shutdown.c:581
    #3   0xc033ab03 in trap_fatal (frame=0xc99f8ccc, eva=0)
         at ../../i386/i386/trap.c:956
    #4   0xc033a4ba in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16,
           tf_edi = -1069794208, tf_esi = -1069630360, tf_ebp = -912290520,
           tf_isp = -912290568, tf_ebx = -1069794208, tf_edx = 10, tf_ecx = 10,
           tf_eax = -1, tf_trapno = 9, tf_err = 0, tf_eip = -1071889589, tf_cs = 8,
           tf_eflags = 66182, tf_esp = 1024, tf_ss = 6864992})
         at ../../i386/i386/trap.c:618
    #5   0xc01c434b in malloc (size=1024, type=0xc03c3c60, flags=0)
         at ../../kern/kern_malloc.c:233 
    #6   0xc01f015c in allocbuf (bp=0xc3a6f7cc, size=1024)
         at ../../kern/vfs_bio.c:2380 
    #7   0xc01effa6 in getblk (vp=0xc9642f00, blkno=0, size=1024, slpflag=0,
         slptimeo=0) at ../../kern/vfs_bio.c:2271 
    #8   0xc01eded2 in bread (vp=0xc9642f00, blkno=0, size=1024, cred=0x0,
         bpp=0xc99f8e3c) at ../../kern/vfs_bio.c:504 
    #9   0xc02d0634 in ffs_read (ap=0xc99f8ea0) at ../../ufs/ufs/ufs_readwrite.c:273 
    #10 0xc02d734e in ufs_readdir (ap=0xc99f8ef0) at vnode_if.h:334
    #11 0xc02d7cd1 in ufs_vnoperate (ap=0xc99f8ef0)
        at ../../ufs/ufs/ufs_vnops.c:2382
    #12 0xc01fbc3b in getdirentries (p=0xc9a53ac0, uap=0xc99f8f80)
        at vnode_if.h:769
    #13 0xc033adb5 in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47,
          tf_edi = 134567680, tf_esi = 134554336, tf_ebp = -1077937404,
          tf_isp = -912289836, tf_ebx = 672064612, tf_edx = 134554336,
          tf_ecx = 672137600, tf_eax = 196, tf_trapno = 7, tf_err = 2,
          tf_eip = 671767876, tf_cs = 31, tf_eflags = 582, tf_esp = -1077937448,
          tf_ss = 47}) at ../../i386/i386/trap.c:1155
    #14 0xc032b825 in Xint0x80_syscall ()
    #15 0x280a1eee in ?? ()
    #16 0x280a173a in ?? () #17 0x804969e in ?? () #18 0x804b550 in ?? () #19 0x804935d in ?? () (kgdb)
    

    The rest of this chapter is only of interest to programmers with a good understanding of C. If you're not a programmer, this is about as far as you can go. Save this information and supply it to whomever you ask for help. It's usually not enough to solve the problem, but it's a good start, and your helper will be able to tell you what to do next.

    Climbing through the stack

    The backtrace outputs information about stack frames, which are built when a function is called. They're numbered starting from the most recent frame, #0, which is seldom the one that interests us. In general, we've had a panic, the most important frame is the function that calls panic:

    #3  0xc033ab03 in trap_fatal (frame=0xc99f8ccc, eva=0)
        at ../../i386/i386/trap.c:956
    

    The information here is:

  • #3 is the frame number. This is a number allocated by gdb. You can use it to reference the frame in a number of commands.
  • 0xc033ab03 is the return address from the call to the next function up the stack (panic in this case).
  • trap_fatal is the name of the function.
  • (frame=0xc99f8ccc, eva=0) are the parameter values supplied to trap_fatal.
  • ../../i386/i386/trap.c:956 gives the name of the source file and the line number in the file. The path names are relative to the kernel build directory, so they usually start with ../../.
  • In this example, the panic comes from a user process. Starting at the bottom, depending on the processor platform, you may see the user process stack. You can recognize them on an Intel platform by the addresses below the kernel base address 0xc0000000.On other platforms, the address might be different. In general, you won't get any symbolic information for these frames, since the kernel symbol table doesn't include user symbols.

    Climbing up the stack, you'll find the system call stack frame, in this example at frames 14 and 13. This is where the process involved the kernel. The stack frame above (frame 12) generally shows the name of the system call, in this case getdirentries.To perform its function, getdirentries indirectly calls ffs_read, the function that reads from a UFS file. ffs_read calls bread, which reads into the buffer cache. To do so, it allocates a buffer with getblk and allocbuf, which calls malloc to allocate memory for buffer cache. The next thing we see is a stack frame for trap: something has gone wrong inside malloc. trap determines that the trap in unrecoverable and calls trap_fatal, which in turn calls panic. The stack frames above show how the system prepares to dump and writes to disk. They're no longer of interest.

    Finding out what really happened

    In general, you start analyzing a panic dump in the stack frame that called panic, but in the case of the fatal trap that we have here, the most important stack frame is the one below trap, in this case frame 5. That's where things went wrong. Select it with the frame command, abbreviated to f, and list the code with list (or l):

    (kgdb) f 5
    #5 0xc01c434b in malloc (size=1024, type=0xc03c3c60, flags=0)
        at ../../kern/kern_malloc.c:233
    233             va = kbp->kb_next;
    (kgdb) l
    228                }
    229                freep->next = savedlist;
    230                if(kbp->kb_last == NULL)
    231                        kbp->kb_last = (caddr_t)freep;
    232                }
    233             va = kbp->kb_next;
    234             kbp->kb_next = ((struct freelist *)va)->next;
    235    #ifdef INVARIANTS
    236             freep = (struct freelist *)va;
    237             savedtype = (const char *) freep->type->ks_shortdesc;
    (kgdb)
    

    You might want to look at the local (automatic) variables. Use info local, which you can abbreviate to I loc:

    (kgdb) I loc
    type = (struct malloc_type *) 0xc03c3c60
    kbp = (struct kmembuckets *) 0xc03ebc68
    kup = (struct kmemusage *) 0x0
    freep = (struct freelist *) 0x0
    indx = 10
    npg = -1071714292
    allocsize = -1069794208
    s=6864992
    va = 0xffffffff <Address 0xffffffff out of bounds>
    cp = 0x0 savedlist = 0x0
    ksp = (struct malloc_type *) 0xffffffff
    (kgdb)
    

    The line where the problem occurs is 233:

    233   va =kbp->kb_next;
    

    Look at the structure kbp:

    (kgdb) p*kbp $2 = {
    kb_next = 0xffffffff <Address 0xffffffff out of bounds>,
    kb_last = 0xc1a31000 "",
    kb_calls = 83299,
    kb_total = 1164,
    kb_elmpercl = 4,
    kb_totalfree = 178,
    kb_highwat = 20,
    kb couldfree = 3812
    }
    

    The problem here is that the pointer kb_next is set to Oxffffffff . It should contain a valid address, but as gdb observes, this isn't not valid.

    So far we have found that the crash is in malloc, and that it's caused by an invalid pointer in an internal data structure. malloc is a function that is used many times a second by all computers. It's unlikely that the bug is in malloc. In fact, the most likely cause is that a function that has used memory allocated by malloc has overwritten its bounds and hit malloc's data structures.

    What do we do now? To quote fortune:

    The seven eyes of Ningauble the Wizard floated back to his hood as he reported to Fafhrd: "I have seen much, yet cannot explain all. The Gray Mouser is exactly twenty-five feet below the deepest cellar in the palace of Gilpkerio Kistomerces. Even though twenty-four parts in twenty-five of him are dead, he is alive.

    "Now about Lankhmar. She's been invaded, her walls breached everywhere and desperate fighting is going on in the streets, by a fierce host which out-numbers Lankhmar's inhabitants by fifty to one -and equipped with all modern weapons. Yet you can save the city."

    "How?" demanded Fafhrd.

    Ningauble shrugged. "You're a hero. You should know."

    -- Fritz Leiber, from "The Swords of Lankhmar"

    From here on, you're on your own. If you get this far, the FreeBSD-hackers mailing list may be interested in giving suggestions.

    Вернуться к учебному плану