Network debugging

Разбить на страницы

Показывать лекцию целиком

The chances are quite good that you'll have some problems somewhere when you set up your network. FreeBSD gives you a large number of tools with which to find and solve the problem.

In this chapter, we'll consider a methodology of debugging network problems. In the process, we'll look at the programs that help debugging. It will help to have your finger in Chapter 16 while reading this section.

How to approach network problems

Recall from Chapter 16 that network software and hardware operate on at least four layers. If one layer doesn't work, the ones above won't either. When solving problems, it obviously makes sense to start at the bottom and work up.

Most people understand this up to a point. Nobody expects a PPP connection to the Internet to work if the modem can't dial the ISP. On the other hand, a large number of messages to the FreeBSD-questions mailing list show that many people seem to think that once this connection has been established, everything else will work automatically. If it doesn't, they're puzzled.

Unfortunately, the Net isn't that simple. In fact, it's too complicated to give a hard-and-fast methodology at all. Much network debugging can look more like magic than anything rational. Nevertheless, a surprising number of network problems can be solved by using the steps below. Even if they don't solve your problem, read through them.

They might give you some ideas about where to look.

Link layer problems

To test your link layer, start with ping. ping is a relatively simple program that sends an ICMP echo packet to a specific IP address and checks the reply. ICMP is the Internet Control Message Protocol, is used for error reporting and testing. See TCP/IP Illustrated, by Richard Stevens, for more information.

A typical ping output might look like:

$ ping bumble
PING bumble.example.org (223.147.37.156): 56 data bytes
64 bytes from 223.147.37.156: icmp_seq=0 ttl=255 time=1.137 ms
64 bytes from 223.147.37.156: icmp_seq=1 ttl=255 time=0.640 ms
64 bytes from 223.147.37.156: icmp_seq=2 ttl=255 time=0.671 ms
64 bytes from 223.147.37.156: icmp_seq=3 ttl=255 time=0.612 ms
^C
--- bumble.example.org ping statistics ---
4packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.612/0.765/1.137/0.216 ms

In this case, we are sending the messages to the system bumble.example.org. By default, ping sends messages of 56 bytes. With the IP header, this makes packets of 64 bytes. By default, ping continues until you stop it—notice the ^C indicating that this invocation was stopped by pressing Ctrl-C.

The information that ping gives you isn't much, but it's useful:

It tells you how long it takes for each packet to get to its destination and back.

It tells you how many packets didn't make it.

It also prints a summary of packet statistics.

But what if this doesn't work? You enter your ping command, and all you get is:

$ ping wait
PING wait.example.org (223.147.37.4): 56 data bytes
^C
--- wait.example.org ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

Obviously, something's wrong here. We'll look at it in more detail below. This is very different, however, from this situation:

$ ping presto
^C

In the second case, even after some time, nothing happened at all. ping didn't print the PING message, and when we hit Ctrl-C there was no further output. This is indicative of a name resolution problem: ping can't print the first line until it has found the IP address of the system, in other words, until it has performed a DNS lookup. If we wait long enough, it will time out, and we get the message ping: cannot resolve presto: Unknown host. If this happens, use the IP address, not the name. DNS is an application, so we won't try to debug it until we've debugged the link and network layers.

If things don't work out, there are two possibilities:

If both systems are on the same network, it's a link layer problem. We'll look at that first.

If the systems are on two different networks, it might be a network layer problem. That's more complicated: we don't know which network to look at. It could be either of the networks on which the systems are located, or it could also be a problem with one of the networks on the way. How do you find out where your packets get lost? First you check the link layer. If it checks out OK, and the problem still exists, continue with the network layer on page 406.

So what can cause link layer problems? There are a number of possibilities:

One of the interfaces (source or destination) could be misconfigured. They should both have the same range of network addresses. For example, the following two interface configurations cannot talk to each other directly, even if they're on the same physical network:

machine 1
dcO: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    inet 223.147.37.81 netmask 0xffffff00 broadcast 223.147.37.255
machine 2
xl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
    options=3<RXCSUM,TXCSUM>
inet 192.168.27.1 netmask 0xffffff00 broadcast 192.168.27.255

If you see something like this on an Ethernet interface, it's pretty clear that it has a cabling problem:

xl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
    options=3<RXCSUM,TXCSUM>
    inet 192.168.27.1 netmask 0xffffff00 broadcast 192.168.27.255
    media: Ethernet autoselect (none)
    status: no carrier

In this case, check the physical connections. If you're using UTP, check that you have the right kind of cable, normally a "straight-through" cable. If you accidentally use a crossover cable where you need a straight-through cable, or vice versa, you will not get any connection. Also, many hubs and switches have a "crossover" switch that achieves the same result.

If you're on an RG-58 thin Ethernet, the most likely problem is a break in the cabling. You can check the static resistance between the central pin and the external part of the connector with a multimeter. It should be approximately $$25\Omega$$. If it's $$50\Omega$$, it indicates that there is a break in the cable, or that one of the terminators has been disconnected.

If your interface is configured correctly, and you're using a 10 Mb/s card, check whether you are using the correct connection to the network. Some older Ethernet boards support multiple physical connections (for example, both BNC and UTP). For example, if your network runs on RG58 thin Ethernet, and your interface is set to AUI, you may still be able to send data on the RG58, but you won't be able to receive any.

The method of setting the connection depends on the board you are using. PCI boards are not normally a problem, because the driver can set the parameters directly, but ISA boards can drive you crazy. In the case of very old boards, such as the Western Digital 8003, you may need to set jumpers. In others, you may need to run the setup utility under DOS, and with others you can set it with the link flags to ifconfig. For example, on a 3Com 3c509 "combo" board, you can set the connection like this:

# ifconfig ep0 –link0         setBNC
# ifconfig ep0 link0 -link1   set AUI
# ifconfig ep0 link0 link1    set UTP

This example is correct for the ep driver, but not necessarily for other Ethernet boards: each board has its own flags. Read the man page for the board for the correct flags.

If your interface looks OK, the next thing to do is to see whether you can send data to other machines on the network. If so, of course, you should continue your search on the machine that isn't responding. If none are working, you probably have a cabling problem.

On a wireless network, you need to check for a number of additional problems. ifconfig should show something like this:

wi0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    inet6 fe80::202:2dff:fe04:93a%wi0 prefixlen 64 scopeid 0x3
    inet 192.168.27.17 netmask 0xffffff00 broadcast 192.168.27.255
    ether 00:02:2d:21:54:4c
    media: IEEE 802.11 Wireless Ethernet autoselect (DS/11Mbps)
    status: associated
    ssid "FreeBSD IBSS" 1:""
    stationname "FreeBSD WaveLAN/IEEE node"
    channel 3 authmode OPEN powersavemode OFF powersavesleep 100
    wepmode OFF weptxkey 1
    wepkey 2:64-bit 0x123456789a 3:128-bit 0x123456789abcdef123456789ab

There are many things to check here:

Do you have the same operating mode? This example shows a card operating in BSS or IBSS mode. By contrast, you might see this:

media: IEEE 802.11 Wireless Ethernet autoselect (DS/11Mbps <adhoc, flag0>)

In this case, the interface is operating in so-called "Lucent demo ad-hoc" mode, which is not the same thing as "ad-hoc" mode (which in turn is better called IBSS mode). IBSS mode ("ad-hoc") and BSS mode are compatible. IBSS mode and "Lucent demo ad-hoc" mode are not. See Chapter 17, page 306 for further details.

Is the status associated? The alternative is no carrier. Some cards, including this one, show no carrier when communicating with a station operating in IBSS mode, but they never show associated unless they are really associated.

If the card is not associated, check the frequencies and the network name.

Check the WEP (encryption) parameters to ensure that they match. Note that ifconfig does not display the WEP key unless you are root.

Your card may show associated even if the WEP key doesn't match. In such a case, it knows about the network, but it can't communicate with it.

After checking all these things, you should have a connection. But you may not be home yet:

If you have a connection, check if all packets got there. Lost packets could mean line quality problems. That's not very likely on an Ethernet, but it's very possible on a PPP or DSL link. There's an uncertainty about dropped packets: you might hit Ctrl-C after the last packet went out, but before it came back. If the line is very slow, you might lose multiple packets. Compare the sequence number of the last packet that returns with the total number returned. If it's one less, all the packets except the ones at the end made it.

Check that each packet comes back only once. If not, there's definitely something wrong, or you have been pinging a broadcast address. That looks like this:

$ ping 223.147.37.255
PING 223.147.37.255  (223.147.37.255): 56 data bytes
64 bytes from 223.147.37.1: icmp_seq=0 ttl=255 time=0.428 ms
64 bytes from 223.147.37.88: icmp_seq=0 ttl=255 time=0.785 ms (DUP!)
64 bytes from 223.147.37.65: icmp_seq=0 ttl=64 time=1.818 ms (DUP!)
64 bytes from 223.147.37.1: icmp_seq=1 ttl=255 time=0.426 ms
64 bytes from 223.147.37.88: icmp_seq=1 ttl=255 time=0.442 ms (DUP!)
64 bytes from 223.147.37.65: icmp_seq=1 ttl=64 time=1.099 ms (DUP!)
64 bytes from 223.147.37.126: icmp_seq=1 ttl=255 time=45.781 ms (DUP!)

FreeBSD systems do not respond to broadcast pings, but most other systems do, so this effectively counts the number of non-BSD machines on a network.

Check the times. A ping across an Ethernet should take between about 0.2 and 2 ms, a ping across a wireless connection should take between 2 and 12 ms, a ping across an ISDN connection should take about 30 ms, a ping across a 56 kb/s analogue connection should take about 100 ms, and a ping across a satellite connection should take about 250 ms in each direction. All of these times are for idle lines, and the time can go up to over 5 seconds for a slow line transferring large blocks of data across a serial line (for example, ftping a file). In this example, some line traffic delayed the response to individual pings.

Network layer problems

Once we know the link layer is working correctly, we can turn our attention to the next layer up, the network layer. Well, first we should check if the problem is still with us.

We need additional tools for the network layer. , and the remaining ones are taking up to 5 seconds to get through. Where's the problem? Based on the recent ''upgrade'' your ISP performed, and the fact that you've had trouble getting to other sites, you suspect that the performance problems might be occurring in the ISP's net. How can you find out?

As we saw while investigating the link layer, a complete failure is often easier to fix than a partial failure. If nothing at all is getting through, you probably have a routing problem. Check the routing table with netstat. On bumble, you might see:

$ netstat -r
Routing tables

Internet:
Destination     Gateway            Flags  Refs     Use  Netif  Expire
default         gw                 UGSc      0       8    xl0
localhost       localhost          UH        2     525    lo0
223.147.37      link#1             UC        6       0    xl0
sat-gw          00:80:c6:f9:d3:fa  UHLW      0       0    xl0    1150
bumble          00:50:da:cf:17:d3  UHLW      0      24    lo0
presto          00:80:c6:f9:a6:c8  UHLW      0       5    xl0    1200
freebie         00:50:da:cf:07:35  UHLW      6  760334    xl0    1159
223.147.37.255  ff:ff:ff:ff:ff:ff  UHLW      1     403    xl0

The default route is via gw, which is correct. The first thing is to ensure that you can ping gw; that's a link level issue, so we'll assume that you can. But what if you try to ping a remote system and you see something like this?

# ping rider.fc.net
PING rider.fc.net (207.170.123.194): 56 data bytes
36 bytes from gw.example.org (223.147.37.5):	Destination Host Unreachable
Vr  HL  TOS   Len    ID    Flg   off  TTL  Pro   cks       Src      Dst
 4   5   00  6800  c5da      0  0000   fe   01  246d  223.147.37.2  207.170.123.194

36 bytes from gw.example.org (223.147.37.5):  Destination Host Unreachable
Vr  HL  TOS   Len    ID    Flg   off  TTL  Pro   cks       Src      Dst
 4   5   00  6800  c5e7      0  0000   fe   01  2460  223.147.37.2  207.170.123.194

^C
--- rider.fc.net ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

These are ICMP messages from gw indicating that it does not know where to send the data. This is almost certainly a routing problem; on gw you might see something like:

$ netstat -r
Routing tables

Internet:
Destination         Gateway            Flags  Refs       Use  Netif Expire
localhost           localhost          UH        1       123    lo0
free-gw.example.ne  exorg-gw.example.  UH       23         0   ppp0
223.147.37          link#1             UC       11         0    dc0
sat-gw              00:80:c6:f9:d3:fa  UHLW      5   1295329    dc0  1027
bumble              00:50:da:cf:17:d3  UHLW      2    760207    dc0  802
flame               08:00:20:76:6c:7b  UHLW      2    426341    dc0  532
wantadilla          00:02:44:17:f8:da  UHLW     36  19778224    dc0  1073
presto              00:80:c6:f9:a6:c8  UHLW      1   1122321    dc0  742
freebie             00:50:da:cf:07:35  UHLW     24   3279563    lo0
air-gw              00:00:b4:33:6d:a2  UHLW      4      2484    dc0  653
kimchi              00:00:21:ca:6e:f1  UHLW      0         1    dc0  829
223.147.37.127      link#1             UHLW      0         5    dc0
fumble              link#1             UHLW      3  51246373    dc0

The problem here is that there is no default route. Add it with the route command:

# route add default free-gw.example.net
# netstat -r
Routing tables

Internet:
Destination  Gateway             Flags  Refs   Use  Netif  Expire
default      free-gw.example.ne  UGSc     24  5724   ppp0
localhost    localhost           UH        1   123    lo0
...etc

See Chapter 17, page 309, for more details, including how to ensure that the routes will be added automatically at boot time.

But what if the routes look right, you don't get any ICMP messages, and no data gets through? You don't always get ICMP messages when the data can't get through. The logical next place to look is free-gw.example.net, but there's a problem with that: as the administrator of example.org, you don't have access to example.net’s machines. You can call them up, of course, but before you do you should be reasonably sure it's their problem. You can find out more information with traceroute.

traceroute

$ traceroute hub.freebsd.org
traceroute to hub.freebsd.org (204.216.27.18), 30 hops max, 40 byte packets
 1 gw(223.147.37.5)  1.138 ms  0.811 ms  0.800 ms
 2 free-gw.example.net (139.130.136.129)  131.913 ms  122.231 ms  134.694 ms
 3 Ethernet1-0.way1.Adelaide.example.net  (139.130.237.65)  118.229 ms  120.040 ms  118.723 ms
 4 Fddi0-0.way-core1.Adelaide.example.net  (139.130.237.226)  171.590 ms  117.911 ms  123.513 ms
 5 Serial5-0.lon-core1.Melbourne.example.net  (139.130.239.21)  129.267 ms  226.927 ms  125.547 ms
 6 Fddi0-0.lon5.Melbourne.example.net  (139.130.239.231)  144.372 ms  133.998 ms  136.699 ms
 7 borderx2-hssi3-0.Bloomington.mci.net  (204.70.208.121)  962.258 ms  482.393 ms  754.989 ms
 8 core2-fddi-1.Bloomington.mci.net  (204.70.208.65)  821.636 ms *  701.920 ms
 9 bordercore3-loopback.SanFrancisco.mci.net  (166.48.16.1)  424.254 ms  884.033 ms  645.302 ms
10 pb-nap.crl.net  (198.32.128.20)  435.907 ms  438.933 ms  451.173 ms
11 E0-CRL-SFO-02-E0X0.US.CEL.NET  (165.113.55.2)  440.425 ms  430.049 ms  447.340 ms
12 T1-CDROM-00-EX.US.CRL.NET  (165.113.118.2)  553.624 ms  460.116 ms *
13 hub.FreeBSD.ORG  (204.216.27.18)  642.032 ms  463.661 ms  432.976 ms

By default, traceroute tries each hop three times and prints out the times as they happen, so if the reponse time is more than about 300 ms, you'll notice it as it happens. If there is no reply after a timeout period (default 5 seconds), traceroute prints an asterisk (*). You'll also occasionally notice a significant delay at the beginning of a line, although the response time seems reasonable. In this case, the delay is probably caused by a DNS reverse lookup for the name of the system. If this becomes a problem (maybe because the global DNS servers aren't reachable), you can turn off DNS reverse lookup using the -n fag.

If you look more carefully at the times in the example above, you'll see three groups of times:

The times to gw are round 1 ms. This is typical of an Ethernet network.

The times for hops 2 to 6 are in the order of 100 to 150 ms. This indicates that the link between http://gw.example.org and http://free-gw.example.net is running PPP over a telephone line. The delay between http://free-gw.example.net and http://Fddi0-0.lon5.Mel-bourne.example.net is negligible compared to the delay across the PPP link, so you don't see much difference.

The times from borderx2-hssi3-0.Bloomington.mci.net to hub.FreeBSD.ORG are significantly higher, between 400 and 1000 ms. We also note a couple of dropped packets. This indicates that the line between Fddi0-0.lon5.Melbourne.example.net and borderx2-hssi3-0.Bloomington.mci.net is overloaded. The length of the link (about 13,000 km) also plays a role: that's a total distance of 26,000 km, which take about 85 ms to transfer. If this were a satellite connection, things would be much slower: the total distance from ground station to satellite and back to the ground is 72,000 km, which takes a total of 240 ms to propagate.

Back to our problem. If we see something like the output in the previous example, we know that there's no reason to call up the people at http://example.net: it's not their problem. This might just be overloading on the global Internet. On the other hand, what about this?

$ traceroute hub.freebsd.org
traceroute to hub.freebsd.org (204.216.27.18), 30 hops max, 40 byte packets 
1 gw(223.147.37.5)  1.138 ms  0.811 ms  0.800 ms
2  *  *  *
3  *  *  *
^C

You've fixed your routing problems, but you still can't get data off the system. There are a number of possibilities here:

The link to the next system may be down. The solution's obvious: bring it up and try again.

gw may not be configured as a gateway. You can check this with:

$ sysctl net.inet.ip.forwarding
net.inet.ip.forwarding: 1

For a router, this value should be 1. If it's 0, change it with:

# sysctl -w net.inet.ip.forwarding=1
net.inet.ip.forwarding: 0 -> 1

See page 313 for further details, including how to ensure that this sysctl is set correctly when the system starts.

You may be trying to use a non-routable IP address such as those in the range 192.168.x.x. You can't do that. If you don't have enough globally visible IP address, you'll need to run some kind of aliasing package, such as NAT. See Chapter 22, page 393, for further details.

Maybe there is something wrong with routing to your network. This is a difficult one to check, but in the case of the reference network, one possibility is to repeat the traceroute from the machine gw: gw's external address on the tun0 interface is 139.130.136.133, which is on the ISP's network. As a result, they are not affected by a routing problem for network 223.147.37.x. If this proves to be the case, contact your ISP to solve it.

Maybe there is something wrong with the other end; if everything else fails, you may have to call the admins at http://example.net even if you have no hard evidence that it's their problem.

But maybe the data gets one hop further:

$ traceroute hub.freebsd.org
traceroute to hub.freebsd.org (204.216.27.18), 30 hops max, 40 byte packets
1 gw  (223.147.37.5)  1.138 ms  0.811 ms  0.800 ms
2 free-gw.example.net  (139.130.136.129)  131.913 ms  122.231 ms  134.694 ms
3  *  *  *
4  *  *  *

In this case, there is almost certainly a problem at http://example.net. This would be the correct time to use the telephone.

High packet loss

But maybe data is getting through. Well, some data, anyway. Consider this ping session:

PING freefall.FreeBSD.org (216.136.204.21): 56 data bytes
64 bytes from 216.136.204.21: icmp_seq=0 ttl=244 time=496.426 ms
64 bytes from 216.136.204.21: icmp_seq=1 ttl=244 time=491.334 ms
64 bytes from 216.136.204.21: icmp_seq=2 ttl=244 time=479.077 ms
64 bytes from 216.136.204.21: icmp_seq=3 ttl=244 time=473.774 ms
64 bytes from 216.136.204.21: icmp_seq=4 ttl=244 time=733.429 ms
64 bytes from 216.136.204.21: icmp_seq=5 ttl=244 time=644.726 ms
64 bytes from 216.136.204.21: icmp_seq=7 ttl=244 time=490.331 ms
64 bytes from 216.136.204.21: icmp_seq=8 ttl=244 time=839.671 ms
64 bytes from 216.136.204.21: icmp_seq=9 ttl=244 time=773.764 ms
64 bytes from 216.136.204.21: icmp_seq=10 ttl=244 time=553.067 ms
64 bytes from 216.136.204.21: icmp_seq=11 ttl=244 time=454.707 ms
64 bytes from 216.136.204.21: icmp_seq=12 ttl=244 time=472.212 ms
64 bytes from 216.136.204.21: icmp_seq=13 ttl=244 time=448.322 ms
64 bytes from 216.136.204.21: icmp_seq=14 ttl=244 time=441.352 ms
64 bytes from 216.136.204.21: icmp_seq=15 ttl=244 time=455.595 ms
64 bytes from 216.136.204.21: icmp_seq=16 ttl=244 time=460.040 ms
64 bytes from 216.136.204.21: icmp_seq=17 ttl=244 time=476.943 ms
64 bytes from 216.136.204.21: icmp_seq=18 ttl=244 time=514.615 ms
64 bytes from 216.136.204.21: icmp_seq=23 ttl=244 time=538.232 ms
64 bytes from 216.136.204.21: icmp_seq=24 ttl=244 time=444.123 ms
64 bytes from 216.136.204.21: icmp_seq=25 ttl=244 time=449.075 ms
?C
--- 216.136.204.21 ping statistics ---
27 packets transmitted, 21 packets received, 22% packet loss
round-trip min/avg/max/stddev = 441.352/530.039/839.671/113.674 ms

In this case, we have a connection. But look carefully at those sequence numbers. At one point, four packets in a row (sequence 19 to 22) get lost. How high a packet drop rate is still acceptable? 1% or 2% is probably still (barely) acceptable. By the time you get to 10%, though, things look a lot worse. 10% packet drop rate doesn't mean that your connection slows down by 10%. For every dropped packet, you have a minimum delay of one second until TCP retries it. If that retried packet gets dropped too—which it will every 10 dropped packets if you have a 10% drop rate—the second retry takes another three seconds. If you're transmitting packets of 64 bytes over a 33.6 kb/s link, you can normally get about 60 packets through per second. With 10% packet loss, the time to get these packets through is about eight seconds, a throughput loss of 87.5%.

With 20% packet loss, the results are even more dramatic. Now 12 of the 60 packets have to be retried, and 2.4 of them will be retried a second time (for three seconds delay), and 0.48 of them will be retried a third time (six seconds delay). This makes a total of 22 seconds delay, a through put degradation of nearly 96%.

Theoretically, you might think that the degradation would not be as bad for big packets, such as you might have with file transfers with ftp. In fact, the situation is worse then: in most cases the packet drop rate rises sharply with the packet size, and it's common enough that ftp times out completely before it can transfer a file.

To get a better over view of what's going on, let's look at another program, tcpdump.

tcpdump

tcpdump is a program that monitors a network interface and displays selected information that passes through it. It uses the BerkeleyPacketFilter (bpf), an optional component of the kernel. It's included in recent versions of the GENERIC kernel, but it's possible to remove it. If you don't configure the BerkeleyPacketFilter, you get a message like:

tcpdump:  /dev/bpf0:  device not configured

tcpdump poses a potential security problem: you can use it to read anything that goes over the network. As a result, you must be root to run it. The simplest way to run it is without any parameters. This causes tcpdump to monitor and display all traffic on the first active network interface, normally Ethernet:

# tcpdump
tcpdump: listening on ep0
1: 13:27:57.757157 arp who-has wait.example.org tell presto.example.org
2: 13:28:06.740047 0:4c:a5:0:0:0 2:0:0:0:45:0 4011 80:
             c93c c06d c589 c06d c5ff 007b 007b 0038
             5ccb 1d03 06ee 0000 5613 0000 1093 cb15
             2512 b7e2 de6b 0ead c000 0000 0000 0000
             0000 0000 0000
3: 13:28:06.740117 freebie.example.org.ntp > 223.147.37.255.ntp: v3 bcast strat 3 p oll 6  prec -18
4: 13:28:08.004715 arp who-has wait.example.org tell presto.example.org
5: 13:28:10.987453 bumble.example.org.who > 223.147.37.255.who: udp 84
6: 13:28:13.790106 freebie.example.org.6000 > presto.example.org.1089: P 536925467:
536925851(384) ack 325114346 win 17280 <nop,nop,timestamp 155186 1163778,nop,no
p,[|tcp]> (DF)
7: 13:28:13.934336 arp who-has freebie.example.org tell presto.example.org
8: 13:28:13.934444 arp reply freebie.example.org is-at 0:a0:24:37:d:2b
9: 13:28:13.935903 presto.example.org.1089 > freebie.example.org.6000:  . ack 536925
851 win 16896 <nop,nop,timestamp 1190189 155186,nop,nop,[|tcp]> (DF)
10: 13:28:13.936313 freebie.example.org.6000 > presto.example.org.1089: P 536925851
:536926299(448) ack 325114346 win 17280 <nop,nop,timestamp 155186 1190189,nop,no
p,[|tcp]> (DF)

This output looks confusing at first. Let's look at it in more detail:

The first message shows the interface on which tcpdump listens. By default, it is the first running interface that it finds. tcpdump searches the list of interfaces in the sequence that ifconfig -a displays. Generally you can assume that it will find the primary Ethernet interface. If you want to listen on another interface, specify it on the command line. For example, to listen on a PPP interface, you would enter

# tcpdump -i tun0

At the beginning of each message is a timestamp, with a resolution of 1 /s. These times are relatively accurate; you'll frequently see time differences of less than 1 ms. In this example, the last two messages are 108 ms apart. These times are important: a lot of network problems are performance problems, and there's a big difference in performance between a net where a reply takes 100 ms sand one in which a reply takes 100 ms.

To make things easier, I have put a line number in italics at the beginning of each line. It doesn't appear in the tcpdump printout.

Line 1 shows an ARP request: system presto is looking for the Ethernet address of wait. It would appear that wait is currently not responding, since there is no reply.

Line 2 is not an IP message at all. tcpdump shows the Ethernet addresses and the beginning of the packet. We don't consider this kind of request in this book.

Line 3 is a broadcast ntp message. We looked at ntp on page 155.

Line 4 is another attempt by presto to find the IP address of wait.

Line 5 is a broadcast message from bumble on the rwho port, giving information about its current load averages and how long it has been up. See the man page rwho(1) for more information.

Line 6 is from a TCP connection between port 6000 on freebie and port 1089 on presto. It is sending 384 bytes (with the sequence numbers 536925467 to 536925851; see page 282), and is acknowledging that the last byte it received from presto had the sequence number 325114346. The window size is 17280.

Line 7 is another ARP request. presto is looking for the Ethernet address of freebie. How can that happen? We've just seen that they have a TCP connection. In fact, ARP information expires after 20 minutes. It's quite possible that all connections between presto and freebie have been dormant for this period, so presto needs to find freebie's IP address again.

Line 8 is the ARP reply from freebie to presto giving its Ethernet address.

Line 9 shows a reply from presto on the connection to freebie that we saw online 6. It acknowledges the data up to sequence number 536925851, but doesn't send any itself.

Line 10 shows another 448 bytes of data from freebie to presto, and acknowledges the same sequence number from presto as in line 6.

Packet loss revisited

Getting back to our packet loss problem, the following example shows the result of communicating on a less-than-perfect , specifically between port 1019 on and http://hub.FreeBSD.org. In addition, tcpdump reports a tos (type of service) field, which has also been removed. It doesn't interest us here.

# tcpdump -i ppp0 host hub.freebsd.org
14:16:35.990506  freebie.1019 > hub.22:  P  20:40(20) ack 77 win 17520 (DF)
14:16:36.552149  hub.22 > freebie.1019:  P  77:97(20) ack 40 win 17520 (DF)
14:16:36.722290  freebie.1019 > hub.22:  .  ack 97 win 17520 (DF)
14:16:39.344229  freebie.1019 > hub.22:  P  40:60(20) ack 97 win 17520 (DF)
14:16:41.321850  freebie.1019 > hub.22:  P  40:60(20) ack 97 win 17520 (DF)

The first line shows freebie sending bytes 20 to 40 of the stream to hub, and also acknowledging receipt of everything up to byte 77 of the stream from hub. On the next line, hub sends bytes 77 to 97 and acknowledges receiving up to byte 40 of the stream from freebie. freebie then sends another 20 bytes and acknowledges what it has received from hub.

After two seconds, freebie has not received an acknowledgment from hub that its data has been received, so on the last line it sends the packet again.

14:16:42.316150 hub.22 > freebie.1019: P 97:117(20) ack 60 win 17520 (DF) 
14:16:42.321773 freebie.1019 > hub.22:  . ack 117 win 17520 (DF)

This is the missing acknowledgment—t came another second later, along with some more data. freebie acknowledges receiving it, but doesn't send anymore data.

14:16:47.428694 freebie.1019 > hub.22:  P 60:80(20) ack 117 win 17520 (DF)
14:16:48.590805 freebie.1019 > hub.22:  P 80:100(20) ack 117 win 17520 (DF)
14:16:49.055735 freebie.1019 > hub.22:  P 100:120(20) ack 117 win 17520 (DF)
14:16:49.190703 hub.22 > freebie.1019:  P 137:157(20) ack 100 win 17520 (DF)

Five seconds later, freebie sends more data, up to byte 120 to hub. hub replies with its own data an acknowledgment up to byte 100. Unfortunately, the data it sent (bytes 137 to 157) don't line up with the last previously received data (byte 117 at 14:16:42.316150): bytes 117 to 137 are missing. freebie thus repeats the previous acknowledgment and then continues sending its data:

14:16:49.190890 freebie.1019 > hub.22:  . ack 117 win 17520 (DF)
14:16:49.538607 freebie.1019 > hub.22: P 120:140(20) ack 117 win 17520 (DF)
14:16:49.599395 hub.22 > freebie.1019: P 157:177(20) ack 120 win 17520 (DF)

Here, hub has sent yet more data, now acknowledging the data that freebie sent at 14:16:49.055735. It still hasn't sent the data in the byte range 117 to 136, so freebie resends the last acknowledgment again and continues sending data:

14:16:49.599538 freebie.1019 > hub.22:  . ack 117 win 17520 (DF)
14:16:49.620506 freebie.1019 > hub.22: P 140:160(20) ack 117 win 17520 (DF)
14:16:50.066698 hub.22 > freebie.1019: P 177:197(20) ack 140 win 17520 (DF)

Again hub has sent more data, still without sending the missing packet. freebie tries yet again, and then continues sending data:

14:16:50.066868 freebie.1019 > hub.22:  . ack 117 win 17520 (DF)
14:16:51.820708 freebie.1019 > hub.22:  P 140:160(20) ack 117 win 17520 (DF)
14:16:52.308992 hub.22 > freebie.1019:  . ack 160 win 17520 (DF)
14:16:55.251176 hub.22 > freebie.1019:  P 117:217(100) ack 160 win 17520 (DF)

Finally, hub resends the missing data, with bytes from 117 to 217. freebie is now happy, and acknowledges receipt of all the data up to 217. That's all we transmitted, so after about 1.5 seconds the two systems exchange final acknowledgments:

14:16:55.251358 freebie.1019 > hub.22: .
14:16:56.690779 hub.login > freebie.1015:
14:16:56.690941 freebie.1015 > hub.login:
ack 217 win 17420 (DF) . ack 3255467530 win 17520 . ack 1 win 17520 (DF)

This connection is less than perfect. Why? You can use traceroute to find out where it's happening, but unless the place is within your ISP's network, you can't do much about it.

Transport and application layers

If you have got this far, the chances are that things will now work. Problems in transport layer are rare. About the only things that can still cause problems are the individual applications. We'll look at some of these in the relevant chapters.

One particular problem is the Domain Name Service. This is such an integral part of the Internet Protocols that people tend to forget that it's really an application. If you get a timeout accessing a web URL, for example, there's a good chance that DNS is causing the problem. Take a look at Chapter 21, The Domain Name Service, for some ideas.

Ethereal

tcpdump is a powerful tool, but the examples above show that the output isn't the easiest thing in the world to read. An alternative is ethereal, a program in the Ports Collection (/usr/ports/net/ethereal) that displays the data in much more detail, as Figure 23-1 shows.

The screen is divided into three windows:

The top part shows individual packets (numbered 51 to 54 in this example). The line in inverse video has been selected for display in more detail.

The middle window shows the full packet. By clicking with the mouse on the boxes on the left, you can expand or reduce the amount of information being displayed.

The bottom window shows the raw data as hexadecimal and ASCII.

In practice, you'd probably want to scale the window much larger than in this example.

This image shows part of the password for a telnet login session being returned. It illustrates one of the reasons you should never use telnet to connect across the Internet.

(рис 23.1) ethereal display

Страницы: