Introduction
to
Administration
of an
Internet-based
Local Network
C R
C S
Computer Science Facilities Group
C I
L S
RUTGERS
The State University of New Jersey
Center for Computers and Information Services
Laboratory for Computer Science Research
24 July 1988
This is an introduction for people who intend to set up or administer
a network based on the Internet networking protocols (TCP/IP).
Copyright (C) 1988, Charles L. Hedrick. Anyone may reproduce this
document, in whole or in part, provided that: (1) any copy or
republication of the entire document must show Rutgers University as
the source, and must include this notice; and (2) any other use of
this material must reference this manual and Rutgers University, and
the fact that the material is copyright by Charles Hedrick and is used
by permission.
Unix is a trademark of AT&T Technologies, Inc.
Table of Contents
1. The problem 1
2. Routing and Addressing 2
3. Choosing an addressing structure 3
3.1 Should you subdivide your address space? 5
3.2 Subnets vs. multiple network numbers 5
3.3 How to allocate subnet or network numbers 6
3.3.1 Dealing with multiple "virtual" subnets on one 7
network
3.4 Choosing an address class 8
4. Setting up routing for an individual computer 9
4.1 How datagrams are routed 11
4.2 Fixed routes 13
4.3 Routing redirects 14
4.4 Other ways for hosts to find routes 16
4.4.1 Spying on Routing 16
4.4.2 Proxy ARP 17
4.4.3 Moving to New Routes After Failures 22
5. Bridges and Gateways 24
5.1 Alternative Designs 25
5.1.1 A mesh of point to point lines 26
5.1.2 Circuit switching technology 27
5.1.3 Single-level networks 27
5.1.4 Mixed designs 28
5.2 An introduction to alternative switching technologies 29
5.2.1 Repeaters 29
5.2.2 Bridges and gateways 30
5.2.3 More about bridges 32
5.2.4 More about gateways 34
5.3 Comparing the switching technologies 34
5.3.1 Isolation 35
5.3.2 Performance 36
5.3.3 Routing 37
5.3.4 Network management 39
5.3.5 A final evaluation 41
5.4 Configuring routing for gateways 44
i
This document is intended to help people who planning to set up a new
network based on the Internet protocols, or to administer an existing
one. It assumes a basic familiarity with the TCP/IP protocols,
particularly the structure of Internet addresses. A companion paper,
"Introduction to the Internet Protocols", may provide a convenient
introduction. This document does not attempt to replace technical
documentation for your specific TCP/IP implementation. Rather, it
attempts to give overall background that is not specific to any
particular implementation. It is directed specifically at networks of
"medium" complexity. That is, it is probably appropriate for a
network involving several dozen buildings. Those planning to manage
larger networks will need more preparation than you can get by reading
this document.
In a number of cases, commands and output from Berkeley Unix are
shown. Most computer systems have commands that are similar in
function to these. It seemed more useful to give some actual examples
that to limit myself to general talk, even if the actual output you
see is slightly different.
1. The problem
This document will emphasize primarily "logical" network architecture.
There are many documents and articles in the trade press that discuss
actual network media, such as Ethernet, Token Ring, etc. What is
generally not made clear in these articles is that the choice of
network media is generally not all that critical for the overall
design of a network. What can be done by the network is generally
determined more by the network protocols supported, and the quality of
the implementations. In practice, media are normally chosen based on
purely pragmatic grounds: what media are supported by the particular
types of computer that you have to connect. Generally this means that
Ethernet is used for medium-scale systems, Ethernet or a network based
on twisted-pair wiring for micro networks, and specialized high-speed
networks (typically token ring) for campus-wide backbones, and for
local networks involving super-computer and other very high-
performance applications.
Thus this document assumes that you have chosen and installed
individual networks such as Ethernet or token ring, and your vendor
has helped you connect your computers to these network. You are now
faced with the interrelated problems of
- configuring the software on your computers
- finding a way to connect individual Ethernets, token rings, etc.,
to form a single coherent network
- connecting your networks to the outside world
My primary thesis in this document is that these decisions require a
bit of advance thought. In fact, most networks need an
1
"architecture". This consists of a way of assigning addresses, a way
of doing routing, and various choices about how hosts interact with
the network. These decisions need to be made for the entire network,
preferably when it is first being installed.
2. Routing and Addressing
Many of the decisions that you need to make in setting up TCP/IP
depend upon routing, so it will be best to give a bit of background on
that topic now. I will return to routing in a later section when
discussing gateways and bridges. In general, IP datagrams pass
through many networks while they are going between the source and
destination. Here's a typical example. (Addresses used in the
examples are taken from Rutgers University.)
network 1 network 2 network 3
128.6.4 128.6.21 128.121
============================ ========== ================
| | | | | | |
___|______ _____|____ __|____|__ __|____|____ ___|________
128.6.4.2 128.6.4.3 128.6.4.1 128.6.21.1 128.121.50.2
128.6.21.2 128.121.50.1
__________ __________ __________ ____________ ____________
computer A computer B gateway R gateway S computer C
This diagram shows three normal computer systems, two gateways, and
three networks. The networks might be Ethernets, token rings, or any
other sort. Network 2 could even be a single point to point line
connecting gateways R and S.
Note that computer A can send datagrams to computer B directly, using
network 1. However it can't reach computer C directly, since they
aren't on the same network. There are several ways to connect
separate networks. This diagram assumes that gateways are used. (In a
later section, we'll look at an alternative.) In this case, datagrams
going between A and C must be sent through gateway R, network 2, and
gateway S. Every computer that uses TCP/IP needs appropriate
information and algorithms to allow it to know when datagrams must be
sent through a gateway, and to choose an appropriate gateway.
Routing is very closely tied to the choice of addresses. Note that
the address of each computer begins with the number of the network
that it's attached to. Thus 128.6.4.2 and 128.6.4.3 are both on
network 128.6.4. Next, notice that gateways, whose job is to connect
networks, have an address on each of those networks. For example,
gateway R connects networks 128.6.4 and 128.6.21. Its connection to
network 128.6.4 has the address 128.6.4.1. Its connection to network
128.6.21 has the address 128.6.21.2.
Because of this association between addresses and networks, routing
decisions can be based strictly on the network number of the
2
destination. Here's what the routing information for computer A might
look like:
From this table, computer A can tell that datagrams for computers on
network 128.6.4 can be sent directly, and datagrams for computers on
networks 128.6.21 and 128.121 need to be sent to gateway R for
forwarding. The "metric" is used by some routing algorithms as a
measure of how far away the destination is. In this case, the metric
simply indicates how many gateways the datagram has to go through.
(This is often referred to as a "hop count".)
When computer A is ready to send a datagram, it examines the
destination address. The network number is taken from the beginning
of the address and looked up in the routing table. The table entry
indicates whether the packet should be sent directly to the
destination or to a gateway.
Note that a gateway is simply a computer that is connected to two
different networks, and is prepared to forward packets between them.
In many cases it is most efficient to use special-purpose equipment
designed for use as a gateway. However it is perfectly possible to
use ordinary computers as gateways, as long as they have more than one
network interface, and their software is prepared to forward
datagrams. Most major TCP/IP implementations (even for
microcomputers) are designed to let you use your computer as a
gateway. However some of this software has limitations that can cause
trouble for your network.
3. Choosing an addressing structure
The first comment to make about addresses is a warning: Before you
start using a TCP/IP network, you must get one or more official
network numbers. TCP/IP addresses look like this: 128.6.4.3. This
address is used by one computer at Rutgers University. The first part
of it, 128.6, is a network number, allocated to Rutgers by a central
authority. Before you start allocating addresses to your computers,
you must get an official network number. Unfortunately, some people
set up networks using either a randomly-chosen number, or a number
taken from examples in vendor documentation. While this may work in
the short run, it is a very bad idea for the long run. Eventually,
you will want to connect your network to some other organization's
network. Even if your organization is highly secret and very
concerned about security, somewhere in your organization there is
going to be a research computer that ends up being connected to a
nearby university. That university will probably be connected to a
large-scale national network. As soon as one of your datagrams
3
escapes your local network, the organization you are talking to is
going to become very confused, because the addresses that appear in
your datagrams are probably officially allocated to someone else.
The solution to this is simple: get your own network number from the
beginning. It costs nothing. If you delay it, then sometime years
from now you are going to be faced with the job of changing every
address on a large network. Network numbers are currently assigned by
the DDN Network Information Center, SRI International, 333 Ravenswood
Avenue, Menlo Park, California 94025 (telephone: 800-235-3155). You
can get a network number no matter what your network is being used
for. You do not need authorization to connect to the Defense Data
Network in order to get a number. The main piece of information that
will be needed when you apply for a network number is that address
class that you want. See below for a discussion of this.
In many ways, the most important decision you have to make in setting
up a network is how you will assign Internet addresses to your
computers. This choice should be made with a view of how your network
is likely to grow. Otherwise, you will find that you have to change
addresses. When you have several hundred computers, address changes
can be nearly impossible.
Addresses are critical because Internet datagrams are routed on the
basis of their address. For example, addresses at Rutgers University
have a 2-level structure. A typical address is 128.6.4.3. 128.6 is
assigned to Rutgers University by a central authority. As far as the
outside world is concerned, 128.6 is a single network. Other
universities send any packet whose address begins with 128.6 to the
nearest Rutgers gateway. However within Rutgers, we divide up our
address space into "subnets". We use the next 8 bits of address to
indicate which subnet a computer belongs to. 128.6.4.3 belongs to
subnet 128.6.4. Generally subnets correspond to physical networks,
e.g. separate Ethernets, although as we will see later there can be
exceptions. Systems inside Rutgers, unlike those outside, contain
information about the Rutgers subnet structure. So once a packet for
128.6.4.3 arrives at Rutgers, the Rutgers network will route it to the
departmental Ethernet, token ring, or whatever, that has been assigned
subnet number 128.6.4.
When you start a network, there are several addressing decisions that
face you:
- Do you subdivide your address space?
- If so, do you use subnets or class C addresses?
- You do you allocate subnets or class C networks?
- How big an address space do you need?
4
3.1 Should you subdivide your address space?
It is not necessary to use subnets at all. There are network
technologies that allow an entire campus or company to act as a single
large logical Ethernet, so that no internal routing is necessary. If
you use this technology, then you do not need to subdivide your
address space. In that case, the only decision you have to make is
what class address to apply for. However we recommend using either a
subnet approach or some other method of subdividing your address space
in all cases:
- In section 5.2 we will argue that internal gateways are desirable
for networks of any degree of complexity.
- Even if you do not need gateways now, you may find later that you
need to use them. Thus it probably makes sense to assign
addresses as if each Ethernet or token ring was going to be a
separate subnet. This will allow for conversion to real subnets
later if it proves necessary.
- For network maintenance purposes, it is convenient to have
addresses whose structure corresponds to the structure of the
network. That is, when you see a stray packet from system
128.6.4.3, it is nice to know that all addresses beginning with
128.6.4 are in a particular building.
3.2 Subnets vs. multiple network numbers
Suppose that you have been convinced that it's a good idea to impose
some structure on your addresses. The next question is what that
structure should be. There are two basic approaches. One is subnets.
The other is multiple network numbers.
The Internet standards specify what constitutes a network number. For
numbers beginning with 128 through 191 (the most common numbers these
days), the first two octets form the network number. E.g. in
140.3.50.1, 140.3 is the network number. Network numbers are assigned
to a particular organization. What you do with the next two octets is
up to you. You could choose to make the next octet be a subnet
number, or you could use some other scheme entirely. Gateways within
your organization will be set up to know the subnetting scheme that
you are using. However outside your organization, no one will know
that 140.3.50 is one subnet and 140.3.51 is another. They will simply
know that 140.3 is your organization. Unfortunately, this ability to
add additional structure to the address via subnets was not present in
the original TCP/IP specifications. Thus some software is incapable
of being told about subnets.
If enough of the software that you are using has this problem, it may
be impractical for you to use subnets. Some organizations have used a
different approach. It is possible for an organization to apply for
5
several network numbers. Instead of dividing a single network number,
say 140.3, into several subnets, e.g. 140.3.1 through 140.3.10, you
could apply for 10 different network numbers. Thus you might be
assigned the range 140.3 through 140.12. All TCP/IP software will
know that these are different network numbers.
While using separate network numbers will work just fine within your
organization, it has two very serious disadvantages. The first, and
less serious, is that it wastes address space. There are only about
16,000 possible class B addresses. We cannot afford to waste 10 of
them on your organization, unless it is very large. This objection is
less serious because you would normally ask for class C addresses for
this purpose, and there are about 2 million possible class C
addresses.
The more serious problem with using several network numbers rather
than subnets is that it overloads the routing tables in the rest of
the Internet. As mentioned above, when you divide your network number
into subnets, this division is known within your organization, but not
outside it. Thus systems outside your organization need only one
entry in their tables in order to be able to reach you. E.g. other
universities have entries in their routing tables for 128.6, which is
the Rutgers network number. If you use a range of network numbers
instead of subnets, that division will be visible to the entire
Internet. If we used 128.6 through 128.16 instead of subdividing
128.6, other universities would need entries for each of those network
numbers in their routing tables. As of this writing the routing
tables in many of the national networks are exceeding the size of the
current routing technology. It would be considered extremely
unfriendly for any organization to use more than one network number.
This may not be a problem if your network is going to be completely
self-contained, or if only one small piece of it will be connected to
the outside world. Nevertheless, most TCP/IP experts strongly
recommend that you use subnets rather than multiple networks. The
only reason for considering multiple networks is to deal with software
that cannot handle subnets. This was a problem a few years ago, but
is currently less serious. As long as your gateways can handle
subnets, you can deal with a few individual computers that cannot by
using "proxy ARP" (see below).
3.3 How to allocate subnet or network numbers
Now that you have decided to use subnets or multiple network numbers,
you have to decide how to allocate them. Normally this is fairly
easy. Each physical network, e.g. Ethernet or token ring, is assigned
a separate subnet or network number. However you do have some
options.
In some cases it may make sense to assign several subnet numbers to a
single physical network. At Rutgers we have a single Ethernet that
spans three buildings, using repeaters. It is very clear to us that
as computers are added to this Ethernet, it is going to have to be
6
split into several separate Ethernets. In order to avoid having to
change addresses when this is done, we have allocated three different
subnet numbers to this Ethernet, one per building. (This would be
handy even if we didn't plan to split the Ethernet, just to help us
keep track of where computers are.) However before doing this, make
very sure that the software on all of your computers can handle a
network that has three different network numbers on it.
You also have to choose a "subnet mask". This is used by the software
on your systems to separate the subnet from the rest of the address.
So far we have always assumed that the first two octets are the
network number, and the next octet is the subnet number. For class B
addresses, the standards specify that the first two octets are the
network number. However we are free to choose the boundary between
the subnet number and the rest of the address. It's very common to
have a one-octet subnet number, but that's not the only possible
choice. Let's look again at a class B address, e.g. 128.6.4.3. It is
easy to see that if the third octet is used for a subnet number, there
are 256 possible subnets and within each subnet there are 256 possible
addresses. (Actually, the numbers are more like 254, since it is
generally a bad idea to use 0 or 255 for subnet numbers or addresses.)
Suppose you know that you will never have more than 128 computers on a
given subnet, but you are afraid you might need more than 256 subnets.
(For example, you might have a campus with lots of small buildings.)
In that case, you could define 10 bits for the subnet number, leaving
6 bits for addresses within each subnet. This choice is expressed by
a bit mask, using ones for the bits used by the network and subnet
number, and 0's for the bits used for individual addresses. Our
normal subnet choice is given as 255.255.255.0. If we chose 10 bit
subnet numbers and 6 bit addresses, the subnet mask would be
255.255.255.192.
Generally it is possible to specify the subnet mask for each computer
as part of configuring its TCP/IP software. The TCP/IP protocols also
allow for computers to send a query asking what the subnet mask is.
If your network supports broadcast queries, and there is at least one
computer or gateway on the network that knows the subnet mask, it may
be unnecessary to set it on the other computers. (This capability
brings with it a whole new set of possible problems. One well-known
TCP/IP implementation would answer with the wrong subnet mask when
queried, thus leading causing every other computer on the network to
be misconfigured.)
3.3.1 Dealing with multiple "virtual" subnets on one network
Most software is written under the assumption that every computer on
the local network has the same subnet number. When traffic is being
sent to a machine with a different subnet number, the software will
generally expect to find a gateway to handle forwarding to that
subnet. Let's look at the implications. Suppose subnets 128.6.19 and
128.6.20 are on the same Ethernet. Consider the way things look from
the point of view of a computer with address 128.6.19.3. It will have
7
no problem sending to other machines with addresses 128.6.19.x. They
are on the same subnet, and so our computer will know to send directly
to them on the local Ethernet. However suppose it is asked to send a
packet to 128.6.20.2. Since this is a different subnet, most software
will expect to find a gateway that handles forwarding between the two
subnets. Of course there isn't a gateway between subnets 128.6.19 and
128.6.20, since they are on the same Ethernet. Thus it must be
possible to tell your software that 128.6.20 is actually on the same
Ethernet.
For the most common TCP/IP implementations, it is possible to deal
with more than one subnet on a network. For example, Berkeley Unix
allows you to define gateways using a command "route add". Suppose
that you get from subnet 128.6.19 to subnet 128.6.4 using a gateway
whose address is 128.6.19.1. You would use the command
route add 128.6.4.0 128.6.19.1 1
This says that to reach subnet 128.6.4, traffic should be sent via the
gateway at 128.6.19.1, and that the route only has to go through one
gateway. The "1" is referred to as the "routing metric". If you use
a metric of 0, you are saying that the destination subnet is on the
same network, and no gateway is needed. In our example, on system
128.6.19.3, you would use
route add 128.6.20.0 128.6.19.1 0
The actual address used in place of 128.6.19.1 is irrelevant. The
metric of 0 says that no gateway is actually going to be used, so the
gateway address is not used. However it must be a legal address on
the local network.
Note that the commands in this section are simply examples. You should
look in the documentation for your particular implementation to see
how to configure your routing.
3.4 Choosing an address class
When you apply for an official network number, you will be asked what
class of network number you need. The possible answers are A, B, and
C. This affects how large an address space you will be allocated.
Class A addresses are one octet long, class B addresses are 2 octets,
and class C addresses are 3 octets. This represents a tradeoff:
there are a lot more class C addresses than class A addresses, but the
class C addresses don't allow as many hosts. The idea was that there
would be a few very large networks, a moderate number of medium-size
ones, and a lot of mom-and-pop stores that would have small networks.
Here is a table showing the distinction:
class range of first octet network rest possible addresses
A 1 - 126 p q.r.s 16777214
B 128 - 191 p.q r.s 65534
8
C 192 - 223 p.q.r s 254
For example network 10, a class A network, has addresses between
10.0.0.1 and 10.255.255.254. So it allows 254**3, or about 16 million
possible addresses. (Actually, network 10 has allocated addresses
where some of the octets are zero, so there are a few more networks
possible.) Network 192.12.88, a class C network has hosts between
192.12.88.1 and 128.12.88.254, i.e. 254 possible hosts.
In general, you will be expected to choose the lowest class that will
provide you with enough addresses to handle your growth over the next
few years. In general organizations that have computers in many
buildings will probably need and be able to get a class B address,
assuming that they are going to use subnetting. (If you are going to
use many separate network numbers, you would ask for a number of class
C addresses.) Class A addresses are normally used only for large
public networks and for a few very large corporate networks.
4. Setting up routing for an individual computer
All TCP/IP implementations require some configuration for each host.
In some cases this is done in a "system generation". In other cases,
various startup and configuration files must be set up on the system.
Still other systems get configuration information across the network
from a "server". While the details differ, the same kinds of
information need to be supplied for most implementations. This
includes
- parameters describing the specific machine, such as its Internet
address.
- parameters describing the network, such as the subnet mask (if
any)
- routing software and the tables that drive it
- startup of various programs needed to handle network tasks
Before a machine is installed on your network, a coordinator should
assign it a host name and Internet address. If the machine has more
than one network interface, you must assign a separate Internet
address for each. (In most cases, the same host name can be used.
The name goes with the machine as a whole, whereas the address is
associated with the connection to a specific network.) The address
should begin with the network number for the network to which it is to
be attached. We recommend that you assign addresses starting from 1.
Should you find that you need more subnets than your current subnet
mask allows, you may later need to expand the subnet mask to use more
bits. If all addresses use small numbers, this will be possible.
Generally the Internet address must be specified individually in a
configuration file on each computer. However some computers
9
(particularly those without permanent disks on which configuration
information could be stored) find out their Internet address by
sending a broadcast request over the network. In that case, you will
have to make sure that some other system is configured to answer the
request. When a system asks for its Internet address, enough
information must be put into the request to allow another system to
recognize it and to send a response back. For Ethernet systems,
generally the request will include the Ethernet address of the
requesting system. Ethernet addresses are assigned by the computer
manufacturers, and are guaranteed to be unique. Thus they are a good
way of identifying the computer. And of course the Ethernet address
is also needed in order to send the response back. If it is used as
the basis for address lookup, the system that is to answer the request
will need a table of Ethernet addresses and the corresponding Internet
address. The only problem in constructing this table will be finding
the Ethernet address for each computer. Generally, computers are
designed so that they print the Ethernet address on the console
shortly after being turned on. However in some cases you may have to
type a command that displays information about the Ethernet interface.
Generally the subnet mask should be specified in a configuration file
associated with the computer. (For Unix systems, the "ifconfig"
command is used to specify both the Internet address and subnet mask.)
However there are provisions in the IP protocols for a computer to
broadcast a request asking for the subnet mask. The subnet mask is an
attribute of the network. That is, it is the same for all computers
on a given subnet. Thus there is no separate subnet table
corresponding to the Ethernet/Internet address mapping table used to
answer address queries. Generally any machine on the network that
believes it knows the subnet mask will answer any query about the
subnet mask. For that reason, an incorrect subnet mask setting on one
machine can cause confusion throughout the network.
Normally the configuration files do roughly the following things:
- enable each of the network interfaces (Ethernet interface, serial
lines, etc.) Normally this involves specifying an Internet
address and subnet mask for each, as well as other options that
will be described in your vendor's documentation.
- establish network routing information, either by commands that
add fixed routes, or by starting a program that obtains them
dynamically.
- turn on the name server (used for looking up names and finding
the corresponding Internet address -- see the section on the
domain system in the Introduction to TCP/IP).
- set various other information needed by the system software, such
as the name of the system itself.
- start various "daemons". These are programs that provide network
services to other systems on the network, and to users on this
system.
10
It is not practical to document these steps in detail, since they
differ for each vendor. This section will concentrate on a few issues
where your choice will depend upon overall decisions about how your
network is to operate. These overall network policy decisions are
often not as well documented by the vendors as the details of how to
start specific programs. Note that some care will be necessary to
integrate commands that you add for routing, etc., into the startup
sequence at the right point. Some of the most mysterious problems
occur when network routing is not set up before a program needs to
make a network query, or when a program attempts to look up a host
name before the name server has finished loading all of the names from
a master name server.
4.1 How datagrams are routed
If your system consists of a single Ethernet or similar medium, you do
not need to give routing much attention. However for more complex
systems, each of your machines needs a routing table that lists a
gateway and interface to use for every possible destination network.
A simple example of this was given at the beginning of this section.
However it is now necessary to describe the way routing works in a bit
more detail. On most systems, the routing table looks something like
the following. (This example was taken from a system running Berkeley
Unix, using the command "netstat -n -r". Some columns containing
statistical information have been omitted.)
Destination Gateway Flags Interface
128.6.5.3 128.6.7.1 UHGD il0
128.6.5.21 128.6.7.1 UHGD il0
127.0.0.1 127.0.0.1 UH lo0
128.6.4 128.6.4.61 U pe0
128.6.6 128.6.7.26 U il0
128.6.7 128.6.7.26 U il0
128.6.2 128.6.7.1 UG il0
10 128.6.4.27 UG pe0
128.121 128.6.4.27 UG pe0
default 128.6.4.27 UG pe0
The first column shows the designation for the controller hardware
that connects the computer to that Ethernet. (This system happens to
have controllers from two different vendors. The first one is made by
Interlan, the second by Pyramid.) The second column is the network
number for the network. The third column is this computer's Internet
address on that network. The last column shows other subnets that
share the same physical network.
11
Now let's look at the routing table. For the moment, let us ignore
the first 3 lines. The majority of the table consists of a set of
entries describing networks. For each network, the other three
columns show where to send datagrams destined for that network. If
the "G" flag is present in the third column, datagrams for that
network must be sent through a gateway. The second column shows the
address of the gateway to be used. If the "G" flag is not present,
the computer is directly connected to the network in question. So
datagrams for that network should be sent using the controller shown
in the third column. The "U" flag in the third column simply
indicates that the route specified by that line is up, i.e. that no
errors have occured indicating that the path is unusable.
The first 3 lines show "host routes", indicated by the "H" flag in
column three. Routing tables normally have entries for entire
networks or subnets. For example, the entry
128.6.2 128.6.7.1 UG il0
indicates that datagrams for any computer on network 128.6.2 (i.e.
addresses 128.6.2.1 through 128.6.2.254) should be sent to gateway
128.6.7.1 for forwarding. However sometimes routes apply only to a
specific computer, rather than to a whole network. In that case, a
host route is used. The first column then shows a complete address,
and the "H" flag is present in column 3. E.g. the entry
128.6.5.21 128.6.7.1 UHGD il0
indicates that datagrams for the specific address 128.6.5.21 should be
sent to the gateway 128.6.7.1. As with network routes, the "G" flag
is used for routes that involve a gateway. The "D" flag indicates
that the route was added dynamically, based on an ICMP redirect
message from a gateway. (See below for details.)
The following route is special:
127.0.0.1 127.0.0.1 UH lo0
127.0.0.1 is the address of the "loopback device". This is a dummy
software module. Any datagram sent out through that "device" appears
immediately as input. It can be used for testing. The loopback
address is also handy for sending queries to programs that are
designed to respond to network queries, but happen to be running on
the same computer. (Why bother to use your network to talk to a
program that is on the same machine you are?)
Finally, there are "default" routes, e.g.
default 128.6.4.27 UG pe0
This route is used for datagrams that don't match any other entry. In
this case, they are sent to a gateway with address 128.6.4.27.
In most systems, datagrams are routed by looking up the destination
address in a table such as the one just described. If the address
12
matches a specific host route, then that is used. Otherwise, if it
matches a network route, that is used. If no other route works, the
default is used. If there is no default, normally the user gets an
error message such as "network is unreachable".
The following sections will describe several ways of setting up these
routing tables. Generally, the actual operation of sending packets
doesn't depend upon which method you use to set up the routes. When a
packet is to be sent, its destination is looked up in the table. The
different routing methods are simply more and less sophisticated ways
of setting up and maintaining the tables.
4.2 Fixed routes
The simplest way of doing routing is to have your configuration
contain commands to set up the routing table at startup, and then
leave it alone. This method is practical for relatively small
networks, particularly if they don't change very often.
Most computers automatically set up some routing entries for you.
Unix will add an entry for the networks to which you are directly
connected. For example, your startup file might contain the commands
These commands specify that in order to reach network 128.6.2, a
gateway at address 128.6.4.1 should be used, and that network 128.6.6
is actually an additional network number for the physical network
connected to interface 128.6.5.35. Some other software might use
different commands for these cases. Unix differentiates them by the
"metric", which is the number at the end of the command. The metric
indicates how many gateways the datagram will have to go through to
get to the destination. Routes with metrics of 1 or greater specify
the address of the first gateway on the path. Routes with metrics of
13
0 indicate that no gateway is involved -- this is an additional
network number for the local network.
Finally, you might define a default route, to be used for destinations
not listed explicitly. This would normally show the address of a
gateway that has enough information to handle all possible
destinations.
If your network has only one gateway attached to it, then of course
all you need is a single entry pointing to it as a default. In that
case, you need not worry further about setting up routing on your
hosts. (The gateway itself needs more attention, as we will see.)
The following sections are intended to provide help for setting up
networks where there are several different gateways.
4.3 Routing redirects
Most Internet experts recommend leaving routing decisions to the
gateways. That is, it is probably a bad idea to have large fixed
routing tables on each computer. The problem is that when something
on the network changes, you have to go around to many computers and
update the tables. If changes happen because a line goes down,
service may not be restored until someone has a chance to notice the
problem and change all the routing tables.
The simplest way to keep routes up to date is to depend upon a single
gateway to update your routing tables. This gateway should be set as
your default. (On Unix, this would mean a command such as "route add
default 128.6.4.27 1", where 128.6.4.27 is the address of the
gateway.) As described above, your system will send all datagrams to
the default when it doesn't have any better route. At first, this
strategy does not sound very good if you have more than one gateway.
After all, if all you have is a single default entry, how will you
ever use the other gateways in the cases where they are better? The
answer is that most gateways are able to send "redirects" when they
get datagrams for which there is a better route. A redirect is a
specific kind of message using the ICMP (Internet Control Message
Protocol). It contains information that generally translates to "In
the future, to get to address XXXXX, please use gateway YYYYY instead
of me". Correct TCP/IP implementations use these redirects to add
entries to their routing table. Suppose your routing table starts out
as follows:
Destination Gateway Flags Interface
127.0.0.1 127.0.0.1 UH lo0
128.6.4 128.6.4.61 U pe0
default 128.6.4.27 UG pe0
This contains an entry for the local network, 128.6.4, and a default
pointing to the gateway 128.6.4.27. Suppose there is also a gateway
128.6.4.30, which is the best way to get to network 128.6.7. How do
14
you find it? Suppose you have datagrams to send to 128.6.7.23. The
first datagram will go to the default gateway, since that's the only
thing in the routing table. However the default gateway, 128.6.4.27,
will notice that 128.6.4.30 would really be a better route. (How it
does that is up to the gateway. However there are some fairly simple
methods for a gateway to determine that you would be better off using
a different one.) Thus 128.6.4.27 will send back a redirect
specifying that packets for 128.6.7.23 should be sent via 128.6.4.30.
Your TCP/IP software will add a routing entry
128.6.7.23 128.6.4.30 UDHG pe0
Any future datagrams for 128.6.7.23 will be sent directly to the
appropriate gateway.
This strategy would be a complete solution, if it weren't for three
problems:
- It requires each computer to have the address of one gateway
"hardwired" into its startup files, as the initial default.
- If a gateway goes down, routing table entries using it may not be
removed.
- If your network uses subnets, and your TCP/IP implementation does
not handle them, this strategy will not work.
How serious the first problem is depends upon your situation. For
small networks, there is no problem modifying startup files whenever
something changes. But some organizations can find it very painful.
If network topology changes, and a gateway is removed, any systems
that have that gateway as their default must be adjusted. This is
particularly serious if the people who maintain the network are not
the same as those maintaining the individual systems. One simple
appoach is to make sure that the default address never changes. For
example, you might adopt the convention that address 1 on each subnet
is the default gateway for that subnet. For example, on subnet
128.6.7, the default gateway would always be 128.6.7.1. If that
gateway is ever removed, some other gateway is given that address.
(There must always be at least one gateway left to give it to. If
there isn't, you are completely cut off anyway.)
The biggest problem with the description given so far is that it tells
you how to add routes but not how to get rid of them. What happens if
a gateway goes down? You want traffic to be redirected back to a
gateway that is up. Unfortunately, a gateway that has crashed is not
going to issue Redirects. One solution is to choose very reliable
gateways. If they crash very seldom, this may not be a problem. Note
that Redirects can be used to handle some kinds of network failure.
If a line goes down, your current route may no longer be a good one.
As long as the gateway to which you are talking is still up and
talking to you, it can simply issue a Redirect to the gateway that is
now the best one. However you still need a way to detect failure of
one of the gateways that you are talking to directly.
15
The best approach for handling failed gateways is for your TCP/IP
implementation to detect routes that have failed. TCP maintains
various timers that allow the software to detect when a connection has
broken. When this happens, one good approach is to mark the route
down, and go back to the default gateway. A similar approach can also
be used to handle failures in the default gateway. If you have mark
two gateways as default, then the software should be capable of
switching when connections using one of them start failing.
Unfortunately, some common TCP/IP implementations do not mark routes
as down and change to new ones. (In particular Berkeley 4.2 Unix does
not.) However Berkeley 4.3 Unix does do this, and as other vendors
begin to base products on 4.3 rather than 4.2, this ability is
expected to be more common.
4.4 Other ways for hosts to find routes
As long as your TCP/IP implementations handle failing connections
properly, establishing one or more default routes in the configuration
file is likely to be the simplest way to handle routing. However
there are two other routing approaches that are worth considering for
special situations:
- spying on the routing protocol
- using proxy ARP
4.4.1 Spying on Routing
Gateways generally have a special protocol that they use among
themselves. Note that redirects cannot be used by gateways.
Redirects are simply ways for gateways to tell "dumb" hosts to use a
different gateway. The gateways themselves must have a complete
picture of the network, and a way to compute the optimal route to each
subnet. Generally they maintain this picture by exchanging
information among themselves. There are several different routing
protocols in use for this purpose. One way for a computer to keep
track of gateways is for it to listen to the gateways' messages.
There is software available for this purpose for most of the common
routing protocols. When you run this software, it maintains a
complete picture of the network, just as the gateways do. The
software is generally designed to maintain your computer's routing
tables dynamically, so that datagrams are always sent to the proper
gateway. In effect, the routing software issues the equivalent of the
Unix "route add" and "route delete" commands as the network topology
changes. Generally this results in a complete routing table, rather
than one that depends upon default routes. (This assumes that the
gateways themselves maintain a complete table. Sometimes gateways
keep track of your campus network completely, but use a default route
for all off-campus networks, etc.)
16
Running routing software on each host does in some sense "solve" the
routing problem. However there are several reasons why this is not
normally recommended except as a last resort. The most serious
problem is that this reintroduces configuration options that must be
kept up to date on each host. Any computer that wants to participate
in the protocol among the gateways will need to configure its software
compatibly with the gateways. Modern gateways often have
configuration options that are complex compared with those of an
individual host. It is undesirable to spread these to every host.
There is a somewhat more specialized problem that applies only to
diskless computers. By its very nature, a diskless computer depends
upon the network and file servers to load programs and to do swapping.
It is dangerous for diskless computers to run any software that
listens to network broadcasts. Routing software generally depends
upon broadcasts. For example, each gateway on the network might
broadcast its routing tables every 30 seconds. The problem with
diskless nodes is that the software to listen to these broadcasts must
be loaded over the network. On a busy computer, programs that are not
used for a few seconds will be swapped or paged out. When they are
activated again, they must be swapped or paged in. Whenever a
broadcast is sent, every computer on the network needs to activate the
routing software in order to process the broadcast. This means that
many diskless computers will be doing swapping or paging at the same
time. This is likely to cause a temporary overload of the network.
Thus it is very unwise for diskless machines to run any software that
requires them to listen to broadcasts.
4.4.2 Proxy ARP
Proxy ARP is an alternative technique for letting gateways make all
the routing decisions. It is applicable to any broadcast network that
uses ARP or a similar technique for mapping Internet addresses into
network-specific addresses such as Ethernet addresses. This
presentation will assume Ethernet. Other network types can be
acccomodated if you replace "Ethernet address" with the appropriate
network-specific address, and ARP with the protocol used for address
mapping by that network type.
In many ways proxy ARP it is similar to using a default route and
redirects, however it uses a different mechanism to communicate routes
to the host. With redirects, a full routing table is used. At any
given moment, the host knows what gateways it is routing datagrams to.
With proxy ARP, you dispense with explicit routing tables, and do
everything at the level of Ethernet addresses. Proxy ARP can be used
for all destinations, only for destinations within your network, or in
various combinations. It will be simplest to explain it as used for
all addresses. To do this, you instruct the host to pretend that
every computer in the world is attached directly to your local
Ethernet. On Unix, this would be done using a command
route add default 128.6.4.2 0
17
where 128.6.4.2 is assumed to be the Internet address of your host.
As explained above, the metric of 0 causes everything that matches
this route to be sent directly on the local Ethernet.
When a datagram is to be sent to a local Ethernet destination, your
computer needs to know the Ethernet address of the destination. In
order to find that, it uses something generally called the ARP table.
This is simply a mapping from Internet address to Ethernet address.
Here's a typical ARP table. (On our system, it is displayed using the
command "arp -a".)
FOKKER.RUTGERS.EDU (128.6.5.16) at 8:0:20:0:8:22 temporary
CROSBY.RUTGERS.EDU (128.6.5.48) at 2:60:8c:49:50:63 temporary
CAIP.RUTGERS.EDU (128.6.4.16) at 8:0:8b:0:1:6f temporary
DUDE.RUTGERS.EDU (128.6.20.16) at 2:7:1:0:eb:cd temporary
W20NS.MIT.EDU (18.70.0.160) at 2:7:1:0:eb:cd temporary
OBERON.USC.EDU (128.125.1.1) at 2:7:1:2:18:ee temporary
gatech.edu (128.61.1.1) at 2:7:1:0:eb:cd temporary
DARTAGNAN.RUTGERS.EDU (128.6.5.65) at 8:0:20:0:15:a9 temporary
Note that it is simply a list of Internet addresses and the
corresponding Ethernet address. The "temporary" indicates that the
entry was added dynamically using ARP, rather than being put into the
table manually.
If there is an entry for the address in the ARP table, the datagram is
simply put on the Ethernet with the corresponding Ethernet address.
If not, an "ARP request" is broadcast, asking for the destination host
to identify itself. This request is in effect a question "will the
host with Internet address 128.6.4.194 please tell me what your
Ethernet address is?". When a response comes back, it is added to the
ARP table, and future datagrams for that destination can be sent
without delay.
This mechanism was originally designed only for use with hosts
attached directly to a single Ethernet. If you need to talk to a host
on a different Ethernet, it was assumed that your routing table would
direct you to a gateway. The gateway would of course have one
interface on your Ethernet. Your computer would then end up looking
up the address of that gateway using ARP. It would generally be
useless to expect ARP to work directly with a computer on a distant
network. Since it isn't on the same Ethernet, there's no Ethernet
address you can use to send datagrams to it. And when you send an ARP
request for it, there's nobody to answer the request.
Proxy ARP is based on the concept that the gateways will act as
proxies for distant hosts. Suppose you have a host on network
128.6.5, with address 128.6.5.2. (computer A in diagram below) It
wants to send a datagram to host 128.6.4.194, which is on a different
Ethernet (subnet 128.6.4). (computer C in diagram below) There is a
gateway connecting the two subnets, with address 128.6.5.1 (gateway
R):
18
network 1 network 2
128.6.5 128.6.4
============================ ==================
| | | | | |
___|______ _____|____ __|____|__ __|____|____
128.6.5.2 128.6.5.3 128.6.5.1 128.6.4.194
128.6.4.1
__________ __________ __________ ____________
computer A computer B gateway R computer C
Now suppose computer A sends an ARP request for computer C. C isn't
able to answer for itself. It's on a different network, and never
even sees the ARP request. However gateway R can act on its behalf.
In effect, your computer asks "will the host with Internet address
128.6.4.194 please tell me what your Ethernet address is?", and the
gateway says "here I am, 128.6.4.194 is 2:7:1:0:eb:cd", where
2:7:1:0:eb:cd is actually the Ethernet address of the gateway. This
bit of illusion works just fine. Your host now thinks that
128.6.4.194 is attached to the local Ethernet with address
2:7:1:0:eb:cd. Of course it isn't. But it works anyway. Whenever
there's a datagram to be sent to 128.6.4.194, your host sends it to
the specified Ethernet address. Since that's the address of a gateway
R, the gateway gets the packet. It then forwards it to the
destination.
Note that the net effect is exactly the same as having an entry in the
routing table saying to route destination 128.6.4.194 to gateway
128.6.5.1:
128.6.4.194 128.6.5.1 UGH pe0
except that instead of having the routing done at the level of the
routing table, it is done at the level of the ARP table.
Generally it's better to use the routing table. That's what it's
there for. However here are some cases where proxy ARP makes sense:
- when you have a host that does not implement subnets
- when you have a host that does not respond properly to redirects
- when you do not want to have to choose a specific default gateway
- when your software is unable to recover from a failed route
The technique was first designed to handle hosts that do not support
subnets. Suppose that you have a subnetted network. For example, you
have chosen to break network 128.6 into subnets, so that 128.6.4 and
128.6.5 are separate. Suppose you have a computer that does not
understand subnets. It will assume that all of 128.6 is a single
network. Thus it will be difficult to establish routing table entries
to handle the configuration above. You can't tell it about the
gateway explicitly using "route add 128.6.4.0 128.6.5.1 1" Since it
thinks all of 128.6 is a single network, it can't understand that you
19
are trying to tell it where to send one subnet. It will instead
interpret this command as an attempt to set up a host route to a host
who address is 128.6.4.0. The only thing that would work would be to
establish explicit host routes for every individual host on every
other subnet. You can't depend upon default gateways and redirects in
this situation either. Suppose you said "route add default 128.6.5.1
1". This would establish the gateway 128.6.5.1 as a default. However
the system wouldn't use it to send packets to other subnets. Suppose
the host is 128.6.5.2, and wants to send a datagram to 128.6.4.194.
Since the destination is part of 128.6, your computer considers it to
be on the same network as itself, and doesn't bother to look for a
gateway.
Proxy ARP solves this problem by making the world look the way the
defective implementation expects it to look. Since the host thinks
all other subnets are part of its own network, it will simply issue
ARP requests for them. It expects to get back an Ethernet address
that can be used to establish direct communications. If the gateway
is practicing proxy ARP, it will respond with the gateway's Ethernet
address. Thus datagrams are sent to the gateway, and everything
works.
As you can see, no specific configuration is need to use proxy ARP
with a host that doesn't understand subnets. All you need is for your
gateways to implement proxy ARP. In order to use it for other
purposes, you must explicitly set up the routing table to cause ARP to
be used. By default, TCP/IP implementations will expect to find a
gateway for any destination that is on a different network. In order
to make them issue ARP's, you must explicitly install a route with
metric 0, as in the example "route add default 128.6.5.2 0".
It is obvious that proxy ARP is reasonable in situations where you
have hosts that don't understand subnets. Some comments may be needed
on the other situations. Generally TCP/IP implementations do handle
ICMP redirects properly. Thus it is normally practical to set up a
default route to some gateway, and depend upon the gateway to issue
redirects for destinations that should use a different gateway.
However in case you ever run into an implementation that does not obey
redirects, or cannot be configured to have a default gateway, you may
be able to make things work by depending upon proxy ARP. Of course
this requires that you be able to configure the host to issue ARP's
for all destinations. You will need to read the documentation
carefully to see exactly what routing features your implementation
has.
Sometimes you may choose to depend upon proxy ARP for convenience.
The problem with routing tables is that you have to configure them.
The simplest configuration is simply to establish a default route, but
even there you have to supply some equivalent to the Unix command
"route add default ...". Should you change the addresses of your
gateways, you have to modify this command on all of your hosts, so
that they point to the new default gateway. If you set up a default
route that depends upon proxy ARP (i.e. has metric 0), you won't have
to change your configuration files when gateways change. With proxy
ARP, no gateway addresses are given explicitly. Any gateway can
20
respond to the ARP request, no matter what its address.
In order to save you from having to do configuration, some TCP/IP
implementations default to using ARP when they have no other route.
The most flexible implementations allow you to mix strategies. That
is, if you have specified a route for a particular network, or a
default route, they will use that route. But if there is no route for
a destination, they will treat it as local, and issue an ARP request.
As long as your gateways support proxy ARP, this allows such hosts to
reach any destination without any need for routing tables.
Finally, you may choose to use proxy ARP because it provides better
recovery from failure. This choice is very much dependent upon your
implementation. The next section will discuss the tradeoffs in more
detail.
In situations where there are several gateways attached to your
network, you may wonder how proxy ARP allows you to choose the best
one. As described above, your computer simply sends a broadcast
asking for the Ethernet address for a destination. We assumed that
the gateways would be set up to respond to this broadcast. If there
is more than one gateway, this requires coordination among them.
Ideally, the gateways will have a complete picture of the network
topology. Thus they are able to determine the best route from your
host to any destination. If the gateway coordinate among themselves,
it should be possible for the best gateway to respond to your ARP
request. In practice, it may not always be possible for this to
happen. It is fairly easy to design algorithms to prevent very bad
routes. For example, consider the following situation:
1 2 3
------- A ---------- B ----------
1, 2, and 3 are networks. A and B are gateways, connecting network 2
to 1 or 3. If a host on network 2 wants to talk to a host on network
1, it is fairly easy for gateway A to decide to answer, and for
gateway B to decide not to. Here's how: if gateway B accepted a
datagram for network 1, it would have to forward it to gateway A for
delivery. This would mean that it would take a packet from network 2
and send it right back out on network 2. It is very easy to test for
routes that involve this sort of circularity. It is much harder to
deal with a situation such as the following:
1
---------------
A B
| | 4
| |
3 | C
| |
| | 5
D E
---------------
2
21
Suppose a computer on network 1 wants to send a datagram to one on
network 2. The route via A and D is probably better, because it goes
through only one intermediate network (3). It is also possible to go
via B, C, and E, but that path is probably slightly slower. Now
suppose the computer on network 1 sends an ARP request for a
destination on 2. It is likely that A and B will both respond to that
request. B is not quite as good a route as A. However it is not so
bad as the case above. B won't have to send the datagram right back
out onto network 1. It is unable to determine there is a better
alternative route without doing a significant amount of global
analysis on the network. This may not be practical in the amount of
time available to process an ARP request.
4.4.3 Moving to New Routes After Failures
In principle, TCP/IP routing is capable of handling line failures and
gateway crashes. There are various mechanisms to adjust routing
tables and ARP tables to keep them up to date. Unfortunately, many
major implementations of TCP/IP have not implemented all of these
mechanisms. The net result is that you have to look carefully at the
documentation for your implementation, and consider what kinds of
failures are most likely. You then have to choose a strategy that
will work best for your site. The basic choices for finding routes
have all been listed above: spying on the gateways' routing protocol,
setting up a default route and depending upon redirects, and using
proxy ARP. These methods all have their own limitations in dealing
with a changing network.
Spying on the gateways' routing protocol is theoretically the cleanest
solution. Assuming that the gateways use good routing technology, the
tables that they broadcast contain enough information to maintain
optimal routes to all destinations. Should something in the network
change (a line or a gateway goes down), this information will be
reflected in the tables, and the routing software will be able to
update the hosts' routing tables appropriately. The disadvantages are
entirely practical. However in some situations the robustness of this
approach may outweight the disadvantages. To summarize the discussion
above, the disadvantages are:
- If the gateways are using sophisticated routing protocols,
configuration may be fairly complex. Thus you will be faced with
setting up and maintaining configuration files on every host.
- Some gateways use proprietary routing protocols. In this case,
you may not be able to find software for your hosts that
understands them.
- If your hosts are diskless, there can be very serious performance
problems associated with listening to routing broadcasts.
Some gateways may be able to convert from their internal routing
protocol to a simpler one for use by your hosts. This could largely
22
bypass the first two disadvantages. Currently there is no known way
to get around the third one.
The problems with default routes/redirects and with proxy ARP are
similar: they both have trouble dealing with situations where their
table entries no longer apply. The only real difference is that
different tables are involved. Suppose a gateway goes down. If any
of your current routes are using that gateway, you may be in trouble.
If you are depending upon the routing table, the major mechanism for
adjusting routes is the redirect. This works fine in two situations:
- where the default gateway is not the best route. The default
gateway can direct you to a better gateway
- where a distant line or gateway fails. If this changes the best
route, the current gateway can redirect you to the gateway that
is now best
The case it does not protect you against is where the gateway that you
are currently sending your datagrams to crashes. Since it is down, it
is unable to redirect you to another gateway. In many cases, you are
also unprotected if your default gateway goes down, since there
routing starts by sending to the default gateway.
The situation with proxy ARP is similar. If the gateways coordinate
themselves properly, the right one will respond initially. If
something elsewhere in the network changes, the gateway you are
currently issuing can issue a redirect to a new gateway that is
better. (It is usually possible to use redirects to override routes
established by proxy ARP.) Again, the case you are not protected
against is where the gateway you are currently using crashes. There
is no equivalent to failure of a default gateway, since any gateway
can respond to the ARP request.
So the big problem is that failure of a gateway you are using is hard
to recover from. It's hard because the main mechanism for changing
routes is the redirect, and a gateway that is down can't issue
redirects. Ideally, this problem should be handled by your TCP/IP
implementation, using timeouts. If a computer stops getting response,
it should cancel the existing route, and try to establish a new one.
Where you are using a default route, this means that the TCP/IP
implementation must be able to declare a route as down based on a
timeout. If you have been redirected to a non-default gateway, and
that route is declared down, traffic will return to the default. The
default gateway can then begin handling the traffic, or redirect it to
a different gateway. To handle failure of a default gateway, it
should be possible to have more than one default. If one is declared
down, another will be used. Together, these mechanisms should take
care of any failure.
Similar mechanisms can be used by systems that depend upon proxy ARP.
If a connection is timing out, the ARP table entry that it uses should
be cleared. This will cause a new ARP request, which can be handled
by a gateway that is still up. A simpler mechanism would simply be to
time out all ARP entries after some period. Since making a new ARP
23
request has a very low overhead, there's no problem with removing an
ARP entry even if it is still good. The next time a datagram is to be
sent, a new request will be made. The response is normally fast
enough that users will not even notice the delay.
Unfortunately, many common implementations do not use these
strategies. In Berkeley 4.2, there is no automatic way of getting rid
of any kind of entry, either routing or ARP. They do not invalidate
routes on timeout nor ARP entries. ARP entries last forever. If
gateway crashes are a significant problem, there may be no choice but
to run software that listens to the routing protocol. In Berkeley
4.3, routing entries are removed when TCP connections are failing.
ARP entries are still not removed. This makes the default route
strategy more attractive for 4.3 than proxy ARP. Having more than one
default route may also allow for recovery from failure of a default
gateway. Note however that 4.3 only handles timeout for connections
using TCP. If a route is being used only by services based on UDP, it
will not recover from gateway failure. While the "traditional" TCP/IP
services use TCP, network file systems generally do not. Thus
4.3-based systems still may not always be able to recover from
failure.
In general, you should examine your implementation in detail to
determine what sort of error recovery strategy it uses. We hope that
the discussion in this section will then help you choose the best way
of dealing with routing.
There is one more strategy that some older implementations use. It is
strongly discouraged, but we mention it here so you can recognize it
if you see it. Some implementations detect gateway failure by taking
active measure to see what gateways are up. The best version of this
is based on a list of all gateways that are currently in use. (This
can be determined from the routing table.) Every minute or so, an
echo request datagram is sent to each such gateway. If a gateway
stops responding to echo requests, it is declared down, and all routes
using it revert to the default. With such an implementation, you
normally supply more than one default gateway. If the current default
stops responding, an alternate is chosen. In some cases, it is not
even necessary to choose an explicit default gateway. The software
will randomly choose any gateway that is responding. This
implementation is very flexible and recovers well from failures.
However a large network full of such implementations will waste a lot
of bandwidth on the echo datagrams that are used to test whether
gateways are up. This is the reason that this strategy is
discouraged.
5. Bridges and Gateways
This section will deal in more detail with the technology used to
construct larger networks. It will focus particularly on how to
connect together multiple Ethernets, token rings, etc. These days
most networks are hierarchical. Individual hosts attach to local-area
24
networks such as Ethernet or token ring. Then those local networks
are connected via some combination of backbone networks and point to
point links. A university might have a network that looks in part
like this:
________________________________
| net 1 net 2 net 3 | net 4 net 5
| ---------X---------X-------- | -------- --------
| | | | |
| Building A | | | |
| ----------X--------------X-----------------X
| | campus backbone network :
|______________________________| :
serial :
line :
-------X-----
net 6
Nets 1, 2 and 3 are in one building. Nets 4 and 5 are in different
buildings on the same campus. Net 6 is in a somewhat more distant
location. The diagram above shows nets 1, 2, and 3 being connected
directly, with switches that handle the connections being labelled as
"X". Building A is connected to the other buildings on the same
campus by a backbone network. Note that traffic from net 1 to net 5
takes the following path:
- from 1 to 2 via the direct connection between those networks
- from 2 to 3 via another direct connection
- from 3 to the backbone network
- across the backbone network from building A to the building in
which net 5 is housed
- from the backbone network to net 5
Traffic for net 6 would additionally pass over a serial line. With
the setup as shown, the same switch is being used to connect the
backbone network to net 5 and to the serial line. Thus traffic from
net 5 to net 6 would not need to go through the backbone, since there
is a direct connection from net 5 to the serial line.
This section is largely about what goes in those "X"'s.
5.1 Alternative Designs
Note that there are alternatives to the sort of design shown above.
One is to use point to point lines or switched lines directly to each
host. Another is to use a single-level of network technology that is
capable of handling both local and long-haul networking.
25
5.1.1 A mesh of point to point lines
Rather than connecting hosts to a local network such as Ethernet, and
then interconnecting the Ethernets, it is possible to connect
long-haul serial lines directly to the individual computers. If your
network consists primarily of individual computers at distant
locations, this might make sense. Here would be a small design of
that type.
In the design shown earlier, the task of routing datagrams around the
network is handled by special-purpose switching units shown as "X"'s.
If you run lines directly between pairs of hosts, your hosts will be
doing this sort of routing and switching, as well as their normal
computing. Unless you run lines directly between every pair of
computers, some systems will end up handling traffic for others. For
example, in this design, traffic from 1 to 3 will go through 4, 5 and
6. This is certainly possible, since most TCP/IP implementations are
capable of forwarding datagrams. If your network is of this type, you
should think of your hosts as also acting as gateways. Much of the
discussion below on configuring gateways will apply to the routing
software that you run on your hosts. This sort of configuration is
not as common as it used to be, for two reasons:
- Most large networks have more than one computer per location. In
this case it is less expensive to set up a local network at each
location than to run point to point lines to each computer.
- Special-purpose switching units have become less expensive. It
often makes sense to offload the routing and communications tasks
to a switch rather than handling it on the hosts.
It is of course possible to have a network that mixes the two kinds of
techology. In this case, locations with more equipment would be
handled by a hierarchical system, with local-area networks connected
by switches. Remote locations with a single computer would be handled
by point to point lines going directly to those computers. In this
case the routing software used on the remote computers would have to
be compatible with that used by the switches, or there would need to
be a gateway between the two parts of the network.
Design decisions of this type are typically made after an assessment
of the level of network traffic, the complexity of the network, the
quality of routing software available for the hosts, and the ability
of the hosts to handle extra network traffic.
26
5.1.2 Circuit switching technology
Another alternative to the hierarchical LAN/backbone approach is to
use circuit switches connected to each individual computer. This is
really a variant of the point to point line technique, where the
circuit switch allows each system to have what amounts to a direct
line to every other system. This technology is not widely used within
the TCP/IP community, largely because the TCP/IP protocols assume that
the lowest level handles isolated datagrams. When a continuous
connection is needed, higher network layers maintain it using
datagrams. This datagram-oriented technology does not match a
circuit-oriented environment very closely. In order to use circuit
switching technology, the IP software must be modified to be able to
build and tear down virtual circuits as appropriate. When there is a
datagram for a given destination, a virtual circuit must be opened to
it. The virtual circuit would be closed when there has been no
traffic to that destination for some time. The major use of this
technology is for the DDN (Defense Data Network). The primary
interface to the DDN is based on X.25. This network appears to the
outside as a distributed X.25 network. TCP/IP software intended for
use with the DDN must do precisely the virtual circuit management just
described. Similar techniques could be used with other
circuit-switching technologies, e.g. ATT's DataKit, although there is
almost no software currently available to support this.
5.1.3 Single-level networks
In some cases new developments in wide-area networks can eliminate the
need for hierarchical networks. Early hierarchical networks were set
up because the only convenient network technology was Ethernet or
other LAN's, and those could not span distances large enough to cover
an entire campus. Thus it was necessary to use serial lines to
connect LAN's in various locations. It is now possible to find
network technology whose characteristics are similar to Ethernet, but
where a single network can span a campus. Thus it is possible to
think of using a single large network, with no hierarchical structure.
The primary limitations of a large single-level network are
performance and reliability considerations. If a single network is
used for the entire campus, it is very easy to overload it.
Hierarchical networks can handle a larger traffic volume than
single-level networks if traffic patterns have a reasonable amount of
locality. That is, in many applications, traffic within an individual
department tends to be greater than traffic among departments.
Let's look at a concrete example. Suppose there are 10 departments,
each of which generate 1 Mbit/sec of traffic. Suppose futher than 90%
of that traffic is to other systems within the department, and only
10% is to other departments. If each department has its own network,
that network only needs to handle 1 Mbit/sec. The backbone network
connecting the department also only needs 1 Mbit/sec capacity, since
27
it is handling 10% of 1 Mbit from each department. In order to handle
this situation with a single wide-area network, that network would
have to be able to handle the simultaneous load from all 10
departments, which would be 10 Mbit/sec.
The second limitation on single-level networks is reliability,
maintainability and security. Wide-area networks are more difficult
to diagnose and maintain than local-area networks, because problems
can be introduced from any building to which the network is connected.
They also make traffic visible in all locations. For these reasons,
it is often sensible to handle local traffic locally, and use the
wide-area network only for traffic that actually must go between
buildings. However if you have a situation where each location has
only one or two computers, it may not make sense to set up a local
network at each location, and a single-level network may make sense.
5.1.4 Mixed designs
In practice, few large networks have the luxury of adopting a
theoretically pure design.
It is very unlikely that any large network will be able to avoid using
a hierarchical design. Suppose we set out to use a single-level
network. Even if most buildings have only one or two computers, there
will be some location where there are enough that a local-area network
is justified. The result is a mixture of a single-level network and a
hierachical network. Most buildings have their computers connected
directly to the wide-area network, as with a single-level network.
However in one building there is a local-area network which uses the
wide-area network as a backbone, connecting to it via a switching
unit.
On the other side of the story, even network designers with a strong
commitment to hierarchical networks are likely to find some parts of
the network where it simply doesn't make economic sense to install a
local-area network. So a host is put directly onto the backbone
network, or tied directly to a serial line.
However you should think carefully before making ad hoc departures
from your design philosophy in order to save a few dollars. In the
long run, network maintainability is going to depend upon your ability
to make sense of what is going on in the network. The more consistent
your technology is, the more likely you are to be able to maintain the
network.
28
5.2 An introduction to alternative switching technologies
This section will discuss the characteristics of various technologies
used to switch datagrams between networks. In effect, we are trying
to fill in some details about the black boxes assumed in previous
sections. There are three basic types of switches, generally referred
to as repeaters, bridges, and gateways, or alternatively as level 1, 2
and 3 switches (based on the level of the ISO model at which they
operate). Note however that there are systems that combine features
of more than one of these, particularly bridges and gateways.
The most important dimensions on which switches vary are isolation,
performance, routing and network management facilities. These will be
discussed below.
The most serious difference is between repeaters and the other two
types of switch. Until recently, gateways provided very different
services from bridges. However these two technologies are now coming
closer together. Gateways are beginning to adopt the special-purpose
hardware that has characterized bridges in the past. Bridges are
beginning to adopt more sophisticated routing, isolation features, and
network management, which have characterized gateways in the past.
There are also systems that can function as both bridge and gateway.
This means that at the moment, the crucial decision may not be to
decide whether to use a bridge or a gateway, but to decide what
features you want in a switch and how it fits into your overall
network design.
5.2.1 Repeaters
A repeater is a piece of equipment that connects two networks that use
the same technology. It receives every data packet on each network,
and retransmits it onto the other network. The net result is that the
two networks have exactly the same set of packets on them. For
Ethernet or IEEE 802.3 networks there are actually two different kinds
of repeater. (Other network technologies may not need to make this
distinction.)
A simple repeater operates at a very low level indeed. Its primary
purpose is to get around limitations in cable length caused by signal
loss or timing dispersion. It allows you to construct somewhat larger
networks than you would otherwise be able to construct. It can be
thought of as simply a two-way amplifier. It passes on individual
bits in the signal, without doing any processing at the packet level.
It even passes on collisions. That is, if a collision is generated on
one of the networks connected to it, the repeater generates a
collision on the other network. There is a limit to the number of
repeaters that you can use in a network. The basic Ethernet design
requires that signals must be able to get from one end of the network
to the other within a specified amount of time. This determines a
maximum allowable length. Putting repeaters in the path does not get
29
around this limit. (Indeed each repeater adds some delay, so in some
ways a repeater makes things worse.) Thus the Ethernet configuration
rules limit the number of repeaters that can be in any path.
A "buffered repeater" operates at the level of whole data packets.
Rather than passing on signals a bit at a time, it receives an entire
packet from one network into an internal buffer and then retransmits
it onto the other network. It does not pass on collisions. Because
such low-level features as collisions are not repeated, the two
networks continue to be separate as far as the Ethernet specifications
are concerned. Thus there are no restrictions on the number of
buffered repeaters that can be used. Indeed there is no requirement
that both of the networks be of the same type. However the two
networks must be sufficiently similar that they have the same packet
format. Generally this means that buffered repeaters can be used
between two networks of the IEEE 802.x family (assuming that they have
chosen the same address length), or two networks of some other related
family. A pair of buffered repeaters can be used to connect two
networks via a serial line.
Buffered repeaters share with simple repeaters the most basic feature:
they repeat every data packet that they receive from one network onto
the other. Thus the two networks end up with exactly the same set of
packets on them.
5.2.2 Bridges and gateways
A bridge differs from a buffered repeater primarily in the fact that
it exercizes some selectivity as to what packets it forwards between
networks. Generally the goal is to increase the capacity of the
system by keeping local traffic confined to the network on which it
originates. Only traffic intended for the other network (or some
other network accessed through it) goes through the bridge. So far
this description would also apply to a gateway. Bridges and gateways
differ in the way they determine what packets to forward. A bridge
uses only the ISO level 2 address. In the case of Ethernet or IEEE
802.x networks, this is the 6-byte Ethernet or MAC-level address. (The
term MAC-level address is more general. However for the sake of
concreteness, examples in this section will assume that Ethernet is
being used. You may generally replace the term "Ethernet address"
with the equivalent MAC-level address for other similar technologies.)
A bridge does not examine the packet itself, so it does not use the IP
address or its equivalent for routing decisions. In contrast, a
gateway bases its decisions on the IP address, or its equivalent for
other protocols.
There are several reasons why it matters which kind of address is used
for decisions. The most basic is that it affects the relationship
between the switch and the upper layers of the protocol. If
forwarding is done at the level of the MAC-level address (bridge), the
switch will be invisible to the protocols. If it is done at the IP
level, the switch will be visible. Let's give an example. Here are
30
Note that the bridge does not have an IP address. As far as computers
A, B, and C are concerned, there is a single Ethernet (or other
network) to which they are all attached. This means that the routing
tables must be set up so that computers on both networks treat both
networks as local. When computer A opens a connection to computer B,
it first broadcasts an ARP request asking for computer B's Ethernet
address. The bridge must pass this broadcast from network 1 to
network 2. (In general, bridges must pass all broadcasts.) Once the
two computers know each other's Ethernet addresses, communications use
the Ethernet address as the destination. At that point, the bridge
can start exerting some selectivity. It will only pass packets whose
Ethernet destination address is for a machine on the other network.
Thus a packet from B to A will be passed from network 2 to 1, but a
packet from B to C will be ignored.
In order to make this selection, the bridge needs to know which
network each machine is on. Most modern bridges build up a table for
each network, listing the Ethernet addresses of machines known to be
on that network. They do this by watching all of the packets on both
networks. When a packet first appears on network 1, it is reasonable
to conclude that the Ethernet source address corresponds to a machine
on network 1.
Note that a bridge must look at every packet on the Ethernet, for two
different reasons. First, it may use the source address to learn
which machines are on which network. Second, it must look at the
destination address in order to decide whether it needs to forward the
packet to the other network.
As mentioned above, generally bridges must pass broadcasts from one
network to the other. Broadcasts are often used to locate a resource.
The ARP request is a typical example of this. Since the bridge has no
way of knowing what host is going to answer the broadcast, it must
pass it on to the other network. Some newer bridges have
user-selectable filters. With them, it is possible to block some
broadcasts and allow others. You might allow ARP broadcasts (which
are essential for IP to function), but confine less essential
broadcasts to one network. For example, you might choose not to pass
rwhod broadcasts, which some systems use to keep track of every user
logged into every other system. You might decide that it is
sufficient for rwhod to know about the systems on a single segment of
the network.
31
Now let's take a look at two networks connected by a gateway
Note that the gateway has IP addresses assigned to each interface.
The computers' routing tables are set up to forward through
appropriate address. For example, computer A has a routing entry
saying that it should use the gateway 128.6.5.1 to get to subnet
128.6.4.
Because the computers know about the gateway, the gateway does not
need to scan all the packets on the Ethernet. The computers will send
packets to it when appropriate. For example, suppose computer A needs
to send a message to computer B. Its routing table will tell it to use
gateway 128.6.5.1. It will issue an ARP request for that address.
The gateway will respond to the ARP request, just as any host would.
From then on, packets destinated for B will be sent with the gateway's
Ethernet address.
5.2.3 More about bridges
There are several advantages to using the Mac-level address, as a
bridge does. First, every packet on an Ethernet or IEEE network has
such an address. The address is in the same place for every packet,
whether it is IP, DECnet, or some other protocol. Thus it is
relatively fast to get the address from the packet. A gateway must
decode the entire IP header, and if it is to support protocols other
than IP, it must have software for each such protocol. This means
that a bridge automatically supports every possible protocol, whereas
a gateway requires specific provisions for each protocol it is to
support.
However there are also disadvantages. The one that is intrinsic to
the design of a bridge is
- A bridge must look at every packet on the network, not just those
addressed to it. Thus it is possible to overload a bridge by
putting it on a very busy network, even if very little traffic is
actually going through the bridge.
However there are another set of disadvantages that are based on the
way bridges are usually built. It is possible in principle to design
bridges that do not have these disadvantages, but I don't know of any
plans to do so. They all stem from the fact that bridges do not have
32
a complete routing table that describes the entire system of networks.
They simply have a list of the Ethernet addresses that lie on each of
its two networks. This means
- A bridge can handle only two network interfaces. At a central
site, where many networks converge, this normally means that you
set up a backbone network to which all the bridges connect, and
then buy a separate bridge to connect each other network to that
backbone. Gateways often have between 4 and 8 interfaces.
- Networks that use bridges cannot have loops in them. If there
were a loop, some bridges would see traffic from the same
Ethernet address coming from both directions, and would be unable
to decide which table to put that address in. Note that any
parallel paths to the same direction constitute a loop. This
means that multiple paths cannot be used for purposes of
splitting the load or providing redundancy.
There are some ways of getting around the problem of loops. Many
bridges allow configurations with redundant connections, but turn off
links until there are no loops left. Should a link fail, one of the
disabled ones is then brought back into service. Thus redundant links
can still buy you extra reliability. But they can't be used to
provide extra capacity. It is also possible to build a bridge that
will make use of parallel point to point lines, in the one special
case where those lines go between a single pair of bridges. The
bridges would treat the two lines as a single virtual line, and use
them alternately in round-robin fashion.
The process of disabling redundant connections until there are no
loops left is called a "spanning tree algorithm". This name comes
from the fact that a tree is defined as a pattern of connections with
no loops. Thus one wants to disable connections until the connections
that are left form a tree that "spans" (includes) all of the networks
in the system. In order to do this, all of the bridges in a network
system must communicate among themselves. There is an IEEE proposal
to standardize the protocol for doing this, and for constructing the
spanning tree.
Note that there is a tendency for the resulting spanning tree to
result in high network loads on certain parts of the system. The
networks near the "top of the tree" handle all traffic between distant
parts of the network. In a network that uses gateways, it would be
possible to put in an extra link between parts of the network that
have heavy traffic between them. However such extra links cannot be
used by a set of bridges.
33
5.2.4 More about gateways
Gateways have their own advantages and disadvantages. In general a
gateway is more complex to design and to administer than a bridge. A
gateway must participate in all of the protocols that it is designed
to forward. For example, an IP gateway must respond to ARP requests.
The IP standards also require it to completely process the IP header,
decrementing the time to live field and obeying any IP options.
Gateways are designed to handle more complex network topologies than
bridges. As such, they have a different (and more complex) set of
decisions to make. In general a bridge has only a binary decision to
make: does it or does it not pass a given packet from one network to
the other? However a gateway may have several network interfaces.
Furthermore, when it forwards a packet, it must decide what host or
gateway to send the packet to next. It is even possible for a gateway
to decide to send a packet back onto the same network it came from.
If a host is using the gateway as its default, it may send packets
that really should go to some other gateway. In that case, the
gateway will send the packet on to the right gateway, and send back an
ICMP redirect to the host. Many gateways can also handle parallel
paths. If there are several equally good paths to a destination, the
gateway will alternate among them in round-robin fashion.
In order to handle these decisions, a gateway will typically have a
routing table that looks very much like a host's. As with host
routing tables, a gateway's table contains an entry for each possible
network number. For each network, there is either an entry saying
that that network is connected directly to the gateway, or there is an
entry saying that traffic for that network should be forwarded through
some other gateway or gateways. We will describe the "routing
protocols" used to build up this information later, in the discussion
on how to configure a gateway.
5.3 Comparing the switching technologies
Repeaters, buffered repeaters, bridges, and gateways form a spectrum.
Those devices near the beginning of the list are best for smaller
networks. They are less expensive, and easier to set up, but less
general. Those near the end of the list are suitable for building
more complex networks. Many networks will contain a mixture of switch
types, with repeaters being used to connect a few nearby network
segments, bridges used for slightly larger areas (particularly those
with low traffic levels), and gateways used for long-distance links.
Note that this document so far has assumed that only gateways are
being used. The section on setting up a host described how to set up
a routing table listing the gateways to use to get to various
networks. Repeaters and bridges are invisible to IP. So as far as
previous sections are concerned, networks connected by them are to be
considered a single network. Section 3.3.1 describes how to configure
34
a host in the case where several subnets are carried on a single
physical network. The same configuration should be used when several
subnets are connected by repeaters or bridges.
As mentioned above, the most important dimensions on which switches
vary are isolation, performance, routing, network management, and
performing auxilliary support services.
5.3.1 Isolation
Generally people use switches to connect networks to each other. So
they are normally thinking of gaining connectivity, not providing
isolation. However isolation is worth thinking about. If you connect
two networks and provide no isolation at all, then any network
problems on other networks suddenly appear on yours as well. Also,
the two networks together may have enough traffic to overwhelm your
network. Thus it is well to think of choosing an appropriate level of
protection.
Isolation comes in two kinds: isolation against malfunctions and
traffic isolation. In order to discuss isolation of malfunctions, we
have to have a taxonomy of malfunctions. Here are the major classes
of malfunctions, and which switches can isolate them:
- Electrical faults, e.g. a short in the cable or some sort of
fault that distorts the signal. All types of switch will confine
this to one side of the switch: repeater, buffered repeater,
bridge, gateway. These are worth protecting against, although
their frequency depends upon how often your cables are changed or
disturbed. It is rare for this sort of fault to occur without
some disturbance of the cable.
- Transceiver and controller problems that general signals that are
valid electrically but nevertheless incorrect (e.g. a continuous,
infinitely long packet, spurious collisions, never dropping
carrier). All except the simple repeater will confine this:
buffered repeater, bridge, gateway. (Such problems are not very
common.)
- Software malfunctions that lead to excessive traffic between
particular hosts (i.e. not broadcasts). Bridges and gateways
will isolate these. (This type of failure is fairly rare. Most
software and protocol problems generate broadcasts.)
- Software malfunctions that lead to excessive broadcast traffic.
Gateways will isolate these. Generally bridges will not, because
they must pass broadcasts. Bridges with user-settable filtering
can protect against some broadcast malfunctions. However in
general bridges must pass ARP, and most broadcast malfunctions
involve ARP. This problem is not severe on single-vendor
networks where software is under careful control. However
research sites generally see problems of this sort regularly.
35
Traffic isolation is provided by bridges and gateways. The most basic
decision is how many computers can be put onto a network without
overloading its capacity. This requires knowledge of the capacity of
the network, but also how the hosts will use it. For example, an
Ethernet may support hundreds of systems if all the network is used
for is remote logins and an occasional file transfer. However if the
computers are diskless, and use the network for swapping, an Ethernet
will support between 10 and 40, depending upon their speeds and I/O
rates.
When you have to put more computers onto a network than it can handle,
you split it into several networks and put some sort of switch between
them. If you do the split correctly, most of the traffic will be
between machines on the same piece. This means putting clients on the
same network as their servers, putting terminal servers on the same
network as the hosts that they access most commonly, etc.
Bridges and gateways generally provide similar degrees of traffic
isolation. In both cases, only traffic bound for hosts on the other
side of the switch is passed. However see the discussion on routing.
5.3.2 Performance
This is becoming less of an issue as time goes on, since the
technology is improving. Generally repeaters can handle the full
bandwidth of the network. (By their very nature, a simple repeater
must be able to do so.) Bridges and gateways often have performance
limitations of various sorts. Bridges have two numbers of interest:
packet scanning rate and throughput. As explained above, a bridge
must look at every packet on the network, even ones that it does not
forward. The number of packets per second that it can scan in this
way is the packet scanning rate. Throughput applies to both bridges
and gateways. This is the rate at which they can forward traffic.
Generally this depends upon packet size. Normally the number of
packets per second that a unit can handle will be greater for short
packets than long ones. Early models of bridge varied from a few
hundred packets per second to around 7000. The higher speeds are for
equipment that uses special-purpose hardware to speed up the process
of scanning packets. First-generation gateways varied from a few
hundred packets per second to 1000 or more. However second-generation
gateways are now available, using special-purpose hardware of the same
sophistication as that used by bridges. They can handle on the order
of 10000 packets per second. Thus at the moment high-performance
bridges and gateways can switch most of the bandwidth of an Ethernet.
This means that performance should no longer be a basis for choosing
between types of switch. However within a given type of switch, there
are still specific models with higher or lower capacity.
Unfortunately there is no single number on which you can base
performance estimates. The figure most commonly quoted is packets per
second. Be aware that most vendors count a packet only once as it
goes through a gateway, but that one prominent vendor counts packets
36
twice. Thus their switching rates must be deflated by a factor of 2.
Also, when comparing numbers make sure that they are for packets of
the same size. A simple performance model is
processing time = switching time + packet size * time per byte
That is, the time to switch a packet is normally a constant switching
time, representing interrupt latency, header processing, routing table
lookup, etc., plus a component proportional to packet size,
representing the time needed to do any packet copying. One reasonable
approach to reporting performance is to give packets per second for
minimum and maximum size packets. Another is to report limiting
switching speed in packets per second and throughput in bytes per
second, i.e. the two terms of the equation above.
5.3.3 Routing
Routing refers to the technology used to decide where to send a packet
next. Of course for a repeater this is not an issue, since repeaters
forward every packet.
Bridges are almost always constructed with exactly two interfaces.
Thus routing turns into two decisions: (1) whether the bridge should
function at all, and (2) whether it should forward any particular
packet. The second decision is usually based on a table of MAC-level
addresses. As described above, this is built up by scanning traffic
on both sides of the bridge. The goal is to forward those packets
whose destination is on the other side of the bridge. This algorithm
requires that the network configuration have no loops or redundant
lines. Less sophisticated bridges leave this up to the system
designer. With these bridges, you must set up your network so that
there are no loops in it. More sophisticated bridges allow arbitrary
topology, but disable links until no loops remain. This provides
extra reliability. If a link fails, an alternative link will be
turned on automatically. Bridges that work this way have protocol
that allows them to detect when a unit must be disabled or reenabled,
so that at any instant the set of active links forms a "spanning
tree". If you require the extra reliability of redundant links, make
sure that the bridges you use can disable and enable themselves in
this way. There is currently no official standard for the protocol
used among bridges, although there is a standard in the proposal
stage. If you buy bridges from more than one vendor, make sure that
their spanning-tree protocols will interoperate.
Gateways generally allow arbitrary network topologies, including loops
and redundant links. Because gateways may have more than two
interfaces, they must decide not only when to forward a packet, but
where to send it next. They do this by maintaining a model of the
entire network topology. Different routing techniques maintain models
of greater or lesser complexity, and use the data with varying degrees
of sophistication. Gateways that handle TCP/IP should generally
support the two Internet standard routing protocols: RIP (Routing
37
Information Protocol) and EGP (External Gateway Protocol). EGP is a
special-purpose protocol for use in networks where there is a backbone
under a separate administration. It allows exchange of reachability
information with the backbone in a controlled way. If you are a
member of such a network, your gateway must support EGP. This is
becoming common enough that it is probably a good idea to make sure
that all gateways support EGP.
RIP is a protocol designed to handle routing within small to moderate
size networks, where line speeds do not differ radically. Its primary
limitations are:
- It cannot be used with networks where any path goes through more
than 15 gateways. This range may be further reduced if you use
an optional feature for giving a slow line a weight larger than
one.
- It cannot share traffic between parallel lines (although some
implementations allow this if the lines are between the same pair
of gateways).
- It cannot adapt to changes in network load.
- It is not well suited to situations where there are alternative
routes through lines of very different speeds.
- It may not be stable in networks where lines or gateways change a
lot.
Some vendors supply proprietary modifications to RIP that improve its
operation with EGP or increase the maximum path length beyond 15, but
do not otherwise modify it very much. If you expect your network to
involve gateways from more than one vendor, you should generally
require that all of them support RIP, since this is the only routing
protocol that is generally available. If you expect to use a more
sophisticated protocol in addition, the gateways must have some
ability to translate between their own protocol and RIP. However for
very large or complex networks, there may be no choice but to use some
other protocol throughout.
More sophisticated routing protocols are possible. The primary ones
being considered today are cisco System's IGRP, and protocols based on
the SPF (shortest-path first) algorithms. In general these protocols
are designed for larger or more complex networks. They are in general
stable under a wider variety of conditions, and they can handle
arbitrary combinations of line type and speed. Some of them allow you
to split traffic among parallel paths, to get better overall
throughput. Some newer technologies may allow the network to adjust
to take into account paths that are overloaded. However at the moment
I do not know of any commercial gateway that does this. (There are
very serious problems with maintaining stable routing when this is
done.) There are enough variations among routing technology, and it is
changing rapidly enough, that you should discuss your proposed network
topology in detail with all of the vendors that you are considering.
Make sure that their technology can handle your topology, and can
38
support any special requirements that you have for sharing traffic
among parallel lines, and for adjusting topology to take into account
failures. In the long run, we expect one or more of these newer
routing protocols to attain the status of a standard, at least on a de
facto basis. However at the moment, there is no generally implemented
routing technology other than RIP.
One additional routing topic to consider is policy-based routing. In
general routing protocols are designed to find the shortest or fastest
possible path for every packet. In some cases, this is not desired.
For reasons of security, cost accountability, etc., you may wish to
limit certain paths to certain uses. Most gateways now have some
ability to control the spread of routing information so as to give you
some administrative control over the way routes are used. Different
gateways vary in the degree of control that they support. Make sure
that you discuss any requirements that you have for control with all
prospective gateway vendors.
5.3.4 Network management
Network management covers a wide variety of topics. In general it
includes gathering statistical data and status information about parts
of your network, and taking action as necessary to deal with failures
and other changes. There are several things that a switch can do to
make this process easier. The most basic is that it should have a way
of gathering and reporting statistics. These should include various
sorts of packet counts, as well as counts of errors of various kinds.
This data is likely to be most detailed in a gateway, since the
gateway classifies packets using the protocols, and may even respond
to certain types of packet itself. However bridges and even buffered
repeaters can certainly have counts of packets forwarded, interface
errors, etc. It should be possible to collect this data from a
central monitoring point.
There is now an official Internet approach to network monitoring. The
first stages use a related set of protocols, SGMP and SNMP. Both of
these protocols are designed to allow you to collect information and
to make changes in configuration parameters for gateways and other
entities on your network. You can run the matching interface programs
on any host in your network. SGMP is now available for several
commercial gateways, as well as for Unix systems that are acting as
gateways. There is a limited set of information which any SGMP
implementation is required to supply, as well as a uniform mechanism
for vendors to add information of their own. By late 1988, the second
generation of this protocol, SNMP, should be in service. This is a
slightly more sophisticated protocol. It has with it a more complete
set of information that can be monitored, called the MIB (Management
Information Base). Unlike the somewhat ad hoc collection of SGMP
variables, the MIB is the result of numerous committee deliberations
involving a number of vendors and users. Eventually it is expected
that there will be a TCP/IP equivalent of CMIS, the ISO network
monitoring service. However CMIS, and its protocols, CMIP, are not
39
yet official ISO standards, so they are still in the experimental
stages.
In general terms all of these protocols accomplish the same thing:
They allow you to collect criticial information in a uniform way from
all vendors' equipment. You send commands as UDP datagrams from a
network management program running on some host in your network.
Generally the interaction is fairly simple, with a single pair of
datagrams exchanged: a command and a response. At the moment security
is fairly simple. It is possible to require what amounts to a
password in the command. (In SGMP it is referred to as a "session
name", rather than a password.) More elaborate, encryption-based
security is being developed.
You will probably want to configure the network management tools at
your disposal to do several different things. For short-term network
monitoring, you will want to keep track of switches crashing or being
taken down for maintenance, and of failure of communications lines and
other hardware. It is possible to configurate SGMP and SNMP to issue
"traps" (unsolited messages) to a specified host or list of hosts when
some of these critical events occur (e.g. lines up and down). However
it is unrealistic to expect a switch to notify you when it crashes.
It is also possible for trap messages to be lost due to network
failure or overload. Thus you should also poll your switches
regularly to gather information. Various displays are available,
including a map of your network where items change color as their
status changes, and running "strip charts" that show packet rates and
other items through selected switches. This software is still in its
early stages, so you should expect to see a lot of change here.
However at the very least you should expect to be notified in some way
of failures. You may also want to be able to take actions to
reconfigure the system in response to failures, although security
issues make some mangers nervous about doing that through the existing
management protocols.
The second type of monitoring you are likely to want to do is to
collect information for use in periodic reports on network utilization
and performance. For this, you need to sample each switch
perodically, and retrieve numbers of interest. At Rutgers we sample
hourly, and get the number of packets forwarded for IP and DECnet, a
count of reloads, and various error counts. These are reported daily
in some detail. Monthly summaries are produced giving traffic through
each gateway, and a few key error rates chosen to indicate a gateway
that is being overloaded (packets dropped in input and output).
It should be possible to use monitoring techniques of this kind with
most types of switch. At the moment, simple repeaters do not report
any statistics. Since they do not generally have processors in them,
doing so would cause a major increase in their cost. However it
should be possible to do network management for buffered repeaters,
bridges, and gateways. Gateways are the most likely to contain
sophisticated network management software. Most gateway vendors that
handle TCP/IP are expected to implement the monitoring protocols
described above. Many bridge vendors make some provisions for
collecting performance data. Since bridges are not protocol-specific,
40
most of them do not have the software necessary to implement
TCP/IP-based network management protocols. In some cases, monitoring
can be done only by typing commands to a directly-attached console.
(We have seen one case where it is necessary to take the bridge out of
service to gather this data.) In other cases, it is possible to gather
data via the network, but the monitoring protocol is ad hoc or even
proprietary.
Except for very small networks, you should probably insist that all of
the devices on your network collect statistics and provide some way of
querying them remotely. In the long run, you can expect the most
software to be available for standard protocols such as SGMP/SNMP and
CMIS. However proprietary monitoring tools may be sufficient as long
as they work with all of the equipment that you have.
5.3.5 A final evaluation
Here is a summary of the places where each kind of switch technology
is normally used:
- Repeaters are normally confined to a single building. Since they
provide no traffic isolation, you must make sure that the entire
set of networks connected by repeaters can carry the traffic from
all of the computers on it. Since they generally provide no
network monitoring tools, you will not want to use repeaters for
a link that is likely to fail.
- Bridges and gateways should be placed sufficiently frequently to
break your network into pieces for which the traffic volume is
manageable. You may want to place bridges or gateways in places
where traffic would not require them for network monitoring
reasons.
- Because bridges must pass broadcast packets, there is a limit to
the size network you can construct using them. It is probably a
good idea to limit the network connected by bridges to a hundred
systems or so. This number can be increased somewhat for bridges
with good facilities for filtering.
- Because certain kinds of network misbehavior will be passed,
bridges should be used only among portions of the network where a
single group is responsible for diagnosing problems. You have to
be crazy to use a bridge between networks owned by different
organizations. Portions of your network where experiments are
being done in network technology should always be isolated from
the rest of the network by gateways.
- For many applications it is more important to choose a product
with the right combination of performance, network management
tools, and other features than to make the decision between
bridges and gateways.
41
@section(Configuring Gateways)
This section deals with configuration issues that are specific to
gateways. Gateways than handle TCP/IP are themselves Internet hosts.
Thus the discussions above on configuring addresses and routing
information apply to gateways as well as to hosts. The exact way you
configure a gateway will depend upon the vendor. In some cases, you
edit files stored on a disk in the gateway itself. However for
reliability reasons most gateways do not have disks of their own. For
them, configuration information is stored in non-volatile memory or in
configuration files that are uploaded from one or more hosts on the
network.
At a minimum, configuration involves specifying the Internet address
and address mask for each interface, and enabling an appropriate
routing protocol. However generally a few other options are
desirable. There are often parameters in addition to the Internet
address that you should set for each interface.
One important parameter is the broadcast address. As explained above,
older software may react badly when broadcasts are sent using the new
standard broadcast address. For this reason, some vendors allow you
to choose a broadcast address to be used on each interface. It should
be set using your knowledge of what computers are on each of the
networks. In general if the computers follow current standards, a
broadcast address of 255.255.255.255 should be used. However older
implementations may behave better with other addresses, particularly
the address that uses zeros for the host number. (For the network
128.6 this would be 128.6.0.0. For compatibility with software that
does not implement subnets, you would use 128.6.0.0 as the broadcast
address even for a subnet such as 128.6.4.) You should watch your
network with a network monitor and see the results of several
different broadcast address choices. If you make a bad choice, every
time the gateway sends a routing update broadcast, many machines on
your network will respond with ARP's or ICMP errors. Note that when
you change the broadcast address in the gateway, you may need to
change it on the individual computers as well. Generally the idea is
to change the address on the systems that you can configure to give
behavior that is compatible with systems that you can't configure.
Other interface parameters may be necessary to deal with peculiarities
of the network it is connected to. For example, many gateways test
Ethernet interfaces to make sure that the cable is connected and the
transceiver is working correctly. Some of these tests will not work
properly with the older Ethernet version 1 transceivers. If you are
using such a transceiver, you would have to disable this keepalive
testing. Similarly, gateways connected by a serial line normally do
regular testing to make sure that the line is still working. There
can be situations where this needs to be disabled.
Often you will have to enable features of the software that you want
to use. For example, it is often necessary to turn on the network
management protocol explicitly, and to give it the name or address of
a host that is running software to accept traps (error messages).
42
Most gateways have options that relate to security. At a minimum,
this may include setting password for making changes remotely (and the
"session name" for SGMP). If you need to control access to certain
parts of your network, you will also need to define access control
lists or whatever other mechanism your gateway uses.
Gateways that load configuration information over the network present
special issues. When such a gateway boots, it sends broadcast
requests of various kinds, attempting to find its Internet address and
then to load configuration information. Thus it is necessary to make
sure that there is some computer that is prepared to respond to these
requests. In some cases, this is a dedicated micro running special
software. In other cases, generic software is available that can run
on a variety of machines. You should consult your vendor to make sure
that this can be arranged. For reliability reasons, you should make
sure that there is more than one host with the information and
programs that your gateways need. In some cases you will have to
maintain several different files. For example, the gateways used at
Rutgers use a program called "bootp" to supply their Internet address,
and they then load the code and configuration information using TFTP.
This means that we have to maintain a file for bootp that contains
Ethernet and Internet addresses for each gateway, and a set of files
containing other configuration information for each gateway. If your
network is large, it is worth taking some trouble to make sure that
this information remains consistent. We keep master copies of all of
the configuration information on a single computer, and distribute it
to other systems when it changes, using the Unix utilities make and
rdist. If your gateway has an option to store configuration
information in non-volatile memory, you will eliminate some of these
logistical headaches. However this presents its own problems. The
contents of non-volatile memory should be backed up in some central
location. It will also be harder for network management personnel to
review configuration information if it is distributed among the
gateways.
Starting a gateway is particularly challenging if it loads
configuration information from a distant portion of the network.
Gateways that expect to take configuration information from the
network generally issue broadcast requests on all of the networks to
which they are connected. If there is a computer on one of those
networks that is prepared to respond to the request, things are
straightforward. However some gateways may be in remote locations
where there are no nearby computer systems that can support the
necessary protocols. In this case, it is necessary to arrange for the
requests to be routed back to network where there are appropriate
computers. This requires what is strictly speaking a violation of the
basic design philosophy for gateways. Generally a gateway should not
allow broadcasts from one network to pass through to an adjacent
network. In order to allow a gateway to get information from a
computer on a different network, at least one of the gateways in
between will have to be configured to pass the particular class of
broadcasts used to retrieve this information. If you have this sort
of configuration, you should test the loading process regularly. It
is not unusual to find that gateways do not come up after a power
failure because someone changed the configuration of another gateway
43
and made it impossible to load some necessary information.
5.4 Configuring routing for gateways
The final topic to be considered is configuring routing. This is more
complex for a gateway than for a normal host. Most Internet experts
recommend that routing be left to the gateways. Thus hosts may simply
have a default route that points to the nearest gateway. Of course
the gateways themselves can't get by with this. They need to have
complete routing tables.
In order to understand how to configure a gateway, we have to look in
a bit more detail at how gateways communicate routes. When you first
turn on a gateway, the only networks it knows about are the ones that
are directly connected to it. (They are specified by the
configuration information.) In order to find out how to get to more
distant parts of the network, it engages in some sort of "routing
protocol". A routing protocol is simply a protocol that allows each
gateway to advertise which networks it can get to, and to spread that
information from one gateway to the next. Eventually every gateway
should know how to get to every network. There are different styles
of routing protocol. In one common type, gateways talk only to nearby
gateways. In another type, every gateway builds up a database
describing every other gateway in the system. However all of the
protocols have some way for each gateway in the system to find out how
to get to every destination.
A metric is some number or set of numbers that can be used to compare
routes. The routing table is constructed by gathering information
from other gateways. If two other gateways claim to be able to get to
the same destination, there must be some way of deciding which one to
use. The metric is used to make that decision. Metrics all indicate
in some general sense the "cost" of a route. This may be a cost in
dollars of sending packets over that route, the delay in milliseconds,
or some other measure. The simplest metric is just a count of the
number of gateways along the path. This is referred to as a "hop
count". Generally this metric information is set in the gateway
configuration files, or is derived from information appearing there.
At a minimum, routing configuration is likely to consist of a command
to enable the routing protocol that you want to use. Most vendors
will have a prefered routing protocol. Unless you have some reason to
choose another, you should use that. The normal reason for choosing
another protocol is for compatibility with other kinds of gateway.
For example, your network may be connected to a national backbone
network that requires you to use EGP (exterior gateway protocol) to
communicate routes with it. EGP is only appropriate for that specific
case. You should not use EGP within your own network, but you may
need to use it in addition to your regular routing protocol to
communicate with a national network. If your own network has several
different types of gateway, then you may need to pick a routing
protocol that all of them support. At the moment, this is likely to
44
be RIP (Routing Information Protocol). Depending upon the complexity
of your network, you could use RIP throughout it, or use a more
sophisticated protocol among the gateways that support it, and use RIP
only at the boundary between gateways from different vendors.
Assuming that you have chosen a routing protocol and turned it on,
there are some additional decisions that you may need to make. One of
the more basic configuration options has to do with supplying metric
information. As indicated above, metrics are numbers which are used
to decide which route is the best. Unsophisticated routing protocols,
e.g. RIP, normally just count hops. So a route that passes through 2
gateways would be considered better than one that passes through 3.
Of course if the latter route used 1.5Mbps lines and the former 9600
bps lines, this would be the wrong decision. Thus most routing
protocols allow you to set parameters to take this sort of thing into
account. With RIP, you would arrange to treat the 9600 bps line as if
it were several hops. You would increase the effective hop count
until the better route was chosen. More sophisticated protocols may
take the bit rate of the line into account automatically. However you
should be on the lookout for configuration parameters that need to be
set. Generally these parameters will be associated with the
particular interface. For example, with RIP you would have to set a
metric value for the interface connected to the 9600 bps line. With
protocols that are based on bit rate, you might need to specify the
speed of each line (if the gateway cannot figure it out
automatically).
Most routing protocols are designed to let each gateway learn the
topology of the entire network, and to choose the best possible route
for each packet. In some cases you may not want to use the "best"
route. You may want traffic to stay out of a certain portion of the
network for security or cost reasons. One way to institute such
controls is by specifying routing options. These options are likely
to be different for different vendors. But the basic strategy is that
if the rest of the network doesn't know about a route, it won't be
used. So controls normally take the form of limiting the spread of
information about routes whose use you want to control.
Note that there are ways for the user to override the routing
decisions made by your gateways. If you really need to control access
to a certain network, you will have to do two separate things: Use
routing controls to make sure that the gateways use only the routes
you want them to. But also use access control lists on the gateways
that are adjacent to the sensitive networks. These two mechanisms act
at different levels. The routing controls affect what happens to most
packets: those where the user has not specified routing manually.
Your routing mechanism must be set up to choose an acceptable route
for them. The access control list provides an additional limitation
which prevents users from supplying their own routing and bypassing
your controls.
For reliability and security reasons, there may also be controls to
allow you to list the gateways from which you will accept information.
It may also be possible to rank gateways by priority. For example,
you might decide to listen to routes from within your own organization
45
before routes from other organizations or other parts of the
organization. This would have the effect of having traffic use
internal routes in preference to external ones, even if the external
ones appear to be better.
If you use several different routing protocols, you will probably have
some decisions to make regarding how much information to pass among
them. Since multiple routing protocols are often associated with
multiple organizations, you must be sure to make these decisions in
consultation with management of all of the relevant networks.
Decisions that you make may have consequences for the other network
which are not immediately obvious. You might think it would be best
to configure the gateway so that everything it knows is passed on by
all routing protocols. However here are some reasons why you may not
want to do so:
- The metrics used by different routing protocols may not be
comparable. If you are connected to two different external
networks, you want to specify that one should always be used in
preference to the other, or that the nearest one should be used,
rather than attempting to compare metric information received
from the two networks to see which has the better route.
- EGP is particularly sensitive, because the EGP protocol cannot
handle loops. Thus there are strict rules governing what
information may be communicated to a backbone that uses EGP. In
situations where EGP is being used, management of the backbone
network should help you configure your routing.
- If you have slow lines in your network (9600 bps or slower), you
may prefer not to send a complete routing table throughout the
network. If you are connected to an external network, you may
prefer to treat it as a default route, rather than to inject all
of its routing information into your routing protocol.
