FTS-0004 EchoMail Specification
This document is directly derived from the documentation of
-------------------------------------------------------------------------------
The Conference Mail System
By
Bob Hartman
Sysop of FidoNet(tm) node 132/101
(C) Copyright 1986,87, Spark Software, Inc.
427-3 Amherst Street
CS 2032, Suite 232
Nashua, N.H. 03061
ALL RIGHTS RESERVED.
-------------------------------------------------------------------------------
version 3.31 of 12 December, 1987.
With Bob Hartman's kind consent, copying for the purpose of technological
research and advancement is allowed.
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
WHAT IS THE CONFERENCE MAIL SYSTEM?
Conference Mail is a technique to permit several nodes on a
network to share a message base, similar in concept to the
conferences available on many of the computer services, but it is
most closely related to the Usenet system consisting of more than
8,000 systems world wide. All systems sharing a given conference
see any messages entered into the conference by any of the
participating systems. This can be implemented in such a way as
to be totally transparent to the users of a particular node. In
fact, they may not even be aware of the network being used to
move their messages about from node to node! Unfortunately, this
has its disadvantages also - most users who are not educated
about Conference Mail do not realize the messages transmitted
cost MANY sysops (system operators) money, not just the local
sysop. This is an important consideration in Conference Mail and
should not be taken lightly. In a conference with 100 systems as
participants the cost per message can get quite high.
The Conference Mail System is designed to operate in conjunction
with a FidoNet compatible mail server. The currently supported
mail servers are Fido(tm), SEAdog(tm), Opus, and Dutchie. Since
the mail server is a prerequisite to using the Conference Mail
System, it will be assumed you already have your mail server
operating correctly on your system, and you are connected into
FidoNet or a compatible network.
HISTORY OF THE CONFERENCE MAIL SYSTEM
In late 1985, Jeff Rush, a Fido sysop in Dallas, wanted a
convenient means of sharing ideas with the other Dallas sysops.
He created a system of programs he called Echomail, and the
Dallas sysops' Conference was born.
Within a short time sysops in other areas began hearing of this
marvelous new gadget and Echomail took on a life of its own.
Today, a scant year and a half later, the FidoNet public network
boasts a myriad of conferences varying in size from the dozen-or-
so participants in the FidoNet Technical Standards Committee
Conference to the Sysops' Conference with several hundred
participants. It is not uncommon for a node to carry 30 or more
conferences and share those conferences with 10 or more nodes.
HOW IT WORKS
The Conference Mail System is functionally compatible with the
original Echomail utilities. In general, the process is:
1. A message is entered into a designated area on a FidoNet
compatible system.
2. This message is "Exported" along with some control information
to each system "linked" to the conference through the originating
system.
3. Each of the receiving systems "Import" the message into the
proper Conference Mail area.
4. The receiving systems then "Export" these messages, along with
additional control information, to each of their conference
links.
5. Return to step 3.
As you can see, the method is quite simple - in general. Of
course, following the steps literally would mean messages would
never stop being Exported and transmitted to other systems. This
obviously would not be desired or the network would quickly
become overburdened. The information contained in the 'control
information' section is used to prevent transmitting the same
message more than once to a single system.
CONFERENCE MAIL MESSAGE CONTROL INFORMATION
There are five pieces of control information associated with a
Conference Mail message. Some are optional, some are not.
Normally this information is never entered by the person creating
the message. The following control fields determine how
Conference Mail is handled:
1. Area line
This is the first line of a conference mail message. Its
actual appearance is:
AREA:CONFERENCE
Where CONFERENCE is the name of the conference. This line is
added when a conference is being "Exported" to another
system. It is based upon information found in the AREAS.BBS
(configuration) File for the designated message area. This
field is REQUIRED by the receiving system to "Import" a
message into the correct Conference Mail area.
2. Tear Line
This line is near the end of a message and consists of three
dashes (---) followed by an optional program specifier.
This is used to show the first program used to add Echomail
compatible control information to the message. The tear line
generated by Conference Mail looks like:
--- <a small product-specific banner>
This field is optional for most Echomail compatible
processors, and is added by the Conference Mail System to
ensure complete compatibility. Some systems will place this
line in the message when it is first created, but it is
normally added when the message is first "exported."
3. Origin line
This line appears near the bottom of a message and gives a
small amount of information about the system where it
originated. It looks like:
* Origin: The Conference Mail BBS (1:132/101)
The " * Origin: " part of the line is a constant field.
This is followed by the name of the system as taken from the
AREAS.BBS file or a file named ORIGIN located in the DOS
directory of the designated message area. The complete
network address (1:132/101 in this case) is added by the
program inserting the line. This field is generated at the
same time as the tear line, and therefore may either be
generated at the time of creation or during the first
"export" processing. Although the Origin line is not
required by all Echomail processors, it is added by the
Conference Mail System to ensure complete compatibility.
4. Seen-by Lines
There can be many seen-by lines at the end of Conference
Mail messages, and they are the real "meat" of the control
information. They are used to determine the systems to
receive the exported messages. The format of the line is:
SEEN-BY: 132/101 113 136/601 1014/1
The net/node numbers correspond to the net/node numbers of
the systems having already received the message. In this way
a message is never sent to a system twice. In a conference
with many participants the number of seen-by lines can be
very large. This line is added if it is not already a part
of the message, or added to if it already exists, each time
a message is exported to other systems. This is a REQUIRED
field, and Conference Mail will not function correctly if
this field is not put in place by other Echomail compatible
programs.
5. PATH Lines
These are the last lines in a Conference Mail message and
are a new addition, and therefore is not supported by all
Echomail processors. It appears as follows:
^aPATH: 132/101 1014/1
Where the ^a stands for Control-A (ASCII character 1) and
the net/nodes listed correspond to those systems having
processed the message before it reached the current system.
This is not the same as the seen-by lines, because those
lines list all systems the message has been sent to, while
the path line contains all systems having actually processed
the message. This is not a required field, and few echomail
processors currently support it, however it can be used
safely with any other system, since the line(s) will be
ignored. For a discussion on how the path line can be
helpful, see the "Advanced Features" section of this manual.
METHODS OF SENDING CONFERENCE MAIL
To this point the issue of how Conference Mail is actually sent
has been glossed over entirely. The phrase has been, "the message
is exported to another system." What exactly does this mean?
Well, for starters lets show what is called the "basic" setup:
In this setup exported mail is placed into the FidoNet mail area.
Each message exported from a Conference Mail area has one
message generated for each receiving system. This mail is then
sent the same as any other network mail. When Echomail was first
created this was the only way mail could be sent.
The "basic" method has some disadvantages. First, since Echomail
has grown so large it is not uncommon to get 200 new messages per
day imported into various message bases. It is also not uncommon
for a system to be exporting messages to 4 or 5 other systems.
Simple arithmetic shows 800-1000 messages per day would be sent
in normal netmail! This puts a tremendous strain on any netmail
system, not to mention transmission time and the resultant phone
charges. When this limitation of Echomail was first noticed a lot
of people started scratching their heads wondering what to do. If
a solution could not be found it appeared Echomail would
certainly overrun the capabilities of FidoNet.
Thom Henderson (from System Enhancement Associates) came up with
the original ARCmail program. Having previously written the ARC
file archiving and compression program, he knew the savings
achievable by having all of the netmail messages placed in .ARC
format for transmission. As a byproduct, the messages no longer
appeared in the netmail area, but were included in a file
attached to a message (see your FidoNet mailer manual for file
attaches). In this way the tremendous number of messages
generated, and the phone bill problems were both solved.
Unfortunately, ARCmail required the messages to first be placed
into the netmail area before it could be run. In effect, it
caused the messages to be scanned once when they were exported,
once during the ARCmail phase, once when ARCmail was run at the
other end to get the messages out of .ARC format, and once when
those messages were later imported into a message base on the
receiving system. The Conference Mail System solves this problem
by eliminating the ARCmail program. Conference Mail builds the
ARCmail files during Export, and unpacks them during Import. This
way messages are exported directly to ARCmail style file
attaches, and imported directly from ARCmail style file attaches.
The scanning phases between importing and exporting messages are
totally removed and processing time is proportionally reduced.
This is now the most common method for sending Conference Mail
between systems. The overhead involved in doing it during the
importing and exporting phases is much less than what is involved
if ARCmailing is not utilized. This was a primary consideration
in the design and implementation of the Conference Mail System,
and as a result the entire system is optimized for this type of
use. Please refer to the Import and Export functions for
specifics on how to use the ARCmailing feature.
CONFERENCE TOPOLOGY
The way in which systems link together for a particular
conference is called the "conference topology." It is important
to know this structure for two reasons: 1) It is important to
have a topology which is efficient in the transfer of the
Conference Mail messages, and 2) It is important to have a
topology which will not cause systems to see the same messages
more than once.
Efficiency can be measured in a number of ways; least time
involved for all systems to receive a message, least cost for all
systems to receive a message, and fewest phone calls required for
all systems to receive a message are all valid indicators of
efficiency. Users of Echomail compatible systems have determined
(through trial and error) the best measure of efficiency is a
combination of all three of the measurements given above.
Balancing the equation is not trivial, but some guidelines can be
given:
1. Never have two systems attempting to send Conference mail
to each other at the same time. This results in "collisions"
that will cause both systems to fail. To avoid this, one
system should be responsible for polling while the other
system is holding mail. This arrangement can alternate based
upon various criteria, but both systems should never be
attempting to call each other at the same time.
2. Have nodes form "stars" for distribution of Conference
Mail. This arrangement has several nodes all receiving their
Conference Mail from the same system. In general the systems
on the "outside" of the star poll the system on the
"inside". The system on the "inside" in turn polls other
systems to receive the Conference Mail that is being passed
on to the "outside" systems.
3. Utilize fully connected polygons with a few vertices.
Nodes can be connected in a triangle (A sends to B and C, B
sends to A and C, C sends to A and B) or a fully connected
square (all corners of the square send to all of the other
corners). This method is useful for getting Conference Mail
messages to each node as quickly as possible.
All of these efficiency guidelines have to be tempered with the
guidelines dealing with keeping duplicate messages from being
exported. Duplicates will occur in any topology that forms a
closed polygon that is not fully connected. Take for example the
following configuration:
A ----- B
| |
| |
C ----- D
This square is a closed polygon that is not fully connected. It
is capable of generating duplicates as follows:
1. A message is entered on node A.
2. Node A exports the message to node B and node C placing
the seen-by for A, B, and C in the message as it does so.
3. Node B sees that node D is not listed in the seen-by and
exports the message to node D.
4. Node C sees that node D is not listed in the seen-by and
exports the message to node D.
At this point node D has received the same message twice - a
duplicate was generated. Normally a "dup-ring" will not be as
simple as a square. Generally it will be caused by a system on
one end of a long chain accidentally connecting to a system on
the other end of the chain. This causes the two ends of the chain
to become connected, forming a polygon.
In FidoNet this problem is reduced somewhat by having "Regional
Echomail Coordinators" (RECS) that try to keep track of Echomail
connections within their regions of the world. A further rule
which is followed is that only the RECS are allowed to make
inter-regional connections for the larger conferences. In return,
the RECS have established a very efficient topology which gets
messages from coast to coast, and onto over 200 systems in less
than 24 hours. If no one were willing to follow the rules, then
this system would collapse, but due to the excellent efficiency
it has remained intact for over a year.
Why a PATH line?
As was previously mentioned, the PATH line is a new concept in
Echomail. It stores the net/node numbers of each system having
actually processed a message. This information is useful in
correcting the biggest problem encountered by nodes running an
Echomail compatible system - the problem of finding the cause of
duplicate messages. How does the PATH line help solve this
problem? Take the following path line as an example:
^aPATH: 107/6 107/312 132/101
This shows the message was processed by system 107/6 and
transferred to system 107/312. It further shows system 107/312
transferred the message to 132/101, and 132/101 processed it
again. Now take the following path line as the example:
^aPATH: 107/6 107/312 107/528 107/312 132/101
This shows the message having been processed by node 107/312 on
more than one occasion. Based upon the earlier description of the
'information control' fields in Echomail messages, this clearly
is an error in processing (see the section entitled "How it
Works"). This further shows node 107/528 as the node which
apparently processed the message incorrectly. In this case the
path line can be used to quickly locate the source of duplicate
messages.
In a conference with many participants it becomes almost
impossible to determine the exact topology used. In these cases
the use of the path line can help a coordinator of the conference
track any possible breakdowns in the overall topology, while not
substantially increasing the amount of information transmitted.
Having this small amount of information added to the end of each
message pays for itself very quickly when it can be used to help
detect a topology problem causing duplicate messages to be
transmitted to each system.
-30-
