FSC-0024 - A Proposal for a Type-3 Mail Bundle - Oliver McDonald
Notes on Type three bundlers. Notes on Type three bundlers.
The first important note is that without Wynn Wagner's
work on FTSC-0014, none of this would have come to fruition.
I owe him a great debt in this area, as well as the debt for
Opus itself that got me into this. Thanks Wynn.
Type 3 bundlers offer opportunities for new levels of
sophistication in mail processing. As the first step Aurora
Computer Technologies plans to provide the minimum specified
by the existing Type 3 bundle specifications with one minor
addition. This addition is the inclusion of the features of
ReMapper. This addition is not a required inclusion for
other software authors producing a Type 3 bundler.
To sum up, standard required features are:
Must be able to create and unbundle Type 2 Bundles (See
FSC-0001)
Must be able to create and unbundle Type 3 Bundles (See
attached)
Must be able to Toss EchoMail from either Type 2 OR 3
bundles
Must generate an update-required message for the sysop if
the MinorVersion changes.
Must generate an update-required message if it encounters
a misc packet type it does not recognize.
Initial optional features are:
May Duplicate the functionality of ReMapper.
May automatically generate an F.Req. from source of bundle
when the minor version changes.
May generate an F.Req from source of bundle if it
encounters a misc packet type it does not recognize.
All error messages are placed in Matrix Mail messages to
the Sysop.
Will create outbound bundles on the fly from the inbound
bundle. Does not need to scan these messages. Note, if
this option is exercised it is IMPERATIVE that the areas
are scanned prior to the unbundle process.
Type 3.0 proposal (preliminary) Type 3.0 proposal (preliminary)
This proposal allows for automatic updating of the Type
3 bundle to allow for further revisions and enhancements.
Thus we will refer to it as a Type 3.0 with further versions
becoming Type 3.1 etc.
All multi-byte data forms (int/long) are considered to
have the MSB first and the LSB last. Int is two bytes, and
Long is four.
Bundle Header Bundle Header
struct _BundleHeader {
struct _Address B_Destination;
struct _Address B_Origination;
unsigned nybble B_BundlerVersionMajor;
unsigned nybble B_BundlerVersionMinor;
unsigned byte B_ProductCode;
unsigned byte B_VersionMinor;
unsigned byte B_VersionMajor;
unsigned long B_CreationTime;
unsigned byte B_Password[8];
};
Bundle Header Notes
This works out to 32 bytes which is a nice size to work
with.
Here follows a short explanation for each field:
"B_BundlerVersionMajor/Minor" provide for version
numbers from 0.0 to 16.16, this should be enough for all
except TJ.
"B_ProductCode" is the FTSC assigned product code.
This can be used to identify just which type 3 bundler
created the bundle; it should not be considered an error if
this is unidentified, and need not be processed on
unbundling but MUST be included _correctly_ at the bundling
stage.
"B_VersionMinor" is a version number that will
initially start at Zero and is used to allow non backward
compatible changes to Type 3 bundles, such as header length
change. If this is LOWER in the bundle than the
corresponding version number in the unbundler it should
abend. It is suggested that a short message be written to
the Sysop in NETMAIL with as much information gleaned from
the header as possible. (All info up to and including
"B_VersionMajor".)
"B_VersionMajor", always 3. This and all data prior to
this point is position dependant and will never be changed
in future Type 3.0 bundle revisions.
"B_CreationTime" is an Unix 1970 based creation time
indicating the time/date the bundle was created.
"B_Password" is a NULL padded character array that may
contain uppercase alpha bytes or ASCII digits. It should
not contain lowercase characters, punctuation, control
characters etc. A maximum of 8 characters are significant.
Struct _Address Struct _Address
struct _Address {
unsigned int Zone;
unsigned int Net;
unsigned int Node;
unsigned int Point;
};
struct _AddressShort {
unsigned int Net;
unsigned int Node;
};
Bundle Footer Bundle Footer
Struct _BundleEnd {
Unsigned Byte E_Packet_Type /* Always 0 *
/
};
Bundle Footer notes.
All bundles end with this packet. It is not optional
and the packet should be considered grundged if it is
missing.
Area header Area header
Struct _AreaHeader
Unsigned byte E_Packet_Type /* Always 1 */
Unsigned byte E_NameLength /* Actual bytes
in E_NAME */
Unsigned Byte E_Name[1] /* Variable
length field */
Area Header Notes
The area header packet marks the start of a sequence of
messages destined to the same message area. The area
indicated in the Area Header will remain valid until either
the end of the bundle OR another Area Header is encountered.
E_Name will usually contain the area name of the echo
area that subsequent messages should go. If E_NameLength is
zero then the subsequent messages should go the NetMail
area. Any messages that occur prior to the first Area
Header in a bundle should also go to the Netmail area.
The Maximum value for E_NameLength is 63.
E_Name is NOT null terminated.
Message Header Message Header
Struct _MessageHeader {
Unsigned byte M_Packet_Type /* Always 2 */
Struct _Address M_Destination /* Final
Destination */
Struct _Address M_Origin /* Where the
message was entered */
Unsigned Long M_CreationTime /* When the
message was entered */
Unsigned int M_Attributes /* FTSC
bitweights */
Unsigned byte M_FromLength
Unsigned byte M_ToLength
Unsigned byte M_SubLength
Unsigned byte M_From[1]
Unsigned byte M_To[1]
Unsigned byte M_Sub[1]
};
Message Header Notes
Every message begins with a message header packet. It
should be created by the system where the message
originated. If there are any intermediate stops along the
way it is the responsibility of the intermediate systems
along the way to maintain all of the information without
modification.
None of M_From, M_To, or M_Sub are to be NULL
terminated.
Message Body Message Body
Struct _Text {
Unsigned byte T_Packet_Type /* Always 3 */
Unsigned int T_ByteCount /* # of bytes (
< 0x1000 ) */
Unsigned byte T_Data[1] /* Variable
length field */
Message Body Notes:
The message body is considered one or more _Text
packets.
No _Text packet will be more that 1000H bytes long
(that's 4096 to the terminally base 10 folks). Of course
there may be a near infinite number of _Text packets per
bundle/Message header, but you are absolutely positively
guaranteed that with the type 3.x method you will never need
a buffer larger than 1000H.
In addition to ASCII values 20h through 7Eh (inclusive)
the following control codes are legal for TEXT data. Note
that <CR> and <softCR> are NOT in this list, thus type three
packers will eliminate spurious 0Dh's.
<LF> 0Ah Forced
<CR>/<LF>
<dle> 10h Replicate
Other control characters and values 7Fh and above are
symptomatic of a grundged message.
Replicate is a three byte sequence: <dle> <value>
<length>. For example if a packet contains the bytes 10h,
20h, 09h it should be expanded in the message body as nine
<space> characters.
There is no minimum or maximum line length, it is
assumed that the reader can supply the appropriate line
wraps.
One "line" of a message may cross from one _Text packet
to another.
EchoMail: EchoMail:
Struct _EchoMailInfo {
Unsigned byte EI_Packet_Type /* Always 4 */
Struct _EID EI_Parent /* Up message
thread */
Struct _EID EI_Child /* Down message
thread */
Unsigned byte EI_SeenByCount
Unsigned byte EI_PathCount
Struct _AddressShort EI_SeenBy[1]
Struct _Address EI_Path[1]
);
EchoMail notes:
The EI_Child and EI_Parent fields are used to
reconstruct the message thread.
Type 3 bundles uses binary seenby and path information,
but should convert to a normal seenby/path in the unbundled
messages. If the auto-rebundleing is used it is not
necessary to process the seenby's into the unpacked
messages. It is suggested that if this approach is used it
is HEAVILY tested prior to implementation, and that it still
store the data somewhere for retrieval in case of unresolved
dupe problems.
Cargo Info. Cargo Info.
Struct _PointInfo {
Unsigned byte CI_Packet_Type /* always 05h
*/
Unsigned byte CI_File Count /* Number of
files */
Unsigned byte CI_FileName[1] /* Filenames
(10 bytes */
};
Cargo Info Notes
The Cargo info packet will only be found in a Type 3
arcmail bundle that contains files. It will always be the always
second packet in a bundle.
Node info Node info
Struct _NodeInfo
Unsigned byte NI_Packet_Type /* always 06h *
/
Unsigned int NI_Flags /* Flags for
node */
};
This packet is sent to a Type 3.x node in the first
bundle to be sent to that node. The bundler should detect
that the node can accept a Type 3.x bundle from a nodelist
flag. It will automatically generate this packet at that
point. Should a type 3.x bundle come from a node that is
not identified in the nodelist as type 3.x capable the
bundler should mark that node as Type 3.x capable and
generate a warning message.
NI_Flags, is a bit mapped field that identifies the
characteristics of the node. Some of this information will
duplicate that information found in the nodelist. This is
used as a check.
Bit: Meaning:
0: Type 3
1: Packing Protocol Bits.
2: " " "
3: " " "
4: |Bits 3 & 4 are used together
5: |to determine mail handleing.
6: Ingate
7: OutGate
8: Net Host
9: Net Hub
A: <Reserved>
B: <Reserved>
C: <Reserved>
D: <Reserved>
E: Sent info
F: Got Info
Bits 1, 2, & 3 are used to determine how mail may be
packed for this node (SEA, PKWare, ZOO)
Bits 1 & 2 & 3: Meaning:
000: No packing.
001: SEA Archive format.
011: PKWare LZW packing format.
100: ZOO Compression system.
Note that these bits were intended to be combined, so
that if a node could handle ZOO and SEA Archives it would
set the bits to '101'. Since by definition PKWare can
handle SEA format Arc's it is considered standard to set
both bits for a PKWare capable system.
Bits 4 and 5 are used to determine how mail should be
sent to this node (CM, hold, direct)
Bit 4 & 5: Meaning:
00: Direct
01: Continous
11: Hold
The sysop should be able to clear the sent info bit
should the status of his system change (ie becomeing an NC).
Zone gates may be identified by the fact that they are in
Net 1 and they are both an ingate and an outgate. The zone
they are the gate for is identified by their node number.
MiscInfo (IFNA Kludge). MiscInfo (IFNA Kludge).
Struct _MiscInfo {
Unsigned byte MI_Packet_Type /* Always 06h -
FFh, assigned by FTSC */
Unsigned byte MI_ByteCount /* # of bytes
of miscinfo */
Unsigned Byte MI_WhoKnows[1] /* Misc Stuff
*/
};
MiscInfo Notes:
The Misc info packet(s) are the last packets associated
with a message, there may be more than one in extreme
circumstances, but this should prove to be unlikely. The
bundler must retain any information in these packets
unchanged if it is a routed message.
This is a catch all packet that replaces the dreaded
IFNA kludge. It is designed to only be used as an interim
method. At present all IFNA kludges are handled in other
special purpose packets, it is foreseen that any further
kludges will be handled only by miscinfo packets until the
Type 3.x bundle specification can be updated and coded to
handle the new data.
MiscInfo packets in the range 80h - F0h should be
preserved if not understood. It should be considered an
error condition to find a MiscInfo packet with in the range
04h - 7Fh, this range is reserved for future expansions on
Type 3 packets. Packets in the range F1h - FFh should be
unpacked AND a warning message should be generated. These
numbers will be used on an extremely temporary basis as they
are designed for ESSENTIAL IFNA kludges and will be added
into the Type 3.x specification as quickly as possible.
Arcmail and Type 3.x Arcmail and Type 3.x
Type 3.x bundlers support arcmail much the same way
that the type 2 bundlers do. There are some enhancements in
the arcmail naming scheme however, that help reduce system
overhead for routed mail.
For arcmail destined for type 2 based systems the old
reliable method of arcmail file naming will be used, IE:
NNNNnnnn.ww#
Where NNNN is a four hex digit net number, nnnn is a
four hex digit node number, ww is a two character weekday-
name identifier, and # is the packet number for that day.
Type 3.x packers SHOULD generate the day name correctly
rather than the OMMM 1.08 cyclic method.
Here follows a suggested Type 3.x ArcMail naming
scheme, basically a modification of Roeland Meyer's original
proposal. I have been made aware that Roeland has some
things to say on this, but there seems to be a
communications break between us, so until I can contact him
I will stick with this.
For Arcmail destined to a Type 3.x system (with Type
3.x bundles internally), a variation of the method first
proposed by Roeland Meyer will be used. Here follows a
quick synopsis:
New address specifier (Re-edited by Oliver McDonald)
This is designed for the Type 3.0 Arcmail naming convention
of:
ZZNNNOOO.Fxx
| | | ||
| | | |`----> Incremental sequence number, base 10, max
= 99d
| | | | Starts at 00 and counts to '99' then
wraps
| | | | back to 00. No "Day-of-week" info.
| | | | This is strictly to avoid bundle
collisions.
| | | | An 'empty' version of the bundle is
kept
| | | | around to help the router remember
what the
| | | | last sequence number was.
| | | |
| | | `-----> Flag to indicate bundle type
| | | Allowed values:
| | | 'All non-specified flags are
reserved.
| | | 'U' - ZOO File bundle
| | | 'V' - ZOO Mail only bundle for a
Point.
| | | 'W' - ZOO Mail only bundle
| | | 'X' - File bundle
| | | 'Y' - Mail only bundle for a
point.
| | | 'Z' - Mail only bundle
| | | For files with the 'Y' flag it is
| | | sent as per normal until it reaches
the
| | | node specified by the arcname. At
this
| | | point the node will unarc the FIRST
bundle
| | | in the arc, and read the Message
Header,
| | | and then attach the bundle to the
point
| | | specified.
| | | For File bundles if the files are to
be
| | | forwarded, the node will unarc the
bundle
| | | in the arc. It will check the
message header
| | | for address (match against name),
and will open
| | | the Cargo Info Bundle, and attach
those files
| | | to the destination.
| | | Note: If the addresses do not
match it considered
| | | an error to forward the files.
| | | Note: The point address is not
considered for
| | | matching purposes.
| | |
| | `--------> Node address, base 36, max = 56,654d
| | Allowed values: '000' to 'ZZZ'
| | This is the Node part of the
destination
| | address of the bundle.
| | Special values:
| | '000'- Destination is the Net Host
given by
| | ZNNN, not forwarded to any
Nodes.
| | 'ZZZ'- This a broadcast bundle to
ALL Nodes
| | in the Net given by ZNNN,
as
| | well as, the Net Host given
by same.
| |
| `-----------> Net address, base 36, max = 55,654d
| Allowed values: '000' to 'ZZZ'
| This is the Net part of the
destination
| address of the bundle.
| Special values:
| '000' - Destination is the
ZoneGate given by
| ZZ, not forwarded to any
Nets.
| 'ZZZ' - This a broadcast bundle to
ALL Nets
| in the Zone given by ZZ,
as well as,
| the ZoneGate given by
same.
|
`--------------> Zone address, base 36, max = 1,294d
Allowed values: '00' to 'ZZ'
This is the Zone part of the
destination
address of the bundle.
Special values:
'00' - Destination is the current
ZoneGate.
'ZZ' - This a broadcast bundle to
ALL
ZoneGates given by the
NodeList, as
well as, the ZoneGate given
by same.
Note, Point numbers are specifically NOT included in
the file name identifier. There were a couple of reasons
for this; first, we wanted to allow the maximum range of
Zone:Net/Node numbers to be available; second, anyone
running points should not be doing so on a minimal system
anyway.
Note(2), Special bundle names (ZZZ or 000) are
implemented optionally by the destination. You should not
assume that it will work. A future Type 3.x spec will
include password protection for this.
The logic for providing for up to 100 packets allowable
to a specific node is that I have seen cases of a :CM Net
Host generating in excess of 10 messages for a node in one
day, and the next logical number is 100.
Should a Type 3.x destination fall outside the range
available to the Type 3.x arcmail limits, then the bundler
will fall back and use the Type 2 arcmail naming scheme.
Notes on Zone Gates Notes on Zone Gates
With type 3 bundles the ZoneGates software load is MUCH
easier, all it has to do is simply forward the Type 3.x
bundle. It is suggested that it may be VERY desirable to VERY
have Type 3.x bundlers duplicate the functionality of the
Zone Gate software. At the very least it is STRONGLY
suggested that ZoneGates upgrade to Type 3.x capable
bundlers as soon as they become available.
Notes for Type 3.x Developer's: Notes for Type 3.x Developer's:
The latest specs for Type 3.x will be available in the
FTSC library at all times and at 1:342/1. Developer's who
register with 1:342/1 will have upcoming changes netmailed
to them as they are confirmed. Any upcoming change notices
will have a date officially implemented. This date will
always be in the future and should be considered an official
release date of the new Type 3.x standard.
Every attempt will be made to allow developer's a
reasonable time period to upgrade to the new standard. It
is important that developer's attempt to meet this date as
these changes are usually NOT backward compatible. Code
samples will also be F.Req'able from 1:342/1 as the magic
file name TYPE3x where x is the latest revision to the Type
3 standard. Should a developer be unable to meet the
release date he should notify the FTSC and/or 342/1
immediately. The release date is based on an estimate made
by Aurora Computer Technologies. If there is a good reason
the release date will be pushed back, and ALL developers
will be notified. As the new Type 3.x standard will not be
official until the release date no developer will release
his code early.
On that subject, care should be taken by the developer
to let no new format bundles escape his beta test systems.
There are a couple of approaches we recommend for this, the
first is to have beta test versions only generate the new
format bundle for specific zone:net/node addresses, the
second is to set up a completely separate private net for
testing purposes.
Release method: Since the bundler will automatically
spread news of itself with use, a simple zero effort release
program may be used. As different versions of the Type 3.x
bundlers will require different operating environments, you
should try to get your bundler made available on Echo-
BackBone and Echo Hub systems. The reasoning behind this,
is that it is from these systems that the existence of the
new bundler will become common knowledge. The other place
to send it would be the Zone, Regional, and maybe Net Co-
ordinators.
Future of Type 3 Future of Type 3
Since the Type 3 format proposed provides for a new
level of information exchange in Matrix mail I provide here
a few advance hints of what is planned.
AutoEcho built in. AutoEcho built in.
Replace AutoEcho/AreaFix with automatic security. This
security is such that the hub will not need to pick a
password and send it in netmail to the downstream node prior
to the downstream node requesting echos.
Instead, the downstream node will request an echo, at
which point the Hub's bundler will generate a netmail
message to the Hub Sysop. Now the hub Sysop may decide to
give it to him. If he does, he simply tells his bundler to
start sending it downstream to him.
Now since this last paragraph has already confused
people, I will provide a scenario with names. Here in Net
342 we have our NEC as Brian McCullough (BDMc for short),
and our REC is Steve Barnes (SB). We have BDMc requesting
the echo NET_DEV from SB. The sequence is as follows:
BDMc requests NET_DEV.
BDMc's bundler sees this and generates the echo request
packet.
This packet is bundled and sent to SB.
SB's bundler finds the bundle.
SB's bundler sees that BDMc is authorized to have that
echo.
SB's Bundler generates an acknowledge packet and starts
sending the echo to BDMc.
BDMc's Bundler gets the acknowledge and sets up the area.
BDMc's Bundler will use the password it was sent for
future requests.
If there was a problem with access to the requested
echo Steve Barnes would have received a NetMail message from
his bundler and he would be able to make a decision at this
point. Other than that he need not even be in the country.
Minor details on this, the Hub (or upstream node) can
specify levels of permission for this autoecho request
process, and deny certain echos to certain downstream nodes.
If a downstream node requests a denied echo, the upstream
node's bundler will again generate a netmail message to the
Hub Sysop informing him of what happened.
This will probably be implemented as a downward
compatible upgrade with the request for new software
triggered by the first request for a new echo. Note, if
standard distribution applies this should never generate a
request. However as things do not always work that way, the
automatic notification and optional file request should
solve any major problems.
The Future of the Aurora Type 3 Bundler The Future of the Aurora Type 3 Bundler
Fowarding of bundles that costs money. Fowarding of bundles that costs money.
All forwarded bundles that will cost money will be
marked as HOLD unless either the receiving OR the sending
node are marked as send-to or accept-from appropriately.
All keywords will be valid in these cases. This is a
completely backwards compatible change.
Forwarding Cost bundles from Points. Forwarding Cost bundles from Points.
The forwarding of cost bundles from points will be done
on the basis of a credit that the point has. The credit
will be monitored in the USER.BBS file, with the record
number corresponding to the point number. This is a
completely backwards compatible change.
Final Notes: Final Notes:
Final Note: Would all those planning on writing a Type
3.0 bundler please contact me (Oliver McDonald) via NetMail
(1:342/1).
Final Note(2): There are already some planned
extensions to Type 3.0, they will not be strictly required
and will not create a new VersionMinor number, but will add
functionality, and will when used require an update. It is
my feeling that if you are aware of these plans, you will be
able to integrate them better at the point they are
"officialized". It is not my desire to become the only Type
3.x developer out there. It is merely my desire to be able
to be one of them, and also to be able to make Type 3.x so
attractive to all that everyone will want to run it.
Final Note(3): Send Code (tm, Bob Hartman).
Final Note(4): Convince me(tm) of suggested changes.
Kudos: Kudos:
Thanks to all the people in Net_Dev who have made
suggestions and comments on this proposal as I worked on it.
Your comments are appreciated (even those I have not used).
I would like to especially thank the following people:
Wynn Wagner III FSC-0014 and support.
Roeland Meyer Work on ArcName routing.
Randy Bush Suggestions and support.
Brian McCullough Sounding board and Cattle
Prod.
