Regular Broadcasting of PACSAT File Headers
DRAFT
04 November 1991
[All changes and additions in this DRAFT are PROPOSED, and should not be
considered final until a subsequent RELEASE version of this document is
published.]
JW Ward, G0/K8KA
HE Price, NK6K
DISCUSSION
In the past 18 months the PACSAT Broadcast Protocol has proven to be a very
efficient means of downloading information from the LEO store-and-forward
satellites. It makes good use of the available bandwidth, and especially
efficient use of available pass time. The authors always intended to extend
this datagram facility to the provision of a "file directory" for the PACSAT
server. Since all stations need an up to date directory, this is a prime
candidate for point-to-multipoint distribution.
A PACSAT file server's "directory entries" are, in fact, PACSAT File Headers
(PFHs). The PFH contains all of the important information about a file (name,
number, upload time, title, addresses, etc.). This information is present in
the PFH form on the server, and routines for interpretation of the PFH are
present in the client software. The PFH is the natural data structure in which
to present "broadcast directories."
The broadcast directory service will, therefore, consist of periodic
transmission of PFHs in special Directory Broadcast datagrams. The
transmissions will be either automatic or client initiated. This protocol
(see Appendix A) specifies the format for Directory Broadcasts and for
Directory Broadcast Requests. The protocol does not specify when a server
might periodically transmit PFHs without client requests, but such automatic
directory broadcasts will be implemented on the existing PACSAT servers.
CHOOSING A HANDLE
Having decided that a "directory entry" is a PFH and that "directory
broadcasting" is PFH broadcasting, we must address the problem of how the
client can assure that it has received a complete and up-to-date directory
from the server.
Although the server uniquely identifies each file with an identification
number, this is not necessarily the best way to for the client to request
directory entries from the server. A system based on file numbers will not
always assure that the client receives directories for all "new" files. The
following cases illustrate this:
(1) Partial Uploads
When a client begins uploading a file, the server assigns a file number. The
client may not finish uploading the file for some time. Only when the file
upload is finished is the file a "new" file.
(2) Non-existent Files
Some file numbers are used only briefly (e.g. for temporary files) and may
never be associated with a "new" file (as seen by the clients).
(3) Non-PACSAT Files
Files created by on-board processes, but lacking the PFH. They cannot be
downloaded or uploaded. As far as the PACSAT protocol suite is concerned,
these are like (2).
(4) Daily Log Files
The server and its processes also maintain daily logs which, although they may
become "new" several times during a day, will cease to grow after midnight.
(5) Continuous Log Files
The server and processes on the server maintain log files. These files become
"new" every time something is written into them.
During the design of the directory broadcast protocol, we attempted to solve
these problems using simple measures which would still allow file number to be
the primary directory request handle. Problems persisted both in the complete
specification of the protocol and in a prototype implementation of the
groundstation receiving software.
Eventually the file-number-based scheme was abandoned in favour of a slightly
more complex protocol providing two "time limits" with each broadcast
directory entry: an "older limit" , t_old, and a "newer limit", t_new. Thus,
as well as containing a PFH, each broadcast directory entry gives the client a
pair numbers (t_old, t_new). The proper interpretation of this pair is
"There are no files other than this file with
UPLOAD_TIME >= t_old and <= t_new."
The client maintains a list of holes in its directory time line using the
t_old and t_new information. Requests to the server for hole fills use these
times, rather than file numbers.
EXAMPLES
[N.B.: Small times have been used deliberately in the examples; real satellite
times are the number of seconds elapsed since 00:00:00 1/1/80. Also, the
satellite file system assures that no two files will ever have the same
UPLOAD_TIME.]
For example, a client with no previous directory information assumes that it
has a directory hole from
0 - FOREVER
Upon receiving a directory broadcast with (t_old, t_new)
(0, 50)
the client's directory hole list would look like
51 - FOREVER
and the client software is confident that there are no older files.
The list can get fragmented, e.g. if the client now receives
(120, 150)
and the time hole list looks like:
51 - 119
151 - FOREVER
Requests from the client to the sever are time hole lists. In effect, they
request that the server transmit any directory entries in the hole OR an entry
which shows that there are no directories in the hole.
FLEETING HOLES
The server sets time limits for a particular directory broadcast by examining
the next newer and previous older files on the system. These files, however,
may move or be deleted, hence the need to show that a hole no longer holds any
directories. Starting from the previous example, say that the server creates
files at t=151, t=152 and t=153, thus it would transmit
[a] (120, 151) (PFH for t=151)
[b] (152, 152) (PFH for t=152)
[c] (153, 153) (PFH for t=153)
A client receiving [a] and [c], but missing [b] would have a hole
152 - 152
in its directory time line, and might assume that there was a file to fill
that hole. But the server might delete the file t=152. Thus, when the client
requests a fill for
(152,152)
the server would respond with
[c'] (152, 153) (PFH for t=153)
which allows the client to fill its hole, but does not contain a directory
entry for t=152.
MARKING THE LIMITING FILES
It is convenient for the client to know when it has received the directory for
the current newest file. The server will mark this file with a special flag.
The oldest file will always have t_old==0.
BACKWARD COMPATIBLE IMPLEMENTATION
Unfortunately, adding new data and request frame formats to the existing
broadcast protocol in a backward compatible manner required some compromises.
On inspection of the existing FRAME_HEADER for broadcast data packets, there
is no clear indication of frame type! Strictly speaking, the reserved bits of
the FRAME_HEADER.flags byte could be used to indicate that the frame was a
directory broadcast frame. Unfortunately, PB at least does not check these
bits, and any new directory broadcasts would terminally confuse PB. We expect
that clients will try to get updated broadcast receiver software, but we don't
want to force the pace of this change by causing the current implementation to
crash.
We completely avoid this problem, by specifying a separate PID for directory
broadcast and request frames. This should allow existing groundstation
software to co-exist with the new broadcast directories.
IMPLEMENTATION NOTES
A single implementation of PACSAT Protocol Suite software runs on UO-14, AO-
16, LO-19 and UO-22. Although there are minor variations from satellite to
satellite, these are generally related to ongoing upgrades rippling through
the system. There are some implementation peculiarities, and some ad-hoc
additions have been made to the standard broadcast protocol. These matters are
documented here, but are not added to the formal specification.
Queue Management
----------------
The server maintains a queue with 10 entries; each entry is a hole list
request or broadcast start request. A particular station (as identified by
callsign, not including SSID) can have only one entry on the queue.
Entries are removed from the queue:
after 10 minutes;
after a hole list has been completely transmitted (for hole list);
after a file has been completely transmitted (for start request);
when a new request is received from a station already in the queue;
if the file associated with the entry cannot be opened and read;
When the server receives a request frame from a client, it will respond
immediately indicating the success or failure of the request. This response is
intended for use in manually operated stations; because of timing and
reliability problems, its use in automated broadcast request systems is
discouraged.
The request response message is an AX.25 UI frame from the server callsign to
the client callsign with the PID 0xf0. [NB - this is a change of the PID.] The
data portion of the packet has no header or CRC. The data is in ASCII as
follows:
<"OK "|"NO ">[<errorcode>]<callsign><CR>
"OK " indicates that the request was successful. There is no <errorcode>.
"NO " indicates that the request failed for some reason. In this case, the
<errorcode> is an ascii representation of an error number followed by one
blank space, e.g. "NO -1 ". The error codes for broadcast start requests and
hole list requests should be interpreted as follows:
-1 Temporary problem, e.g. queue full or server closed
-2 Permanent problem, e.g. file does not exist.
Broadcast Server Status Message
-------------------------------
The format of the PBLIST message will be altered to include (at the option of
spacecraft controllers) the number of the file being broadcast. The format of
the PBLIST message will now be one of the following -
\P indicates that the request is permanent
\D indicates that the request is a directory request
<call> is the callsign of the requesting station, without SSID
<fnumber> is the hexadecimal file number of the requested file
If the queue is empty, the message will be :
PB: Empty.
These messages will be transmitted to PBLIST-0 with PID 0xF0.
File Types
----------
There is still some confusion regarding compressed files. The intent was to
place all compression information in the compression_type field and all file
format information in the file_type field. Thus, as per the FTL0 update
released 22 March 1991:
Delete file type 0x0a, "Simple ASCII text file, but
compressed" from Appendix A, PACSAT FILE TYPES. This
type of file should be indicated by file_type 0x00 and
a non-zero compression_type item.
We propose to fix the conflict between "miscelaneous binary" and "activity
log" by changing the activity log file type. Since only Harold and I have to
take action to implement this solution, it is more likely to succeed than
trying to recall every copy of PFHADD.
Suggestions regarding additional, more-informative, file types are solicited.
Compression Types
-----------------
The PFH protocol was designed to be flexible. We should now add a compression
type for the popular LHARC program. Add to APPENDIX B of the document "PACSAT
File Header Definition"
0x03 body compressed using LHARC.
--------------------------------------------------------------
--------------------------------------------------------------
BROADCAST DIRECTORY PROTOCOL PROPOSED ADDITIONS TO APPENDIX A
OF THE PACSAT BROADCAST PROTOCOL SPECIFICATION
--------------------------------------------------------------
The server will transmit directory information automatically and/or as the
result of Directory Requests from clients. The directory information provided
is the PACSAT File Header (PFH). For a particular file, the directory entry
will be transmitted in one or more Directory Broadcast frames, each of which
is carried in the data portion of an AX.25 UI frame. Each frame will also
contain the DIR_HEADER described below.
To aid the client in maintaining its directory database and requesting up-to-
date directory entries, Directory Broadcast frames include time limits. These
limits allow the client to maintain a "time line" of directory entries and to
determine when the time line is complete and up-to-date.
Servers will periodically transmit the directories for the most recently
uploaded files. Other directory entries can be requested by the client using a
Directory Request frame. The Directory Request format allows clients to
request that the server transmit directories for all files with UPLOAD_TIME in
a particular range. This is equivalent to filling holes in the directory time
line.
A.5.1 Directory Broadcast frame format
A directory entry is transmitted as one or more directory broadcast frames.
Directory broadcast frames consist of header, data and CRC. The header is
fixed length, and the data is variable length.
A directory broadcast frame is wholly contained in the data portion of an
AX.25 UI frame.
A <UI> frame containing a directory broadcast frame uses a PID of 0xbd.
A <UI> frame containing a directory broadcast frame uses a source address of
the transmitting station, and a destination address of QST-1.
A.5.1.1 Frame Header
Each directory broadcast frame consists of a header occupying the first 17
bytes of the frame. The structure of the frame header is as follows:
<flags><file_id><offset><t_old><t_new>
This structure follows the PACSAT Data Specification Standards as regards
field size and byte order; these are defined in a separate document.
struct DIR_HEADER {
unsigned char flags;
unsigned long file_id;
unsigned long offset;
time_t t_old;
time_t t_new;
}
flags A bit field as follows:
7 6 5 4 3 2 1 0
/----------------------\
|* N E 0 V V T T|
\----------------------/
TT Two bit frame type identifier
00 PFH broadcast
01 reserved
10 reserved
11 reserved
VV Two bit version identifier. This version is 00.
0 Always 0 indicates a server generated frame.
E 1 Last byte of frame is the last byte of the directory PFH.
0 Not the last frame.
N 1 This is the newest file on the server.
0 This is not the newest file on the server.
* Reserved, always 0.
file_id A number which identifies the file. All directory broadcast
frames which are part of the same file's PFH are tagged with this number.
offset This is the offset from the start of the PFH for the first data
byte in this frame.
t_old Number of seconds since 00:00:00 1/1/80. See below.
t_new Number of seconds since 00:00:00 1/1/80. See below.
There are no files other than the file identified by
file_id with t_old <= UPLOAD_TIME <= t_new.
A.5.1.2 Data
The data portion of a directory broadcast frame will contain all or part of
the PACSAT File header from the file identified by <file_id>. The <offset>
field indicates where the data from the current frame belongs in the PFH.
An <offset> of 0 and a <flags> field with the E bit set to 1 indicates that
this directory broadcast frame contains the entire PFH for the identified
file.
A.5.1.3 CRC
The CRC covers the header and data portions of the directory broadcast frame.
It is the XMODEM CRC as defined in section A.2.1.3.
A.5.2 Directory Broadcast Requests
Clients may request that the server transmit directory broadcasts for files
with UPLOAD_TIME within a certain range. If entries from several ranges are
required, multiple ranges can be included in a single request.
A.5.2.1 Request Frame
A directory broadcast request frame consists of header and data. The header is
fixed length, and the data portion is variable length.
The directory broadcast request frame is wholly contained within a standard
AX.25 <UI> frame. The total length of the request frame may not exceed the
length of a legal AX.25 <UI> frame.
A <UI> frame containing a request frame uses the same PID as a directory
broadcast frame: 0xbd. The source address is the address of the transmitting
station, the destination address is the address of the server to which the
request is directed.
A.5.2.2 Request Header
The format of the directory broadcast request header is:
struct DIR_REQ_HEADER{
unsigned char flags;
int block_size;
}
flags - A bit field as follows:
7 6 5 4 3 2 1
/-------------------\
|* * 1 V V C C|
\-------------------/
CC Two bit field as follows:
00 directory fill request.
01 reserved
10 reserved
11 reserved
VV Two bit version identifier. This version is 00.
1 Always 1 indicates a client-generated frame.
* Reserved, must be 0.
<block_size> is the number of data bytes which the server should place in each
directory broadcast frame.
5.2.3 Data
If the CC field is 00, a list of UPLOAD_TIME ranges is present.
The format of the list is pairs times (number of seconds since 00:00:00
1/1/80.)
struct PAIR {
time_t start
time_t end
};
For each PAIR, the server will transmit directories for those files with
UPLOAD_TIME greater than or equal to <start> and less than or equal to <end>.
If there are no files within the range, the directory for the first file with
upload time greater than <end> will be transmitted. If there is no such file,
the directory for this first file with upload time less than <start> will be
transmitted. In either case, the <t_old> and <t_new> fields in this directory
will indicate to the client that there are no entries between <start> and
<end>.
