1.) Each new entry should be added to the end of a location's data file.
Do not try to insert data into these huge files!
2.) An index of each datafile's contents should be created and maintained in
memory, such that the location in the file of an entry can be quickly located.
This index would have the seek location as well as the size and date for each
entry. The date is important because entries are distributed in an order
based upon their size and date. The smallness and oldness of a day's entry
are used to calculate its priority of transmission.
3.) The index may wish to use an abbreviated hash, with an accompanying table
for those few entries that happen to have identical abbreviated hashes. This
accompanying table will have the full hashes and can also be used for analysis
and tuning of the abbreviated hash length.
Hosts that are on a budget may decide that they only want data from the last
day or two. All systems should consider their RAM and storage space to
ensure the proper management thereof.
