Subj : *.MSG vs. packed help?
To : All
From : Mike Luther
Date : Sun Jul 08 2001 07:54 am
Looking for suggestions here.
Long ago and far away I gave a small amount of money for a DOS database
analysis tool called askSam. The askSam, at least the professional version,is
written totally in assembler and is VERY fast and good. Later on the askSam
utility was re-written or Win 3.1 and has, I think, also been ported to WIN-9x,
but not OS/2. As well, it was merged into a full CGI script driven Web page
creation and hosting tool that uses the core capabilities of the askSam engine
as well.
What's neat about askSam? Well, for one thing, you just read the whole mess
for any old text into the database. Then *AFTER* you have any database
created, you can create fields, manipulate all the fields, and whatever,without
ever messing with the actual database! Intriquing indeed. The tool has, so
said, a fair nitch market in law enforcement, for munching in huge piles of
information. Then you go in and research patterns in the information which
will not, perhaps, be apparent as to how you can prove how this or that record
is related to this or that other record until askSam goes through all the
needles in the haystack and shows you.
The US DEA broke the Manuel Noriega case with it, for example. They crunched
in all the surveilance traffic and used it to ferret out who talked to whom and
about what over megabytes of text .. ;)
In essence, for about $65 back then on special, I got, all those years back,a
full text search engine for all the Fido traffic as well. Preposterous? Not at
all! For a smaller FidoNet net like Net 117 here, all the local traffic for
the last ten years isn't any more than about 36MB or so! That's because the
engine is a full relational database engine!
What I've been doing all these years is simple -- as long as I use a *.MSG
format. All I do is take the inbound messages, and export them into a pure
ASCI text base dupe file! Then I call askSam as a command line input function.
I then punch in the entire message .. seen by's .. origin lines,the whole 9
yards into the master database. From that point on I can tell you, for
example, the identity of every node that ever used this or that word of
profanity .. or whatever. E.v.e.r.y W.o.r.d.
Interesting. Blows the Hades out of Fido elections and complaints at times.
Anyway.
Problem is that as I and a few folks have gotten more interested in other
things and higher traffic volume echos, changes are necessary. It matters not
what BBS system is involved, the pricipal is the same. One of the reasons we
all moved away from *.MSG format is because it takes too long to scan and
manipulate the message base in that format as the number off message areas and
messages in them grows up!
I've reached that point now where the system still works fine, but the
maintenace operations for the utilties take too long in the *.MSG format for
the system.
What do I do now?
I need a *.MSG format for the inbound traffic, keyboard or toss produced, it
doesn't matter. I need that, or at least I think I do, to export that into the
askSam ghoul and utilities I hand wrote to convert the *.MSG format to what I
want for Uncle Sambo!
It doesn't really matter if the results are not done in synchronization with
the BBS system or not. In-as-much as OS/2 is threaded, if I just have the pile
of files to smunch into the ghoul, that part of the process can go on
irrespective of whether or not the BBS has been returned to service.
But what is missing, if I go to, for example SQUISH and a SQUISH message base,
is the fact that in the toss process, we don't get a new cap and a new stack of
*.MSG's in a directory to clue the askSam thread task, to "Aha! traffic! Munch
in all from the old cap to the new one!" That's how I wrote the interface code
hard coded untility which is called now to do the task I wrote ass MSG2ASK.EXE
all these years ago.
At 36 MB of file askSam isn't even cooking yet. It can handle a 2 GB ull
relational database and 4 GB under some file systems! Split across an OS/2 LVM
oriented drive, who knows where the limit is! I suspect it's actually larger
still, but at the time it was written, getting to 4GB was a feat!
Advice?
Suggestions?
Mike @ 1:117/3001
--- Maximus/2 3.01
* Origin: Ziplog Public Port (1:117/3001)
