Subj : ET phone home
To : Nick Andre
From : mark lewis
Date : Sun Jan 08 2017 11:44 am
On 2017 Jan 08 10:30:12, you wrote to me:
NA> On 08 Jan 17 09:16:24, Mark Lewis said the following to Nick Andre:
ML>> exactly sure on the details but i know that they greatly influenced me
ML>> to my software place the MSGID as close to the beginning of the control
ML>> lines possible so that db would not detect messages posted within one
ML>> second as dupes... this was especially important when testing at 100+
ML>> posts per second... are you willing to share information about how db
ML>> does its dupe detection so others can understand more? please?
NA> Its not that hard to understand. A CRC is computed from the header and
date
NA> of the tossed message. I would have to dig into the code and I'm not sure
NA> how many bytes are being included from the start of the message.
yeah, these little details are what is/was being sought... if the header and X
bytes are being read into a buffer and then the CRC calculated on that entire
raw buffer or if each field is read individually and then fed to the CRC
calculator... knowing this may help others who are trying to work out how to do
dupe checking that doesn't rely on MSGID alone... that because messages without
MSGID can'tbe checked that way so an alternative or three is desired/needed...
NA> Each Echomail area has a cache database file. In the case of *.MSG,
NA> this is called DBRIDGE.DUP and resides in each area and for
NA> Hudson/QBBS there is one database segmented slightly different. The
NA> CRC's are kept in there. I believe the code sets the cache database
NA> size at 1,024 entries.
i remember the different dup cache files... i didn't know they were limited to
a paltry 1024 entries, though... i never dug that deep ;)
NA> Interestingly it appears that there is a "reputation" method for the
NA> cache database. It appears as it is loaded into RAM during a toss, any
NA> time a CRC match is encountered, that CRC is pushed up the cache
NA> table, while CRC's of legitimate messages end up being pushed down.
NA> The CRC table is saved into that cache file every time the Echomail
NA> area changes in the toss cycle; or there are no more packets to toss.
that's pretty interesting... i guess that's so that messages with more dupes
can be detected faster with their CRCs at the top of the queue... interesting
idea and i'm sure one that was important back in the day of slower machines :)
)\/(ark
Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it
wrong...
... Actually, if they leak, you've pumped them too many times.
---
* Origin: (1:3634/12.73)
