* * * * *

                          Eight years of greylisting

I've been hacking on my greylist daemon [1] over the past few days. I'm not
sure what, exactly, prompted me to start hacking away at it though. The last
code change was in December of 2011 [2]—all code changes since then have been
tweaks to the Makefile (the file that describes how to build the program). As
I'm hacking on it, I've come to hate the code handing the protocol the
components use for communications (there's the main component that manages
the data and logic; there's the component that interfaces with sendmail [3]
and another one that interfaces with postfix [4]).

And over the past few days, I've reflected over what I would do differently
if I were to write the greylist daemon now and how well my decisions eight
years ago held up.

One decision I made eight years ago was to write my own “key/value” store
instead of using a pre-existing one. I rejected outright the use of an SQL
(Structured Query Language) database engine (like MySQL [5] or PostgreSQL
[6]) and I don't think I would change my mind now. The data stored is short
lived (six hours for most entries, otherwise thirty-six days) and I don't
think such churn is good for database engines.

In addition, the only NoSQL [7] based solution (as they're now called) at the
time was memcached [8] (written in 2003; redis [9] wasn't released until
2009, two years after I released the greylist daemon). memcached (and redis)
can expire entries automatically, and it could handle five out of the six
lookups the greylist daemon makes. The one lookup neither one can handle (as
far as I can tell) is the IP (Internet Protocol) address lookup.

This lookup compares the IP address of the sending SMTP (Simple Mail
Transport Protocol) server against a list. The list describes an address
range [10] and what to do if the given IP address “matches.” For example:

>          0.0.0.0/0 GREYLIST
>  205.211.164.50/32 ACCEPT
>    206.214.64.0/19 REJECT
>    207.115.11.0/26 ACCEPT
>

If, say, the IP address is 207.115.11.8, then the email is accepted and
further processing is skipped because of the matching rule:

>    207.115.11.0/26 ACCEPT
>

An IP address of 206.214.64.10 is rejected, because of the matching rule:

>    206.214.64.0/19 REJECT
>

An address like 66.252.224.242 will match the rule

>          0.0.0.0/0 GREYLIST
>

and because the result is GREYLIST, futher checks are made.

There does not appear to be a way of handling this type of query using
memcached or redis. I would have to write code to store the IP addresses
anyway. Also, memcached is a pure memory cache—if it crashes, all the data
goes away (and remember—at the time I wrote this, this was really the only
key/value store that existed that wasn't an SQL database engine) which is
something I didn't want to happen. So my decision at the time to write my own
key/value store wasn't a bad one.

Today?

Today I might consider using redis to store what I could, but it's another
component that, if it isn't available, I can't greylist incoming email (I
have to allow the email in—fail safe and all that). Also, the code I wrote to
store the non-IP address data was easy to write. I dunno. It's hard to say
how I would store the data today.

The protocol between my components is something I would handle completely
differently today. I can't say what the actual protocol would be though.

There are basically two methods of sending data—a series of values in a fixed
order (which is how the protocol works today) or as a series of tagged
values, which can appear in any order. The former doesn't really deal well
with optional data (you end up tagging such values anyway) while the later is
harder to parse (since the values aren't in a fixed order, you have to deal
with missing values in addition to duplicate values).

The biggest issue I have with the protocol now is what I said above, the code
that handles the protocol is a mess—it's all over the place instead of in a
few isolated routines. That makes updating the protocol (say, adding new
fields, fixed or optional) very difficult.

What I would do now is make the protocol handing portion more of a module—a
module for version 1.0 of the protocol, a module for version 1.1 of the
protocol, etc., load them all up, and based upon the version embedded in the
packet (something I do have, by the way), farm out the processing to the
proper protocol version module. It would make updating the protocol easier to
deal with in the codebase. The lack of this approach to the protocol is, I
think, the biggest problem with the codebase today.

One last aspect I would change is the logging of verious statistics, or “key
performance indicators [11]” as they are called in the industry. Instead of
incrementing a bunch of variables in the codebase and every so often dumping
them out to syslog [12] (messy code requiring the use of signals and all the
problems that entail [13], and several lines of code modified for every new
KPI (Key Performance Indicator) added) I would use the method they use at
Etsy [14]—statsd [15]—or at least, my own take on it. I don't need the full
blown “all-singing, all-dancing, all-graphing” statsd that Etsy [16]
developed but one that just logs to syslog(). And given the whole concept is
easy, a small version that just logs to syslog() is pretty trivial to write
(I wrote a version in Lua [17] with 225 lines of code, and a full quarter of
that is just parsing the command line). The nice thing about a statsd-like
concept is that it is trivial to add new KPIs to the codebase, and they're
logged automatically without any other changes. The logging and potentially
resetting of values is all isolated in statsd, in the way that log messages
are logged to files or forwarded to another server is isolated in syslog.

There's not anything else I would really modify in the greylist daemon.
Really, the only bad decision I made eight years ago was not fully isolating
the protocol. Everything else was an okay decision.

And frankly, I'm not even sure if the greylist daemon needs any more work
done on it.

[1] http://www.x-grey.com/
[2] gopher://gopher.conman.org/0Phlog:2011/12/04.1
[3] https://www.sendmail.com/sm/open_source/
[4] http://www.postfix.org/
[5] https://www.mysql.com/
[6] http://www.postgresql.org/
[7] http://en.wikipedia.org/wiki/NoSQL
[8] http://en.wikipedia.org/wiki/Memcached
[9] http://redis.io/
[10] http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notation
[11] http://en.wikipedia.org/wiki/Performance_indicator
[12] http://en.wikipedia.org/wiki/Syslog
[13] gopher://gopher.conman.org/0Phlog:2007/10/18.1
[14] https://codeascraft.com/2011/02/15/measure-anything-measure-everything/
[15] https://github.com/etsy/statsd
[16] http://www.etsy.com/
[17] http://www.lua.org/

Email author at [email protected]

You are viewing proxied material from gopher.conman.org. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.