And here I thought dating was easy …

* * * * *

And here I thought dating was easy …

I'm building on the work of indexing my filesystem [1] by indexing all of my
email. I have a ton of it spread across various directories and when ever I
have to search for something (such as the time I flamed an entire department
at FAU (Florida Atlantic University) [2] on a public mailing list—ah, those
were the days), it's a long drawn out ordeal to find it.

Initial stab at the problem is to just index a few email headers, like From:,
To: (and the related Cc:), Date: and Subject:—the primary headers one would
be interested in.

I decided to tackle one of the harder fields to process first— From:. While
the format is specified in RFC-822 (STANDARD FOR THE FORMAT OF ARPA INTERNET
TEXT MESSAGES) [3] and RFC-2822 (Internet Message Format) [4], there's still
quite a bit of variance in the format to be annoying.

I was able to squish 23 different formats into four cases:

1. email address and real name aren't delimited, in which case, the only
thing to parse is the email address;
2. email isn't delimited, but the real name is (between parentheses, or
quotes), so extract the real name from between the delimeters, and
anything that isn't delimited is the email address;
3. email is delimited (between angle brackets or square brackets), but the
real name isn't, so extract the email address, and anything that isn't
delimited is the real name;
4. both the email address and real name are delimited, so it's trivial to
extract both.

Then, I decided to parse the Date: header. Now, this is specified, quite
plainly:

> 5. DATE AND TIME SPECIFICATION
>
> 5.1. SYNTAX
>
> date-time = [ day "," ] date time ; dd mm yy
> ; hh:mm:ss zzz
>
> day = "Mon" / "Tue" / "Wed" / "Thu"
> / "Fri" / "Sat" / "Sun"
>
> date = 1*2DIGIT month 2DIGIT ; day month year
> ; e.g. 20 Jun 82
>
> month = "Jan" / "Feb" / "Mar" / "Apr"
> / "May" / "Jun" / "Jul" / "Aug"
> / "Sep" / "Oct" / "Nov" / "Dec"
>
> time = hour zone ; ANSI and Military
>
> hour = 2DIGIT ":" 2DIGIT [":" 2DIGIT]
> ; 00:00:00 - 23:59:59
>
> zone = "UT" / "GMT" ; Universal Time
> ; North American : UT
> / "EST" / "EDT" ; Eastern: - 5/ - 4
> / "CST" / "CDT" ; Central: - 6/ - 5
> / "MST" / "MDT" ; Mountain: - 7/ - 6
> / "PST" / "PDT" ; Pacific: - 8/ - 7
> / 1ALPHA ; Military: Z = UT;
> ; A:-1; (J not used)
> ; M:-12; N:+1; Y:+12
> / ( ("+" / "-") 4DIGIT ) ; Local differential
> ; hours+min. (HHMM)
>

“STANDARD FOR ARPA INTERNET TEXT MESSAGES, § 5.1 [5]”

Okay, clear if you're into such things. And from the most recent
specification:

>
> date-time = [ day-of-week "," ] date FWS time [CFWS]
>
> day-of-week = ([FWS] day-name) / obs-day-of-week
>
> day-name = "Mon" / "Tue" / "Wed" / "Thu" /
> "Fri" / "Sat" / "Sun"
>
> date = day month year
>
> year = 4*DIGIT / obs-year
>
> month = (FWS month-name FWS) / obs-month
>
> month-name = "Jan" / "Feb" / "Mar" / "Apr" /
> "May" / "Jun" / "Jul" / "Aug" /
> "Sep" / "Oct" / "Nov" / "Dec"
>
> day = ([FWS] 1*2DIGIT) / obs-day
>
> time = time-of-day FWS zone
>
> time-of-day = hour ":" minute [ ":" second ]
>
> hour = 2DIGIT / obs-hour
>
> minute = 2DIGIT / obs-minute
>
> second = 2DIGIT / obs-second
>
> zone = (( "+" / "-" ) 4DIGIT) / obs-zone
>

“RFC-2822: Internet Message Format, § 3.3 [6]”

Really, the only thing this does is mandate that the year be four digits
long, moves to a numeric-only timezone format and clarifies a bit where white
space appears, but otherwise, is pretty much the same as the older spec.

So, if I ignore the timezone for now (because the Standard C library has such
piss-poor support for it, but that's a rant for another time), the only real
issue is handling two or four digit years.

And in poking around the man pages for the various Standard C library
routines, I came across strptime(), which is the functional opposite of
strftime()—instead of converting the time to a human representation, it'll
take a human representation and convert it to a time value. It isn't a
Standard C call, but hey, why not use it for now?

And it appears that the two-digit/four-digit year isn't a problem for
strptime():

> When a century is not otherwise specified, values in the range [69,99]
> shall refer to years 1969 to 1999 inclusive, and values in the range
> [00,68] shall refer to years 2000 to 2068 inclusive; leading zeros shall be
> permitted but shall not be required.
>
> “man page for strptime()”
>

Sounds perfect!

Only it blew up when it encounted Wen, 2 Mar 2005 01:39:42 +0000.

Sigh.

Okay, make sure I start parsing past the optional day of the week. It then
blew up on Sat Mar 5 18:58:36 2005.

What the—? That's not even a standard format! And then there was Wed,19 _十二月_
2001 20:23:05 ([DELETED-I added the question marks because I can't determine
the character set that was used for the month—there's nothing in that
particular email that even hints what language it might be-DELETED] I found
out which language---Chinese. Figures).

And let's not forget 9/8/99 1:01:12 AM Pacific Daylight Time (lovely) or Fri
Jun 28 10:07:44 PDT 2002 or even Wed 8-Jan-2003 08:24:20.

Oh, and we mustn't forget Tue, 23 May 100 22:18:56 -0400.

Double sigh.

I found it amazing—one of the more strictly defined fields in an email and
yet there still was an amazing amount of garbage to be found (although to be
fair, these anomalies account for less than one per cent of all the emails
scanned, but when you have thousands of emails, it can still add up).

(And one more interesting note—I did not see one email use the military time
zone format.)

[1] gopher://gopher.conman.org/0Phlog:2009/05/07.1
[2] http://www.fau.edu/
[3] http://www.ietf.org/rfc/rfc0822.txt
[4] http://www.ietf.org/rfc/rfc2822.txt
[5] http://www.ietf.org/rfc/rfc0822.txt
[6] http://www.ietf.org/rfc/rfc2822.txt

Email author at [email protected]