----------------------------------------
Email Setup with NotMuch
January 07th, 2021
----------------------------------------
A while back I had read a post by solderpunk describing
their progress toward an "offline first" computing setup. At
the time I was struggling to focus on writing my
dissertation and thought a similar strategy would benefit me
also. Well, things have changed, and I am now rarely behind
the keyboard of my computer (less than I'd like in fact).
I've come to rely on an old android phone running LineageOS
with Termux on it. I've started to move my essential
documents and tools to my Raspberry Pi so that I can access
them from my phone and computer without having to worry
about syncing files, etc. This way I can do minor work while
my sleepless children lay on and next to me at night time. A
big part of this minor work is really just email.
gemini://gemini.circumlunar.space/users/gemlog/progress-
toward-offline-first.gmi
What actually made me want to change up my email setup was a
desire to manage my personal emails in mutt locally. Since I
started frequenting pubnixes and tildes last year, I got
really accustomed to mutt, and the Maildir mail system in
particular. I like how snappy it is--there's not loading
times, timeouts, disconnects. These aren't huge issues,
but the pubnix world had spoiled me.
So for some time I had been using aerc on my workstation,
and thoroughly enjoyed it. It is a very powerful,
customizable, easy-to-use and configure client. The only
problem I ever really had with it was that it would
disconnect from the IMAP server after my computer went to
sleep, so I would have to close and reopen it. It's not a
big deal, but if I forgot to do this then there would be
times I'd try to send an email only to fail and have it lost
forever! These experiences further motivated me to try out a
Maildir type solution using sendmail queues a la
solderpunk's offline first setup. This writeup is basically
a chronicle of the rabbit hole that process became.
My first step was to setup getmail. I think this entire
process would not have happend the way it did (with me
getting into notmuch) if I had first tried OfflineIMAP
instead. The reason is that (apparently) OfflineIMAP will
sort your Maildirs according to how you have folders setup
on the IMAP server. I just assumed this is how getmail would
work, but it does not! A common solution is to use an MDA
like procmail to sort the mail getmail "gets." I thought at
the time that procmail would be tedious to setup and require
more effort than I was willing to put in. But here I am now
after having invested much more time into messing around
with notmuch (which I don't mean negatively).
I initially decided to look into notmuch because I knew that
aerc had built-in support for it. I was familiar with using
and configuring aerc, so I figured I'd give it a shot.
Setting up notmuch with aerc is super easy. All you have to
do is initialize a notmuch databse in the root folder of
your maildir directory (by telling notmuch the path in the
notmuch config file), do some initial tagging with notmuch
(I'll get to that in a sec), then in the aerc/accounts.conf
file, just do something like the following:
source = notmuch://home/mieum/mail
query_map = /home/mieum/.config/aerc/notmuch-tags
The notmuch-tags file is just a list of virtual "folders,"
which are arbitrary notmuch queries. For example:
Inbox=tag:inbox and not tag:killed and not tag:archived
Sent=tag:sent
Gemini=tag:gemini and not tag:killed and not
tag:archived
In order for this to work, you need to actually tag your
email with notmuch first. I followed a few tutorials around
and ended up with something that works for me. The first
thing to do is to run `notmuch setup` and give it all the
info it needs, such as where the databse is to be located,
your email addresses, etc. One field that should be changed
in the config file is the [new] section:
[new] tags:new
This will ensure that when you run `notmuch new` notmuch
will tag all the new mails with the tag "new," rather than
automatically marking them with the tags "unread" and
"inbox." This simplifies the tagging process, because you
don't necessarily want everything to be put into your inbox,
like sent mails, spam, archives, etc., and it's much easier
to add the "inbox" tag after you have applied your own
filters (such as those for spam and sent mail) and removed
the "new" tag from them.
So if you run `notmuch new` without any further
configuration, it will just tag all of your email as "new."
A good way to automate the tagging process is to setup
pre-new and post-new hooks. These are executable shell
scripts saved to the folder "hooks" within the notmuch
databse directory.
In my pre-new hook, I invoke getmail to retrieve emails from
my different accounts and also remove some tags from emails
I've archived or killed. It looks something like this:
#!/bin/sh
notmuch tag -inbox -unread tag:archive or tag:kiled
getmail --rcfile <account one>
getmail --rcfile <account two>
You could also purge your trash here, but I choose to do
that manually, because I sometimes need to recover something
I've erroneously tagged as trash or spam, etc. In the
post-new hook is where all the notmuch action happens. I
have it setup to read from a few "tagfiles" in order to keep
things neat. That file looks something like thius:
#!/bin/sh
# tag all messages from me as sent and remove "new' tag
notmuch tag --input=/home/mieum/mail/.notmuch/schemes
/me.notmuch
# tag mail from different accounts
notmuch tag --input=/home/mieum/mail/.motmuch/shcemes
/folders.notmuch
# tag mailinglists
notmuch tag --input=/home/mieum/mail/.notmuch/schemes
/lists.notmuch
#put all that's leftover into inbox and tag it "unread"
notmuch tag -new +inbox +unread -- tag:new
What happens here is that notmuch first marks all mail from
my accounts as "sent," and keeps them out of the inbox. Then
it moves on the the taglist "folder.notmuch." All this does
is apply tags to all emails within an individual maildir
(which contains all the emails for an individual account).
To do this, it is necessary to use the search term "path"
instead of "folder," otherwise subdirectories will not be
searched. For example:
+disroot -- path:/disroot/**
Finally, the post-new hook removes the "new" tag from all
leftoever emails and marks them unread in the inbox. There's
really not much to it. So now in aerc, you can create
virtual folders based on arbitrary queries including any
combination of these tags. So that's what I did, and it
worked great! It was fast and snappy and very easy to modify
the tags within the client. Aerc's support for notmuch is
slightly limited, however. Managing tags, filtering, and
searching work great, but they only operate on selected
messages or messages within the current view. So the search
command in aerc will not search the entire notmuch database,
but just those messages in the current "folder." This isn't
that big of a deal, since you can make the same queires in a
terminal (even one within aerc itself).
One problem I encountered with aerc is that it was not very
mobile friendly, at least at first. Aerc has a nice sidebar
which lists the folders of the current account, but on
mobile this takes up too much real estate. So I decided to
make all my folder names just single Chinese characters that
describe their content, that way the sidebar could be just 2
or 3 columns wide:
namu
保 12-15 Zach DeCook Re: CCEL.org books in gem
冊 12-14 Zach DeCook CCEL.org books in gemini
受 12-03 John Cowan Fwd: Gutengemi project
廢 11-02 Drew DeVault Re: The brick and mortar
校 11-02 Arav K. Re: The brick and mortar
爲 11-01 Oliver Leaver-Smi Tanelorn City (existing pr
軌 10-31 Callum Brown Re: The brick and mortar
送 10-31 Arav K. Re: The brick and mortar
雙 10-31 Callum Brown Re: The brick and mortar
It looks better with color, but I was happy with how
functional this is on a narrow mobile screen.
My plan with aerc was to have different query_maps for
different mail accounts, that way I could use tabs in aerc
to organize the various folders/tags of respective accounts.
But it does not work this way, understandably. Aerc will
make just a single connection to the notmuch database, so
whatver notmuch tags or queries you want to view in aerc
will have to be included as folders of a single "account."
This isn't that big of a deal, especially if you are using
notmuch to supplement your other acccounts in aerc. But it
becomes problematic if you want to rely on notmuch as your
backend, because then you have no simpele way to respond to
emails from different accounts (as far as I know). You can
setup multiple smtp accounts in aerc, so this should be
posible in theory. But at this point I was kind of jazzed on
notmuch and wanted a more robust interface with it.
Before trying some dedicated notmuch clients, I wanted to
see how well it would work with mutt. The answer is that it
works great. I actually ended up using neomutt, which is
built with notmuch support, so you don't have to worry about
patching anything or dealing with plugins. If you have a
notmuch database setup, all you really need to do to use it
with neomutt is just source a ~/.mailboxes file that lists
all the virtual folders (notmuch queries) to include. In the
muttrc or .neomuttrc there are a few other options that
make the experience a little better, but I will not get into
it here (because I didn't get too into it myself). Mutt is
nice to use with notmuch for the fact that it is capable of
altering tags of emails AND making queries to the notmuch
database itself. These queries are spawned as virtual
mailboxes you can explore like any other. In the process of
using mutt with notmuch, I found that the way emails are
marked for action (like tagging) is somewhat awkward. For
anyone who has used mutt extensively, this is not really a
problem, but having come from aerc with its intuitive
vi-style selection keybinds, it made the experience a little
uncomfortable. This, again, is not really a big deal, but it
was enough for me to wonder what other options are out there.
So ultimately I ended up giving a dedicated notmuch client a
try.
There are a few different options when it comes to
"dedicated" notmuch clients. The most popular one seems to
be an emacs plugin. There is also a vim plugin, but I didn't
try it. The version of vim that came from the Debian repos
was not built with ruby support, and I didn't feel like
fooling around with it. Plus, I thought that a vim plugin as
an interface for notmuch would have similar kinds of quirks
that would make using notmuch uncomfortable, like those I
found in neomutt and aerc.
Utlimately I ended up with alot. Design-wise alot is not
perfect, at least to my tastes or what I'm
familiar/comfortable with, but it is pretty much THE tool
for the job if you want a full-featured notmuch interface.
As an email client, it is quite nice too: encryption,
threading, etc. The standard colorscheme appeared very
strangely in my terminal, which was a turn off at first, but
after applying the "mutt" theme with minor adjustments, it
feels like home.
To use alot, you actually don't really need to do anything.
It just finds your notmuch database and allows you to
interact with it and the emails it has indexed. Setting up
your accounts to send mail is fairly simple (I ended up
using msmtp), and making other tweaks to the configuration
is as easy as adding key value pairs to the config file. It
is VERY customizable. One neat thing is that you can have
the names of tags "translated" in the UI, so you can have a
unicode key symbol displayed in place of the word
"encryption," for example. This is nice if you are using
alot on Termux where screen space is limited, because then
the list of tags is actually visible (on a narrow screen
they become hidden in favor of the subject string).
One nice feature of alot is that you can save queries as
named queries and view them later in the "namedqueries"
buffer, or bind them to a keypress. You can use this to save
favorite queries as virtual folders. Another thing about
alot that I like is it is written in Python, and since I'm
learning Python, it gives me a chance to learn by hacking on
it. In fact, in the process of using alot I've encountered
another notmuch tool called afew, also written in Python,
that is a utility for handling your notmuch tagging. I only
discovered this tool today, so I don't have much to say
about it, but I think eventually I will end up converting my
current setup to just use afew. What is cool about afew is
that it has functions for applying notmuch tags based on
regex queries of the email content and headers. This is
actualy how I found out about afew--in trying to figure out
how I could parse the headers of emails to apply a tag to
emails exchanged through DeltaChat.
Anyway, this is a verbose, but not very informative write-up
of my little adventure with notmuch email recently. Maybe I
can share some more concrete info in a follow-up post with
some examples of custom filters written in Python. One
filter I have in mind is one to filter out the mass of junk
mail I get form my university. There's no way to
unsubscribe! And the majority of emails have no content--all
the text is embedded in an image served from their servers.
I could block these accounts individually, but sometimes
there are important messages sent from them, plus all of my
colleagues use email addresses from those domains. So my
goal is to write a few filters that do some regex searches
of the sender address and some of the headers.
Another aspect to this little adventure has been backing up
all of my pubnix/tilde email to my server and indexing it.
In that process I've learned a little about ssh-agent and
rsync, and also how use procmail to deliver mail to a
maildir. So I still have lots to write about. I just felt
like I needed to write this now otherwise I would never get
around to it.
~peace
