TITLE: Gmail on macOS command line: neomutt + offlineimap + notmuch
+ pass + vim + launchd (brew services) + w3m
DATE: 2018-07-15
AUTHOR: John L. Godlee
====================================================================


I've been experimenting with a new email client on the command
line. I used alpine for a long time, but I've heard a lot about
mutt as well. It seems to be the standard terminal email program,
but my past attempts to set it up have always failed, so I set
aside a couple of hours over the last few days to really get into
the subject. This is the setup I have so far, which uses the
neomutt fork. It's an amalgamation of lots of different guides and
snippets that I've found scattered around on the internet. I've
referenced the important ones at the end.

y setup is fairly particular to me and my needs. I use macOS rather
than Linux which produces some interesting idiosyncrasies, and I
wanted to integrate my existing command line programs where
possible, notably vim (text editor) and pass (GPG enabled password
manager), and w3m (web browser). I also use Gmail, so interacting
with that requires understanding their slightly odd IMAP
configuration.

Offlineimap

For now we won't even install neomutt, there isn't anything to see
yet anyway and much of the config won't make sense until we've set
up the back end stuff. The first thing to set up is the syncing of
emails with Gmail's IMAP system.

neomutt can handle IMAP email, but grabbing from a remote server to
read an email requires that there be an internet connection
everytime you open neomutt. To keep an offline database of my Gmail
account, so I can search it when travelling, I use offlineimap,
which can also be installed using Homebrew:



 [Homebrew]: https://brew.sh/

   brew install offlineimap

Steve Losh

offlineimap takes its configuration from ~/.offlineimaprc. This is
my config:

   general
   ui = TTY.TTYUI
   accounts = [email protected]
   pythonfile = ~/.offlineimap.py
   fsync = False
   ssl = False

   Account [email protected]
   localrepository = johngodlee-local
   remoterepository = johngodlee-remote

   Repository johngodlee-local
   type = Maildir
   localfolders = ~/.mail/[email protected]
   nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
       'sent': '[Google Mail]/Sent Mail',
       'flagged': '[Google Mail]/starred',
       'trash': '[Google Mail]/Bin',
       }.get(folder, folder)

   Repository johngodlee-remote
   maxconnections = 3
   type = Gmail
   remoteuser = [email protected]
   remotepasseval = get_pass()
   realdelete = no
   ssl=true
   sslcacertfile = /usr/local/etc/openssl/cert.pem
   nametrans = lambda folder: {'[Google Mail]/Drafts':    'drafts',
       '[Google Mail]/Sent Mail': 'sent',
       '[Google Mail]/Starred':   'starred',
       '[Google Mail]/Bin':     'trash',
       }.get(folder, folder)

   folderfilter = lambda folder: folder not in ['[Google
Mail]/Bin',
       '[Google Mail]/Important',
       '[Google Mail]/Spam',
       '[Google Mail]/Chats',
       '[Google Mail]/All Mail',
       ]

There is a lot there, so here is a line by line breakdown:

ui = TTY.TTYUI signals which user interface to use. TTY.TTYUI makes
sure that the process runs quietly in the background, there are
also other options which may be better, but this one works for me.

accounts = [email protected] is the email account to be checked.

pythonfile = ~/.offlineimap.py is a python script which offlineimap
calls to get the password for my email account from pass. I'll go
through .offlineimap.py later.

fsync = False just tells offlineimap that it doesn't need to make
sure a full sync is completed every time, if a few emails get lost,
no bother, they will just get synced next time

   Account [email protected]
   localrepository = johngodlee-local
   remoterepository = johngodlee-remote

The section above defines the names of repositories to use for this
email account.

   Repository johngodlee-local
   type = Maildir
   localfolders = ~/.mail/[email protected]

The section above defines the location and mailbox type of the
local repository for the email from my account

nametrans = lambda folder: {'drafts': '[Google Mail]/Drafts',
..}.get(folder, folder) translates Gmail's IMAP folders into
folders in my local mail directory. I think this is necessary as
Gmail's folders contain forward slashes, which would mess up a
normal file system. I only translated the system IMAP folders, as
custom folders (labels) don't use the [Google Mail]/ prefix and are
synced automatically as is. I had some trouble getting offlineimap
to recognise my system folders, as many other guides recommend
using the [Gmail]/ prefix, but for whatever reason my account uses
[Google Mail]/.

   Repository johngodlee-remote
   maxconnections = 3
   type = Gmail
   remoteuser = [email protected]
   remotepasseval = get_pass()
   realdelete = no
   ssl=true
   sslcacertfile = /usr/local/etc/openssl/cert.pem

This section configures the remote repository and how to interact
with it.

maxconnections = 3 defines the number of parallel connections
offlineimap can make when syncing emails. 3 is low enough that
Gmail won't enforce rate limits and break the connection.

type = Gmail tells offlineimap that the account is a Gmail account,
and so it should take that into account when dealing with their
weird IMAP setup.

remoteuser = [email protected] simply tells us what account is
being accessed.

remotepasseval = get_pass() tells offlineimap where to get the
password for the account. get_pass() is the python function that is
created in ~/.offlineimap.py

realdelete = no tells offlineimap not to totally delete an email
when you press delete in neomutt, instead it will keep it in All
Mail.

ssl = true says to always use ssl encryption when syncing

sslcacertfile = /usr/local/etc/openssl/cert.pem gives the location
of a security certificate, which I think helps ssl to further
prevent man in the middle attacks when syncing.

nametrans = lambda folder: {'[Google Mail]/Drafts':    'drafts',
..}.get(folder, folder) is just like the nametrans function
earlier, only it goes in the other direction.

folderfilter = lambda folder: folder not in ['[Google Mail]/Bin',
..] gives a list of folders that should not be synced. In my case,
I don't want to sync the trash (Bin because British), Spam, Chats,
and All Mail.

Later on I'll look at how to get offlineimap to run in the
background using launchd, a macOS alternative to crontab.

Pass and .offlineimap.py

   #!/usr/bin/env python

   from subprocess import check_output

   def get_pass():
     return check_output("/usr/local/bin/pass
email/[email protected]", shell=True).splitlines()[0]

This script looks in pass for the entry email/[email protected],
which contains the password for my email account, and then takes
the first line (splitlines()[0]) of that entry to store it in
get_pass(). This function is then called by remotepasseval in
offlineimaprc. The script should be saved as ~/.offlineimap.py.

Other programs

notmuch

notmuch provides very fast email searching. It has other
capabilities such as tagging and indexing email, but I use Gmail's
IMAP folders (labels) for this, so I only use it for searching.
notmuch can be installed via Homebrew:

   brew install notmuch

When installed, neomutt allows searching using notmuch by defining
a custom macro. You can then use all the notmuch search syntax to
quickly find emails within neomutt. I'll go through that in the
next section where I define the muttrc.

w3m

Neomutt doesn't know how to display emails that are encoded as HTML
only. To render HTML as something legible in the terminal I use
w3m. This is a text based web browser. Once again, this can be
installed using Homebrew:

   brew install w3m

To tell neomutt to forward HTML emails to w3m for plain text
encoding, then send it back to neomutt, we need to create a file
called .mailcap. Keep it in the home directory as convention:

   touch ~/.mailcap

Ten add the following snippet to the file:

   text/html; w3m -dump -o document_charset=%{charset} '%s';
nametemplate=%s.html; copiousoutput

mailcap is then referenced later in the muttrc.

Neomutt

Now that we've set up all the background, we can install the
central part of the workflow, mutt. Specifically, I'm using
neomutt, which is a fork of the original mutt which incorporates
some of the most widely used plugins.

neomutt can be installed using Homebrew:

   brew install neomutt

Then we need to make a configuration file, I like to keep mine in
~/.mutt/muttrc but see man neomutt for more locations where neomutt
will look for a config file:

   mkdir ~/.mutt

   touch ~/.mutt/muttrc

The first thing to add to the configuration is some IMAP settings:

   set imap_user = "[email protected]"

   ## Call pass from within
   set my_pass = "`pass email/[email protected]`"
   set imap_pass = $my_pass

   set from = "[email protected]"
   set realname = "John Godlee"

   set folder = "~/.mail/[email protected]"
   set spoolfile = +INBOX
   set postponed = +drafts
   set record = +sent

   ## Don't automatically move messages after reading
   set move = no

   ## Max time mutt should wait before polling IMAP connections,
in minutes
   set imap_keepalive = 30

   ## Set smtp URL for replying
   set smtp_url = "smtp://[email protected]:587/"

   ## Set the login method for smtp replying
   set smtp_pass = $my_pass
   set smtp_authenticators = 'login'

   ## Allow mutt to open new imap connections to test for new mail
   unset imap_passive

   ## Set cache locations so IMAP polling is quicker on startup
   set header_cache = ~/.mutt/johngodlee/headers
   set message_cachedir = ~/.mutt/johngodlee/bodies
   set certificate_file = ~/.mutt/certificates

   ## If stuck in a prompt, abort after n minutes to check IMAP
(timeout), n minutes between checks (mail_check)
   set timeout = 10
   set mail_check = 5

Some of this is self explanatory, so I'll only go through the
important bits

set my_pass ... calls pass to create the variable my_pass which is
then sent to imap_pass to set the password for my Gmail account,
which is needed for sending mail.

set folder ... sets the location of the mailbox which offlineimap
syncs email to. The value of folder also acts as the prefix for set
spoolfile, set postponed and set record, as indicated by putting +
before the folder names.

set spoolfile ... should be the location of your inbox, where new
mail arrives

set postponed ... is where unfinished new emails are kept if they
are aborted before being sent

set record ... should be a folder where a copy of sent messages are
kept

Note that set smtp_pass ... also calls the my_pass variable

As neomutt comes with the sidebar patch, I wanted to make use of it
by displaying any folders that are synced from offlineimap:

   ## Mailboxes to display in sidebar and check regularly
   mailboxes +INBOX +'starred' +'trash' +'drafts' +'sent'
+'Archived' +'coding_club' +'diss_manuscript' +'personal' +'PhD'
+'SEOSAW_contacts' +'SEOSAW_logos' +'STEB_2018' +'urgent'
+'hemi_lens_proposal' +kew_taxonomy_course +'angola_botanic_garden'

   ## Sort sidebar folders alphabetically
   set sidebar_sort_method = name

   ## Sidebar visible by default
   set sidebar_visible = yes

   ## Don't abbreviate folders in sidebar
   set sidebar_short_path = no

I use a stripped down .vimrc for writing email, as I don't need a
lot of the plugins that I use for programming. I can tell neomutt
to write emails using vim with this alternative .vimrc
(.vimrc_alpine) by adding the following to the neomuttrc:

   set editor = 'vim -u ~/.vimrc_alpine'

Here is my .vimrc_alpine:

   set nocompatible              " be iMproved, required
   filetype off                  " required

   " enable syntax highlighting
   syntax on

   " Stop creating swp and ~ files
   set nobackup
   set noswapfile

   " Ignore case of searches
   set ignorecase

   " Don’t reset cursor to start of line when moving around
   set nostartofline

   " Preserve indentation on wrapped lines
   set breakindent

   " Disable folding in markdown
   let g:vim_markdown_folding_disabled = 1

   " Disable syntax conceal in markdown
   let g:vim_markdown_conceal = 0

   " Normal backspace behaviour
   set backspace=2

   " map A (append at end of line) to a (append in place)
   nnoremap a A

   " Move by visual lines rather than actual lines with k,j
   nnoremap k gk
   nnoremap j gj
   nnoremap gk k
   nnoremap gj j

   " Easier save and quit with ;
   noremap ;w :w<CR>
   noremap ;q :q<CR>

   " Copy and paste from `+` register for interacting with mac
clipboard
   vnoremap y "+y
   vnoremap p "+p
   nnoremap p "+gp
   vnoremap d "+d
   nnoremap dd "+dd

   " Map of modes and their codes for statusline
   let g:currentmode={
       \ 'n'  : 'N ',
       \ 'no' : 'N·Operator Pending ',
       \ 'v'  : 'V ',
       \ 'V'  : 'V·Line ',
       \ '^V' : 'V·Block ',
       \ 's'  : 'Select ',
       \ 'S'  : 'S·Line ',
       \ '^S' : 'S·Block ',
       \ 'i'  : 'I ',
       \ 'R'  : 'R ',
       \ 'Rv' : 'V·Replace ',
       \ 'c'  : 'Command ',
       \ 'cv' : 'Vim Ex ',
       \ 'ce' : 'Ex ',
       \ 'r'  : 'Prompt ',
       \ 'rm' : 'More ',
       \ 'r?' : 'Confirm ',
       \ '!'  : 'Shell ',
       \ 't'  : 'Terminal '
       \}

   " Change statusline based on colour
   function! ChangeStatuslineColor()
     if (mode() =~# '\v(n|no)')
       exe 'hi! StatusLine ctermfg=112'
     elseif (mode() =~# '\v(v|V)' || g:currentmode[mode()] ==#
'V·Block' || get(g:currentmode, mode(), '') ==# 't')
       exe 'hi! StatusLine ctermfg=172'
     elseif (mode() ==# 'i')
       exe 'hi! StatusLine ctermfg=044'
     else
       exe 'hi! StatusLine ctermfg=007'
     endif
     return ''
   endfunction

   " Make statusline always show
   set laststatus=2

   " statusline
       " left side
       set statusline=%{ChangeStatuslineColor()}   " Change colour
       set statusline+=%0*\ %{toupper(g:currentmode[mode()])}  "
Current mode
       set statusline+=\
       set statusline+=%1*%m%r " Modified and read only flags
       set statusline+=%1*%=   " Switch to right side

       " right side
       set statusline+=%0*\        "Space
       set statusline+=%0*%y   " File syntax
       set statusline+=%0*\|   " Vert-line
       set statusline+=%0*%p%% " Percentage through file
       set statusline+=%0*\|\  " Vert-line and Space
       set statusline+=%0*%l:%c    " Line and column number
       set statusline+=%0*\        " Space

   " Set colours for statusline middle section
   hi User1 ctermfg=255 ctermbg=240

   " Ragged right line breaks
   set linebreak

I want to read emails in plain text if possible, with the provided
plain text being used preferentially, and a converted HTML version
if plain text isn't available, which I can set using
alternative_order .... I also want to tell neomutt how to deal with
forced HTML emails, which is to automatically view them and also to
give the path to the .mailcap file we created earlier, which uses
w3m to parse HTML as plain text:

   # Read in plain text if possible
   alternative_order text/plain text/html
   auto_view text/html
   set mailcap_path = ~/.mailcap

   # Enforce encoding in utf8
   set send_charset="utf-8"

To allow notmuch to work I need to allow neomutt to create virtual
folders when searching and also to give the location of the mailbox
to search (set nm_default_uri ...)

I also set up a keybinding to initiate the notmuch search from
within neomutt as \\

   set virtual_spoolfile = yes
   set nm_default_uri =
"notmuch:///Users/johngodlee/.mail/[email protected]"
   macro index,pager \\  "<vfolder-from-query>"

Launchd

Other guides that I read recommended that I use crontab to schedule
offlineimap updates. But on macOS launchd is the proper way to do
things. macOS uses .plist scripts stored in ~/Library/LaunchAgents
to designate jobs, which use an XML format. Helpfully, Homebrew
services can generate .plist scripts for many programs, which can
then be amended later.

To start running Homebrew services, first see what services are
listed:

   brew services list

offlineimap should be listed in the output.

To generate the script to start the service at login:

   brew services start offlineimap

To ensure that notmuch follows behind offlineimap and updates its
database after every pull from Gmail, I added some extra commands
to the .plist file which Homebrew services generated:

   vim ~/Library/LaunchAgents/homebrew.mxcl.offlineimap.plist

Then amend the <array> section so it looks like this:

       <array>

<string>/usr/local/opt/offlineimap/bin/offlineimap</string>
         <string>-u</string>
         <string>quiet</string>
         <string>;</string>
         <string>notmuch</string>
         <string>new</string>
       </array>

This runs a notmuch new everytime offlineimap runs. The other flags
-u and quiet tell offlineimap not to print anything to the terminal
which could disrupt the job running in the background.

Guides I used

With all this setup you should be able to run offlineimap, then
open neomutt and start reading and writing email, though it's
entirely possible that I may have missed something. In any case,
here are the guides that I learnt from and used to write this setup:

-   https://wiki.archlinux.org/index.php/mutt
-   http://stevelosh.com/blog/2012/10/the-homely-mutt/
-   https://smalldata.tech/blog/2016/09/10/gmail-with-mutt
-   https://gist.github.com/amandabee/cf7faad0a6f2afc485ee
-   https://github.com/cbracken/mutt
-
https://www.farces.com/wikis/naked-server/homebrew/brew-services/
-   https://bbs.archlinux.org/viewtopic.php?id=142377
-   https://pbrisbin.com/posts/mutt_gmail_offlineimap/
-
https://baptiste-wicht.com/posts/2014/07/a-mutt-journey-my-mutt-conf
iguration.html

y full setup can be explored in my dotfiles

 [dotfiles]: https://github.com/johngodlee/dotfiles

 ![Mutt
screenshot](https://johngodlee.xyz/img_full/mutt/neomutt.png)

You are viewing proxied material from republic.circumlunar.space. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.