Using Gmail/IMAP Backups for Super Fast Email

Anyone who uses email often enough at some point in life will eventually struggle with handling an ever growing collection of email. Email bankruptcy is a very real prospect. My email address is literally a blackhole where lots of messages go to die. Thanks to the “all you can eat buffet” approach to email storage pioneered by Gmail, I never worried once about managing the crazy collection of text. The best features of Gmail are massive storage, filters, search, and labels/folders all in the cloud.

However, ask yourself what happens to your email when things go bad? With Google Reader shutting down, it can’t hurt to have a backup plan. What’s more since your Gmail is tightly integrated with everything else Google owns, any service violation may put all your data at risk. Although unlikely, there are people who got their accounts shutdown due to mixups of one kind or another. Google’s customer support is notoriously slow not to mention the challenge of finding a channel to communicate with them.

The advantages of having a copy of your email on your hard drive not only gives you peace of mind but can also make you more productive when combined with old school email clients like Mutt or Alpine. Marissa Mayer, the CEO of Yahoo, plows through her email with Pine.

Having your email offline allows you to take your email with you on your laptop without worrying about a connection. Read and respond on the train or in a cafe and when you get a connection it’ll all sync up. Another advantage is the search. You can combine it with local search engines to find the emails you need fast or create “smart lists” for email that meets a certain criteria to stay on top of things.

Handling your email in a no frills environment can really help you with staying focused. As we all know, using the web interface of anything can lead us to lots of distractions.

Getting Your IMAP Offline

The great thing about IMAP is that all your folders and messages reside on the server/cloud. This means that no matter what client you set up, everything will be sorted nice and neatly. However, with the typical setup, you need to have a persistent connection since messages are more or less a temporary cache.

So, to get around this, there are several solutions out there to get your email into a Maildir. With a Maildir, basically all the folders on the IMAP server become a folder on your disk and each message resides in its own text file with a unique filename and various flags indicating the state (unread, starred, etc.). The most popular of these is OfflineIMAP.  OfflineIMAP is written in Python and will pretty much run on any platform. There are other delivery agents as well. I’ve mostly used OfflineIMAP until now but it does have some issues. My biggest problem with OfflineIMAP is that near the tail end of syncing my email, it just becomes this monster of a process that eats an endless supply of RAM (I’m talking GBs and some CPU spikes). It would be dangerous to leave it running as I’d have to restart my mac due to having no RAM. It was just getting crazy.

Recently, I migrated to mbsync and I couldn’t be happier. It’s fast and efficient but consumes so little memory since it’s written in C and pretty sound in terms of design. Mbsync also offers a lot of granular controls so you can give certain folders higher priority and also perform particular operations like “pull new” messages. You can simply do “brew install isync” if you use homebrew on the mac or download the source here.

Once you get the program installed you need to configure it. I mostly followed the instructions here, the section on getting the security certificates and ignoring Gmail’s “All Mail” folder (which creates a lot of duplicates) was particularly helpful. This guide is really good for setting up channels so you can specify how you want to sync specific folders.

Channel gmail-inbox
Master :gmail-imap:INBOX
Slave :gmail-local:INBOX
Create Slave
Expunge Slave
Sync Pull
Channel gmail-sent
# we need the double quotes
Master ":gmail-imap:[Gmail]/Sent Mail"
Slave :gmail-local:sent-mail
Create Slave
Expunge Slave
Sync Pull

This can be done for any directory. Mbsync seems to go through all the folders in order so if there are particular folders you want to always have priority, setting up these channels will ensure that quick update is simply a command away.

mbsync important

You can setup a separate cron job since mbsync can have multiple processes accessing the same account. Incidentally with OfflineIMAP, the lock on an email account is global. This combined with excessive resource consumption and the fact that syncs never complete, meant I had to kill the process and start from the top (combined with the auto refresh, it was a bit of a nightmare).

Once you have the syncing setup (syncing will still take time depending on the size of your mailbox), all you need is a good software program to handle your email. You’ll be able to fly through the messages.

Personally, I use Emacs with mu/mu4e which is an awesome combination. Mu uses Xapian for fast and capable search capabilities which can be used like smart folders. There are several combinations out there for using Maildir and IMAP. Old school email clients might seem scary at first but once you get used to them, it’s fast and efficient.