home

Running a private mail server for six years, easy peasy

May 2021

TL;DR - High-level overview of running my own, small private Linux mail server since late 2015. I’ve encountered surprisingly few issues and many valuable learnings. Initial setup (including monitoring, backups, configuration management) has taken some time, but recurring maintenance since then has been an estimated 10 - 20 minutes per month. Worth it for me, but probably not for most people. The next best thing, in my opinion, is mailbox.org with one’s own domain.

Discussion on Hackernews

Motivation

The main reason for this writeup is meant as a response to the sentiment I keep reading about in IT forums, that it’s “very time demanding”, “impossible to maintain”, “a pain to make sure your mails are being delivered”. I understand the reasons, and tend to agree when it comes to large-ish selfhosted mail deployments with hundreds of users and tens of thousands of mails per day, which also happens to be part of my current day job. It’s true that many IT people understandably don’t want to invest private time into things which appears to be another kind of work assignment. But personally, it fills me with satisfaction to self-host my own infrastructure, my little internet island where I’m root, especially in times of mega corporations trying (and succeeding) in redefining “the internet” as a portfolio of services only they can offer, with little alternative.

The Right Reasons

As mentioned above, I’m working in Linux administration / engineering and know my way around technical aspects of mail systems. I also love to self-host stuff, and it motivates me to approach all kinds of challenges when it comes to making things work. As one of the positive side effects, I’m often able to apply the experience I’ve gained privately during my career.

There’s a few things one should consider before diving into selfhosting mail systems for production use. There’s a nice overview of best practices by Phil Pennock which I mostly agree with. A few more points from my own experience:

Tech Stack

My mail server is hosted inside a KVM / libvirt VM on a dedicated Hetzner server, a hosting provider I can recommend 100%. My OS of choice is Debian stable with backported kernels. An iptables script takes care of internal forwardings from hypervisor to VM. The mail server itself is Postfix, with Dovecot on top, as well as spamassassin, OpenDKIM, MySQL. There’s only one inbox with a catchall rule, where each login or service gets it’s own mail alias. Everything is monitored by Icinga2 and provisioned using Puppet. Since I’m accessing my mails only through either Thunderbird (Desktop) or K-9 Mail (Mobile), there’s no web frontend.

Implementation

While I’m not going into specifics regarding postfix, dovecot, etc. it’s important to mention a few architectual details. The mail server VM (residing as a qcow2 image file inside an encrypted LV, among others) is backed up twice per week using virsh blockcopy and transferred to another remote server. This setup has proven to be quite portable. I’ve since migrated my system to newer dedicated servers several times by just deploying my basic puppet hypervisor role, executing the iptables script, copying the VM the new server and updating my DNS records. I also like to test dist-upgrades by spinning up a local copy of the VM.

Monitoring is underrated when it comes to selfhosting. This is something I’ve learned soon after the initial deployment, in 2016, when postfix was down for about 14 hours due to carelessness on my part. I’ve since added Icinga2 to all of my systems for internal checks, as well as adding a secondary remote AWS EC2 Icinga2 instance for monitoring the monitoring server (yo dawg…) as well as various TCP ports from the outside.

my main Icinga2 instance watching over my mail server

Monitoring mails are delivered to an inbox outside of my mail setup. Same for cron mails. I can recommend an Android app called aNag, which visualizes Icinga2 state changes through push notifications, but I’m not going so far as to add some kind of oncall alerting. If something’s down, it stays down until I have time to fix it - which, so far, has not been the case with my mail server.

Mail rate and Spaminess

My low mail throughput is one of the likely reasons my setup has been working well. Even while being subscribed to a bunch of newsletters and services, there’s only about 20 to 40 incoming mails per week. Looking at my sent folder, there’s just about 550 outgoing mails since late 2015.

I’ve had exactly one problem with deliverabilty during that time, where someone with a Hotmail account complained to never have received my mail - even though the Microsoft server claimed to have accepted it according to my logs. While Microsoft can be notoriously intransparent and unforgiving with (not) accepting mail, in this case it turned out to be a blacklisting issue. I had just moved servers and IP addresses shortly before, with the new IP having been on an internal MS blacklist. I raised a ticket with their mail infrastructure department, and to my surprise, the IP was cleared soon after.

I rarely ever see any spam. Once every few months I’ll receive a french SEO mail, which is more of a mild curiosity than a bother, and not really worth looking into.

Ongoing Maintenance

As mentioned before, I nowadays spend maybe 2 to 5 hours per year on maintance, perhaps a bit more if a Debian dist-upgrade comes along. Every once in a while I’ll grep through my mail logs out of curiosity, but there’s rarely any surprises there. I recommend implementing some kind of auto-upgrade mechanism for security updates as well as subscriptions to various mailing lists, such as Debian Security.

Alternatives

Writing all this down, it does seem to be an insanely inconveniencing thing to do, but I’ve invested many hours tuning my setup and it seems rock solid at this point. If I were to give up selfhosting, my first choice would be to migrate my domain to mailbox.org. I consider mailbox.org to be one of the most capable and trustworthy mail providers out there. I also recently went through the steps of setting up someone elses domain with their MX servers, which has been very easy.

Conclusion

If you’re like me, an up and coming Linux sysadmin or enthusiast, hosting one’s own mail server can add lots of valuable experience. And for better or worse, there’s no one else to blame for if something goes wrong. And soon one thing leads to another, with additional monitoring, config management, blog posts ..

10/10 would selfhost again.

 2022 Jan Schumacher   •  Theme  Moonwalk