Odroid HC1

I finally got my new Odroid SBC and have migrated all the services from my Odriud XU4 over to the new HC1. The HC1 is based on the XU4 but drops multimedia interfaces like the HDMI port and instead adds a SATA connector and the black aluminium case fits a 2.5″ hard drive. The case is stackable and acts as a passive cooling heat sink. Since the HC1 is based on the XU4, software that runs on a XU4 will run on a HC1 without any fiddling.

It doesn’t have an eMMC connector, but you can assign the /  partition to the hard drive/SSD, only the /boot  partition needs to be on the SD Card.

Selfhosting email, and sending email to Microsoft.

About once a year one of my outgoing email servers will magically pop up on Microsofts blacklist of email servers. The exact status in SNDS is “Blocked due to user complaints or other evidence of spamming“, which is a bit ironic since the weekly volume of emails that go to Microsoft controlled domains seldom is higher than 5 and they are all personal emails.
The error message the MTA recieves is “550 5.7.1 Unfortunately, messages from [xxx.xxx.xxx.xxx] weren’t sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.” in case anyone ends up here via google.

I’m not going to bitch and moan or attempt to guess why the server ends up on the list since it isn’t something I can influence
 (just submit a support request and wait), instead I’ll post some tips and pointers to useful tools you should be using:

SPFSender Policy Framework
Specify which IPs are allowed too send mail for your domains, also determine what should happen with mail that is recieved from unauthorized IPs.

DKIMDomainKeys Identified Mail
Outgoing mail servers for a domain can sign the emails and thereby allow recipients to verify that the email really came from a valid mail server and is not from a spoofed sender.

DMARCDomain-based Message Authentication, Reporting & Conformance
DMARC allows you to specify policies for domains or subdomains in regards to “what should happen if an email fails DKIM or SPF” and where/if to send reports for Emails recieved by other mail servers.

SNDSSmart Network Data Service
Allows you to monitor the IPs of your mailservers as viewed by the Outlook.com system.

JMRPJunk Email Reporting Program
Forwards the full message with headers of any email marked as “junk” or “phishing”  by Outlook.com users

MX Toolbox
MXtoolbox is a great website for testing your mail server settings, they also have a free_monitoring service for one IP,  so you can get alerted if your IP shows up on a Blacklist.

Last but not least I use the following small script to monitor the status of my servers according to Microsoft. I call the script via a daily cronjob, and if a server is blacklisted, it sends an alert. Go to Automated Data Access first to generate an API key.

And if all else fails and your IP does end up blacklisted in SNDS, you can go here to submit a ticket to get back off (although I’d suggest checking your mail server logs and the status of the IP on other RBLs first, just to make sure).



Setting up multidomain DKIM with Exim

Update November 2018:

A reader contacted me and pointed out that removing the {} around DKIM_DOMAIN solves the errors in the original example I found and had problems using. I’ve updated the code below (line 8) to reflect those changes in case anyone ends up here via google.

He also shared a nifty way to make selector rollovers easier by adding them to the filename:

And last but not least an elegant way to populate the  DKIM_DOMAIN variable: https://bugs.exim.org/show_bug.cgi?id=1019

Original Posting:

I was recently setting up SPF, DKIM and DMARC for multiple domains and was having trouble getting Exim to sign emails for the different domains. I found an article here explaining the steps. But I kept getting the following error in my exim logs:

failed to expand dkim_private_key: missing or misplaced { or }

The suggested configuration was the following:

I’m not quite sure why, but Exim was having trouble using the macros in the following macros, so I ended up changing it to the following snippet instead. If you don’t use DKIM_FILE you can omit it. Also you might want to set DKIM_STRICT to true if you published a DMARC policy that will reject or quarantine email failing the DKIM tests (unset, or “false” tells Exim to send the message unsigned if it ran into problems signing the email). The default setting for DKIM_CANON is “relaxed“, so it also can be omitted.

Other than that, just make sure the exim process has permissions to access the dkim directory and certificate files and everything should work nicely.

How to prevent changes to a tag via svn hook

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.


Downtime, Backups, and IPv6

Sorry for the unexpected downtime that lasted a few days and affected all the services here. One of the hard drives in the raid on my server suddenly showing miserable I/O performance, since the other drive was fine according to tests, SMART and logs I decided to take the slow drive offline to have it replaced. That turned out fatal since the other drive was also kaput and all my virtual machines on that server were now corrupted.

On the bright side the 2nd MX server worked fine and held all the email like it supposed to do until I got the main server back up and running (yaay to “exim -Mvb” to at least be able to read important mails that are in the mail server queue).

Long story short, since these drives have caused me much trouble in the past too, I decided to just get a new server with different hardware and restore everything from backups. Like any sysadmin I have my regular backups and scripts to check that they are working and occasionally test them to ensure I can restore a server from backups, but I’ve never had to really actually restore everything from a backup  on one of my private servers. So when doing so I noticed minor things that aren’t perfect and needed to be changed to make my life easier in the future and decided to share my experience.

Backups. Instead of making one large archive with all the directories backed up (e.g. /etc, /home, /opt, …), split it up into multiple archives of the applications you are backing up (e.g. /etc/apache2, /var/www/). I should have known better since this is common practice at work. It makes your life a lot easier when restoring from a backup if you have one archive with everything in it associated to a specific application, Digging through a backup of /etc and trying to remember all sub-directories needed to get email back up and running was a chore (webinterface, greylisting, spamassassin, exim, dovecot … some of them store data you also want to restore in other parts of the filesystem). You can always tar all the application backups together into one package before uploading it to the backup server if you want.

Use some kind of configuration and/or deployment management software (Puppet, Chef, Ansible, Salt, own scripts, …). Don’t underestimate how relaxing it is to just press a button to get your server(s) back into the defined state you previously had, just drop in the data from the backups (configuration too depending on how you roll) and you are good to go. Getting networking up and running manually took me longer than it should have (VPN networks, routing on the VM host, sysctl settings, NAT, which interfaces are bridged, which are internal, guest network configuration, …), I’ve put that all into Ansible playbooks now so it’s just a press of a button.

IPv6 is still sorely underused in the internet. I always set it up on all my hosts and mainly I see it being used by core services like email servers, dns, package download servers of large distributions, sometimes ntp, that kind of stuff, not so much on normal websites. Ordering extra IPv4 IPs for my guests normally takes an extra day or so until they are assigned to me (and they each cost extra), so I do enjoy having a huge subnet ip IPv6 IPs free of charge to do with as I please. It’s nice to spin up a virtual server, assign it an IPv6 IP and have it online without having to worry about NAT or port forwarding or other stuff you are confronted with when getting a server/services online that reside on an internal IP.
While I like to push people to embrace IPv6 early and get used to it, it is also important to give it the same attention you give IPv4 to make sure that you aren’t opening yourself up to security problems. Make sure you have the same firewall policy for IPv6 as you do for IPv4 and IPv6 brings a few new features with it that IPv4 didn’t have. Unless you explicitly need/use them it is best to shut them off. Below I’ll ad some IPv6 settings you probably want to set per default unless you are explicitly using them.