Posted in Security, Server, Tech

Setting up multidomain DKIM with Exim

I was recently setting up SPF, DKIM and DMARC for multiple domains and was having trouble getting Exim to sign emails for the different domains. I found an article here explaining the steps. But I kept getting the following error in my exim logs:

failed to expand dkim_private_key: missing or misplaced { or }

The suggested configuration was the following:

I’m not quite sure why, but Exim was having trouble using the macros in the following macros, so I ended up changing it to the following snippet instead. If you don’t use DKIM_FILE you can omit it. Also you might want to set DKIM_STRICT to true if you published a DMARC policy that will reject or quarantine email failing the DKIM tests (unset, or “false” tells Exim to send the message unsigned if it ran into problems signing the email). The default setting for DKIM_CANON is “relaxed“, so it also can be omitted.

Other than that, just make sure the exim process has permissions to access the dkim directory and certificate files and everything should work nicely.

Posted in Server, Tech

How to prevent changes to a tag via svn hook

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.


Posted in Server, Tech

Downtime, Backups, and IPv6

Sorry for the unexpected downtime that lasted a few days and affected all the services here. One of the hard drives in the raid on my server suddenly showing miserable I/O performance, since the other drive was fine according to tests, SMART and logs I decided to take the slow drive offline to have it replaced. That turned out fatal since the other drive was also kaput and all my virtual machines on that server were now corrupted.

On the bright side the 2nd MX server worked fine and held all the email like it supposed to do until I got the main server back up and running (yaay to “exim -Mvb” to at least be able to read important mails that are in the mail server queue).

Long story short, since these drives have caused me much trouble in the past too, I decided to just get a new server with different hardware and restore everything from backups. Like any sysadmin I have my regular backups and scripts to check that they are working and occasionally test them to ensure I can restore a server from backups, but I’ve never had to really actually restore everything from a backup  on one of my private servers. So when doing so I noticed minor things that aren’t perfect and needed to be changed to make my life easier in the future and decided to share my experience.

Backups. Instead of making one large archive with all the directories backed up (e.g. /etc, /home, /opt, …), split it up into multiple archives of the applications you are backing up (e.g. /etc/apache2, /var/www/). I should have known better since this is common practice at work. It makes your life a lot easier when restoring from a backup if you have one archive with everything in it associated to a specific application, Digging through a backup of /etc and trying to remember all sub-directories needed to get email back up and running was a chore (webinterface, greylisting, spamassassin, exim, dovecot … some of them store data you also want to restore in other parts of the filesystem). You can always tar all the application backups together into one package before uploading it to the backup server if you want.

Use some kind of configuration and/or deployment management software (Puppet, Chef, Ansible, Salt, own scripts, …). Don’t underestimate how relaxing it is to just press a button to get your server(s) back into the defined state you previously had, just drop in the data from the backups (configuration too depending on how you roll) and you are good to go. Getting networking up and running manually took me longer than it should have (VPN networks, routing on the VM host, sysctl settings, NAT, which interfaces are bridged, which are internal, guest network configuration, …), I’ve put that all into Ansible playbooks now so it’s just a press of a button.

IPv6 is still sorely underused in the internet. I always set it up on all my hosts and mainly I see it being used by core services like email servers, dns, package download servers of large distributions, sometimes ntp, that kind of stuff, not so much on normal websites. Ordering extra IPv4 IPs for my guests normally takes an extra day or so until they are assigned to me (and they each cost extra), so I do enjoy having a huge subnet ip IPv6 IPs free of charge to do with as I please. It’s nice to spin up a virtual server, assign it an IPv6 IP and have it online without having to worry about NAT or port forwarding or other stuff you are confronted with when getting a server/services online that reside on an internal IP.
While I like to push people to embrace IPv6 early and get used to it, it is also important to give it the same attention you give IPv4 to make sure that you aren’t opening yourself up to security problems. Make sure you have the same firewall policy for IPv6 as you do for IPv4 and IPv6 brings a few new features with it that IPv4 didn’t have. Unless you explicitly need/use them it is best to shut them off. Below I’ll ad some IPv6 settings you probably want to set per default unless you are explicitly using them.

Posted in Server, Tech

How to find the fingerprints of public keys in authorized_keys

If you use keys for SSH authentication (and you should) then you have probably run into the situation that the auth.log shows that someone logged in, even which local user was used (e.g. root), but you have no idea which of the keys in ~/.ssh/autorized_keys was used. The first step you can do to see what is going on, is increasing the log level of the SSH daemon:


That will spit out the fingerprint of the SSH key used to log in. Example log entry for a successful login:

Now that we have the fingerprint of the ssh key used to login, we will need ssh-keygen to spit out the fingerprints of the public keys in ~/.ssh/authorized_keys to be able to compare them. So I wrote a little wrapper called around ssh-keygen to feed it all the public keys from authorized_keys (if you want you can even fit the whole while loop as a oneliner):

Posted in Server, Tech

New Virtualbox version, script for easy update of extension pack

VirtualBox 4.1.10 was released yesterday with a few nice things in the changelog. Updating virtualbox itself is easy, just download the package and update it. Since I seem to stumble over the update of the extension pack every time (on my headless system) I thought I’d write a small script this time so I don’t have to rethink it next time (automatically downloads and installs the current extpack):


Posted in Internet Stuff, Server

New Server (with IPv6)

I got a new server and I finished moving everything to the new hardware this week. If you notice anything behaving strange just drop me a note. I also set up IPv6 on all my VMs so will be playing around with that a bit. It’s a shame that hardly any providers give customers IPv6 addresses, migrating from IPv4 to IPv6 will take forever if none of the users can use them (and in the meantime IPv4 addresses are getting more and more expensive for me).

The new server is from hetzner since I have been very satisfied with them in the past. Impressive hardware features for low prices and (what really makes me keep choosing them) fantastic technical support. I’ve been with a few companies in the past, but none have had as good support as hetzner. Sooner or later hardware will die, and having a hosting provider that responds promptly and knowledgeably to request is priceless.

Posted in Internet Stuff, Server

How to install ruby 1.9.2 on Ubuntu 10.04

The current LTS version of Ubuntu is 10.04 and the most current version of ruby it ships with is 1.9.1. Unfurtunately 1.9.1 wasn’t that great of a release and anyone using the 1.9 branch really should use the stable 1.9.2.

After doing a bit of researching I found some information on how the best approach to get ruby installed is. Downloading the source, compiling it and registering the installed version with the package manager.

The following little bash script takes care of installing ruby 1.9.2 on a ubuntu or debian based system (or any other version if you change the $Version variable in the script). The script just consolidates information found online and wraps it up into a nice bashscript


Posted in Internet Stuff, Server

Using iFolder as an open source Dropbox alternative

I’ve had a few posting here in my blog over the years about trying to keep data synced over various computers. Nowadays it has gotten easier since it seems everyone is chruning out their own online storage. Classics like Dropbox or SugarSync, and then we have the big companies  Google, Microsoft and Ubuntu trying their luck. Basically they all provide you with 2-5Gb free storage, and more space and features for more money. But they all have one downside, whether encrypted or not, you are leaving your own files to the mercy of an external company. There is not really any voodoo in the whole system, basically you just have some space online, a bit of encryption, and some way to ensure that all clients of a user are synced, nonetheless building your own service can be a hassle as I have found out.

Currently I’m using SVN as a workaround. I just sometimes forget to commit, which leads to me having old data and no access to updated files. Instead of starting from scratch and build strange stuff with rsync I had a look if there were any open source solutions out there. And I was pleased to find a few. Unfortunately they are almost all in the “I have an idea and a few mockups” stage, maybe even some beta version that does something, but nothing really “production” level. Another downside was that most of them either supported Linux or Windows, but often not both.

One open source solution I stumbled across is iFolder,  seasoned software supported by Novell, with clients for Windows, Linux and Mac. And if all else fails you can use the web interface to access your files. Admin and User web interface are both easy to use. I don’t remember how I stumbled across it, but I do know that it’s not one of the main products you see named when searching for alternatives to Dropbox.

If you have SuSE as an OS, then installing the software is immensely simplified due to SuSE belonging to Novell, just follow the SuSE instructions on the website. I prefer debian based installations and found Ubuntu instructions at that were written mainly for Ubuntu, but should work with any debian deviate.

I’m not going to repeat anything from the installation instructions, they already did a fine job of documenting everything. One thing is important however: the Client just plain doesn’t work. unfortunately it is the one linked from the website. Go to the sourceforge repository, and download the newer 3.8.03 client. That one works like a charm.

Posted in Internet Stuff, Server

How to get Teamspeak 3 running on a current Linux

Teamspeak is know for lagging a bit behind with development.

The last few days I have been upgrading my servers to current distributions, today the Voice servers were on the list to get Debian 6 / Ubuntu 11.04. And again I ran into problems with Teamspeak, turns out they won’t work with libmysqlclient 16 libraries and require the good old 15 version (which isn’t available out-of-the-box in the latest Debian and Ubuntu release).

So anybody running into the same problem (do a ldd to check), can hop on over to and download the package for your architecture and install it with dpkg -i

Posted in Internet Stuff, Programming, Server

Wireshark remote capturing

yeah, this is real simple stuff, not really worth writing a script for it. but on the other hand it saves me from remembering how to do it every time I need it (which isn’t often). So here is a little script to setup remote capturing with wireshark.
All it basically does is ssh to the remote host and tcpdump sucking the output via stdout through the ssh connection to a local pipe, that is then used by wireshark to display the stream. Because of this you may want to make sure you aren’t capturing your own ssh data when doing this 😉

Posted in Programming, Server

Controlling SSH identities

SSH has a few strange undocumented “features”. One of which is the way it handles identities via agent and command line. It is possible to specify an identity file to use for ssh via the -i parameter (ssh -i identity_file $host). What the manpage doesn’t mention, is that the specified identity isn’t forced for the connection, it is just added to the list of possible identities.
To make matters worse, ssh tries the identities from the agent first. So if agent forwarding is enabled and valid for the destination the ssh command will never use the identity specified with -i. Why is this “bad”? Because the identity specified may be used for specific tasks with commands linked to them on the destination (e.g. automatic restarts, backups, …)

Sooo, as a solution I whipped up the following function as a workaround in my scripts, I add a function called “xssh”:

I know it looks ugly, if it finds a key in the agent it makes a ssh connection to the current host with agent forwarding deactivated and then executes the ssh parameters passed. If no key is found in the ssh agent it does everything as normal.

Posted in Internet Stuff, Server

XEN 3.4 with ipv6 routing

Yes, there are a few postings out there about getting ipv6 routing running with XEN. But I’ll throw this online anyway since there are a few changes I had to make for it to work on my server. This text is intended for people who know their way around Linux and XEN so it will be a bit technical and won’t spell out every single step you have to make.

Most of the changes are based off scripts and information from BenV and wnagele (latter is interesting for me since I am also running XEN on a hetzner server). Have a look at the two links if anything is unclear. Now let’s start the fun 🙂

First of all we need IPv6 up and running on the host (dom0). Add the IP and gateway to your /etc/network/interfaces
This is what mine looks like:
iface eth0 inet6 static
address 2a01:4f8:100:1123::2
netmask 64
gateway 2a01:4f8:100:1120::1
pre-up ip -6 route add 2a01:4f8:100:1120::1 dev eth0

Check if the IP address is responding to the outside world (e.g. with, if everything looks ok, proceed …
Now we need to enable a few things to get routing and neighbor discovery running on the host (dom0). Edit your /etc/sysctl.conf and add/change these 2 entries (and while you are at it, set them with “sysctl -w” too):

So, your host should by now be online with ipv6 and soon be able to route packets to it’s guests. By default XEN will only take care of IPv4 when a guest is created, so here is a small patchfile that adds support for IPv6: xen-ipv6-vif-route.patch. The patch changes vif-route and, while these files may be in different places depending on your distribution, /etc/xen/scripts/ is where they can commonly be found. Download the patch to the directory with the scripts to be changed and execute a “patch -p0 < xen-ipv6-vif-route.patch” ( gets a few new IPv6 functions, and iptables now won’t try to change stuff for IPv6 IPs. vif-route changes are: ndp is enabled for the vif device and the route/neighbor IPv6 settings are set)

So, now that the scripts know how to setup all our IPv6 needs, we need to add the IPv6 IP to our guest settings (.cfg file typically found in /etc/xen/). What we want to change is the “vif” setting. Add the IPv6 IP of the guest to the IPv4 IP (just the IP without the trailing /network, space separated form the IPv4 IP):
vif = [ 'mac=B1:A3:3F:25:11:B8, ip=2a01:4f8:100:1123::5' ]

Now you can create the guest(domU) and add the IPv6 IP to the /etc/network/interfaces of the guest if you haven’t so already (it uses the host (dom0) as the gateway).

iface eth0 inet6 static
address 2a01:4f8:100:1123::5
netmask 64
gateway 2a01:4f8:100:1123::2

Restart the networking on the guest (or reboot it) and you should now be able to ping the guest from the internet. See, easy wasn’t it 🙂

Posted in Programming, Server

Script of the day – clean up stale .ssh/known_hosts

This little script takes an IP or hostname as a parameter, and if there is an offending key in the .ssh/known_hosts it removes it and replaces it with the current valid one useful if you are moving/reinstalling a large amount of servers …

Posted in Server

back online

The hard drive crash threw me offline a few days due to strange problems with software raids, Xen and acpi. Turns out that using the latest Xen kernel from debian testing branch on a software raid only works of you don’t set “acpi=off” as a kernel parameter. If acpi is turned off, the script “scripts/local-top/mdadm” in the initrd can’t find the devices needed to mount the software raid … causing the whole boot process to come to a grinding halt.

If I find some time I’ll do some more tests, untill then my server will be running with acpi turned on

btw. the hard disk replacement was easy. after the new drive was popped in it was just a copy the partition table and add the partitions of the new disk to the raid

Posted in Internet Stuff, Server, Tech

XEN and Kernel update

Server is up and running again. This morning I finished the xen (3.4) and kernel (2.6.32) updates I prepared yesterday evening. The only pitfall I stumbled across, was that the kernel in the debian repository (2.6.26-xen) seemingly wasn’t compatible with the latest xen in the debian repository (system would hang while booting, I guess that’s what I get for using “testing”).

Anyway, anyone using “lenny” or “testing” and wanting to update xen; here are a few tips to make the transition a breeze:

  • The debian wiki is full of useful information regarding xen
  • has a current xen kernel (linux-image-2.6.32-bpo.4-xen-amd64) that works fine with xen 3.4, just follow the instructions on on how to add it to apt.
  • I had to deactivate the “vfb” (VNC console) setting in the domU config files to get my guests to boot, browsing the Internet I saw people having the same problem with the “dhcp” setting.
  • If you are planning on updating the guests kernels too (advised), remember to change the “kernel” and “ramdisk” settings in the domU config files accordingly

With these few points in mind, the update is a breeze.