Downtime, Backups, and IPv6

Sorry for the unexpected downtime that lasted a few days and affected all the services here. One of the hard drives in the raid on my server suddenly showing miserable I/O performance, since the other drive was fine according to tests, SMART and logs I decided to take the slow drive offline to have it replaced. That turned out fatal since the other drive was also kaput and all my virtual machines on that server were now corrupted.

On the bright side the 2nd MX server worked fine and held all the email like it supposed to do until I got the main server back up and running (yaay to “exim -Mvb” to at least be able to read important mails that are in the mail server queue).

Long story short, since these drives have caused me much trouble in the past too, I decided to just get a new server with different hardware and restore everything from backups. Like any sysadmin I have my regular backups and scripts to check that they are working and occasionally test them to ensure I can restore a server from backups, but I’ve never had to really actually restore everything from a backup  on one of my private servers. So when doing so I noticed minor things that aren’t perfect and needed to be changed to make my life easier in the future and decided to share my experience.

Backups. Instead of making one large archive with all the directories backed up (e.g. /etc, /home, /opt, …), split it up into multiple archives of the applications you are backing up (e.g. /etc/apache2, /var/www/). I should have known better since this is common practice at work. It makes your life a lot easier when restoring from a backup if you have one archive with everything in it associated to a specific application, Digging through a backup of /etc and trying to remember all sub-directories needed to get email back up and running was a chore (webinterface, greylisting, spamassassin, exim, dovecot … some of them store data you also want to restore in other parts of the filesystem). You can always tar all the application backups together into one package before uploading it to the backup server if you want.

Use some kind of configuration and/or deployment management software (Puppet, Chef, Ansible, Salt, own scripts, …). Don’t underestimate how relaxing it is to just press a button to get your server(s) back into the defined state you previously had, just drop in the data from the backups (configuration too depending on how you roll) and you are good to go. Getting networking up and running manually took me longer than it should have (VPN networks, routing on the VM host, sysctl settings, NAT, which interfaces are bridged, which are internal, guest network configuration, …), I’ve put that all into Ansible playbooks now so it’s just a press of a button.

IPv6 is still sorely underused in the internet. I always set it up on all my hosts and mainly I see it being used by core services like email servers, dns, package download servers of large distributions, sometimes ntp, that kind of stuff, not so much on normal websites. Ordering extra IPv4 IPs for my guests normally takes an extra day or so until they are assigned to me (and they each cost extra), so I do enjoy having a huge subnet ip IPv6 IPs free of charge to do with as I please. It’s nice to spin up a virtual server, assign it an IPv6 IP and have it online without having to worry about NAT or port forwarding or other stuff you are confronted with when getting a server/services online that reside on an internal IP.
While I like to push people to embrace IPv6 early and get used to it, it is also important to give it the same attention you give IPv4 to make sure that you aren’t opening yourself up to security problems. Make sure you have the same firewall policy for IPv6 as you do for IPv4 and IPv6 brings a few new features with it that IPv4 didn’t have. Unless you explicitly need/use them it is best to shut them off. Below I’ll ad some IPv6 settings you probably want to set per default unless you are explicitly using them.

New Server (with IPv6)

I got a new server and I finished moving everything to the new hardware this week. If you notice anything behaving strange just drop me a note. I also set up IPv6 on all my VMs so will be playing around with that a bit. It’s a shame that hardly any providers give customers IPv6 addresses, migrating from IPv4 to IPv6 will take forever if none of the users can use them (and in the meantime IPv4 addresses are getting more and more expensive for me).

The new server is from hetzner since I have been very satisfied with them in the past. Impressive hardware features for low prices and (what really makes me keep choosing them) fantastic technical support. I’ve been with a few companies in the past, but none have had as good support as hetzner. Sooner or later hardware will die, and having a hosting provider that responds promptly and knowledgeably to request is priceless.

XEN 3.4 with ipv6 routing

Yes, there are a few postings out there about getting ipv6 routing running with XEN. But I’ll throw this online anyway since there are a few changes I had to make for it to work on my server. This text is intended for people who know their way around Linux and XEN so it will be a bit technical and won’t spell out every single step you have to make.

Most of the changes are based off scripts and information from BenV and wnagele (latter is interesting for me since I am also running XEN on a hetzner server). Have a look at the two links if anything is unclear. Now let’s start the fun 🙂

First of all we need IPv6 up and running on the host (dom0). Add the IP and gateway to your /etc/network/interfaces
This is what mine looks like:
iface eth0 inet6 static
address 2a01:4f8:100:1123::2
netmask 64
gateway 2a01:4f8:100:1120::1
pre-up ip -6 route add 2a01:4f8:100:1120::1 dev eth0

Check if the IP address is responding to the outside world (e.g. with wiberg.nu/iptools.php), if everything looks ok, proceed …
Now we need to enable a few things to get routing and neighbor discovery running on the host (dom0). Edit your /etc/sysctl.conf and add/change these 2 entries (and while you are at it, set them with “sysctl -w” too):
net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.proxy_ndp=1

So, your host should by now be online with ipv6 and soon be able to route packets to it’s guests. By default XEN will only take care of IPv4 when a guest is created, so here is a small patchfile that adds support for IPv6: xen-ipv6-vif-route.patch. The patch changes vif-route and vif-common.sh, while these files may be in different places depending on your distribution, /etc/xen/scripts/ is where they can commonly be found. Download the patch to the directory with the scripts to be changed and execute a “patch -p0 < xen-ipv6-vif-route.patch” (vif-common.sh gets a few new IPv6 functions, and iptables now won’t try to change stuff for IPv6 IPs. vif-route changes are: ndp is enabled for the vif device and the route/neighbor IPv6 settings are set)

So, now that the scripts know how to setup all our IPv6 needs, we need to add the IPv6 IP to our guest settings (.cfg file typically found in /etc/xen/). What we want to change is the “vif” setting. Add the IPv6 IP of the guest to the IPv4 IP (just the IP without the trailing /network, space separated form the IPv4 IP):
vif = [ 'mac=B1:A3:3F:25:11:B8, ip=2a01:4f8:100:1123::5 188.40.34.101' ]

Now you can create the guest(domU) and add the IPv6 IP to the /etc/network/interfaces of the guest if you haven’t so already (it uses the host (dom0) as the gateway).

iface eth0 inet6 static
address 2a01:4f8:100:1123::5
netmask 64
gateway 2a01:4f8:100:1123::2

Restart the networking on the guest (or reboot it) and you should now be able to ping the guest from the internet. See, easy wasn’t it 🙂