A script to diff files/directories on two different servers

Ok,  short one today. This is a straightforward script that simplifies comparing directories on different servers. There is no magic in it, it just rsyncs the directories to a local temp directory and runs diff against them (then deletes the directory afterwards). Mainly intended for config files, I wouldn’t recommend trying to diff gigabytes of binaries with it.


Convert configuration files to ansible templates

I’ve been playing around with ansible a lot lately, and I noticed that while changing stuff from “installed and configured manually” to “installed and configured by ansible” I was running into quite a few configuration files that needed to be manually turned into templates. It can be quite tedious to replace values in a configuration file with placeholders and put all those placeholders in a .yml file with default values.
Automating this is something I would have typically done in perl, but since I wanted to learn more about using regex in bash I decided to have a go at it in bash using regex and ${BASH_REMATCH}

The script takes a configuration file and spits out an ansible template, as well as the variable definitions you will need to add to your defaults/main.yml or vars/main.yml

The whole script is a bit to long to post here, but the interesting part is:

(You can download the full script here ansible_template.sh).

You can use regular expressions in a [[ ]] with =~ (e.g. if [[ “boot” =~ ^b ]]), and you can access the result of the regular expression by using ( ) to mark what parts of the result to store and access them via $BASH_REMATCH (comparable to how you would do it for other languages). Here I am parsing out anything that looks like a key=value from the configfile (with multiple possible separators) and storing the results in BASH_REMATCH[1] and BASH_REMATCH[2]

Usage of the script is pretty straightforward. you give it a prefix for the variable names (so you don’t end up with multiple roles all using a common variable name like “port”), and either a local or remote file to work with, and it spits out something like this:

There a tons of different configuration file formats out there so this script won’t work perfectly 100% of the time, but it does do quite well and reduces the manually copy&pasting to a minimum.

How to prevent changes to a tag via svn hook

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.


Playing around with DD-WRT

I’m currently playing around with my two WL-330GE Access points from asus (see an older posting). Since that posting I was a bit creative using the existing ethernet cabling and ports in the apartment to be able to retire the WiFi bridge without having any cable going through the apartment.

So I decided to use the two access points for something more useful. I’m playing around with dd-wrt to build configurations to use them as WiFi probes (for an IDS), or as Rouge Access Points (for demonstration purposes and to test wireless IDS solutions).  I might compile my own dd-wrt version for the rouge version, there are a few things I miss to build a truly evil device.

I like the size of the devices (very compact) and that you can power them with 5V (you can run them off any USB port, right now the one here is hooked up to the USB port of a printer intended for cameras) the only thing missing to make them perfect would be Power-over-Ethernet and maybe a GSM interface to upload data online.

Fun having a cheap and small device like this with Wifi and ethernet running linux. Provides lots of possibilities and fun.


How to get Teamspeak 3 running on a current Linux

Teamspeak is know for lagging a bit behind with development.

The last few days I have been upgrading my servers to current distributions, today the Voice servers were on the list to get Debian 6 / Ubuntu 11.04. And again I ran into problems with Teamspeak, turns out they won’t work with libmysqlclient 16 libraries and require the good old 15 version (which isn’t available out-of-the-box in the latest Debian and Ubuntu release).

So anybody running into the same problem (do a ldd libts3db_mysql.so to check), can hop on over to http://packages.debian.org/lenny/libmysqlclient15off and download the package for your architecture and install it with dpkg -i

captcha cracking

This is a pretty old posting from 2009 I just recently discovered in my “drafts” directory. Nowadays there are probably easier and more elegant ways of defeating a captcha, but for old times sake, here is my simple approach.

Eclectic and Marko were so kind as to “provide” me a captcha to play around with. Took me a few days of poking around and googling but in the end it was easier than I had thought. As long as there aren’t and logic errors in the code (e.g. bad or no session handling) you probably won’t get around some kind of OCR. As OCR software I decided to use gocr because it is free, runs under linux, and it is fairly easy to train to specific needs. Because I knew which libraries were being used to create the captcha images, it was possible for me to build a testing area. This just speeds things up a bit, the process would have worked just as well off the original website. First off: the spambot in action -> http://captcha.dopefish.de/spambot.php, and the website it accesses: http://captcha.dopefish.de/

Now I’ll describe the steps I took to defeat the captcha. Look at what happens on failed and successful inputs, first write a script that works if you enter the solution manually. I used the following 2 php functions for getting and posting stuff (and keeping the session intact)

Now train a gocr database for the images. Obviously it get’s better the more you train it.
Since curl is taking care of  session handling, we can use the get_url() function for downloading the captcha image. I pipe it through this shell command to make it easier for gocr to read:

It turnes this:

into this:

Since the valid captcha result is always the same length, we can check if gocr matched all the chars. If it looks good we can use post_url() to continue our session and throw all the fields at the form and submit it. See, wasn’t that hard. Most of the time is spent training gocr and converting the image into something easier to read. It doesn’t solve 100% of the images, more like 80-90%, but still better than nothing ;-).

Wireless bridge & dd-wrt

I recently bought the WL-330gE_M from Asus. It is a pair of access points pre-configured to bridge 2 LAN networks via wireless, all you have to do is take them out of the box and plug them in, straightforward and simple, no configuration needed. They are intended to enable hooking up devices to the internet that don’t have wireless and without pulling cables through the house (e.g. dvd player, TV, cable box, …).

The package arrived last week and it was a matter of minutes plugging the devices in and having everything working.  Everything worked without any setup, took me longer to get them out of the box than to hook them up.


Unfortunately our network storage (NAS) is also on the other end of this wireless bridge, and I noticed that when I move large files around (>2GB) or while streaming video/audio off the NAS the connection was dropping out. I don’t mean “ups and downs in the speeed” that is to be expected over wireless, I mean “connections resetting, copy actions aborting with error messages”. Not fun. Unfortunately since the devices are geared toward the “no configuration necessary, just unpack and hook up” crowd, there is no webinterface to see a syslog of what is happening or changing settings. Nada.

After this happening a few times it got really frustrating. I can live with slow, but connections dropping is out of the question. My original plan was to just reset the devices, flash them with a WL-330gE firmware and reconfigure the bridging (the only difference I could find was that the WL-330gE_M is black and not white, and comes preconfigured, and probably has a slightly different firmware without management capabilities).  While I was looking at different options and possibilities I went over to dd-wrt and happily saw that the WL-330gE was supported in the router database. So I decided if I was going to mess around with firmware, I could just as well throw dd-wrt on it.

Even though I am a system administrator, I don’t have the urge to have every device in the house running on Linux with a shell I can ssh in to. I’m perfectly fine with a simple interface that does what I want it to. But the wireless settings I can fine tune in dd-wrt are priceless (especially since I wanted to debug and fix the connection dropouts), normally you only get these options with cisco grade hardware.

The firmware upgrade process of the devices is simple and straightforward. Pull and reapply power with the reset button pressed until the power LED starts flashing, then shove the new firmware onto the device via tftp. Either with the “Firmware Restoration” tool from asus, or with a normal tftp client. I used later. Since this is so straightforward I guess I could also switch over to the official firmware if I wanted to, making two WL-330gE out of the WL-330gE_M pair (saves money since the pair is cheaper that buying two separately).

When in recovery mode (waiting for someone to tftp a new firmware onto it), the device has the IP by default. This is just a rough summary of the steps, anyone wanting to do this should really read through the whole process of deploying dd-wrt with asus, there is important information there (even if the example is a WL500, the WL330 is similar). Just because it worked for my hardware,firmware,setup doesn’t mean you have the same hardware or are deploying the same version I did. Read the dd-wrt documentation before you brick your device.

Clear current settings from the nvram:

Wait 5 min, reboot into recovery, throw a dd-wrt firmware on the device ( I used DD-WRT v24-sp2 (08/12/10) mini – build 14929, standard works fine too).

Wait 5 mins, reboot and open To be on the safe side feel free to navigate to Administration -> Factory Defaults to make sure no junk was left behind.  To get bridging configured there are multiple possibilites depending on your needs. For plain LAN bridging you will probably want WDS or one device setup as a AP and the second as a Client Bridge (I used the latter option). One thing you will want to do is go to Setup -> Networking and set the WAN port to “disabled” since the device only has LAN and Wireless.

The rest is fairly ease, set up one device as an AP, chose WPA2 with a good long strong PSK. After testing if the AP works with e.g. a laptop, you can set up the 2nd device as a Client Bridge, just make sure you are on the same channel, same SSID, same security settings.  After everything is up and running now would be a good time to pull backups from the configuration. Might as well tweak around in the wireless advanced settings. If you mess up anything badly enough that it won’t connect again … well that is why you made the configuration backups 😉

As you probably guessed by now, the connection drops are gone, connection is smooth and stable. Peak speed is not quite as fast as before because I throttled some things and tweaked settings for stability, but still good. Turning the TX antenna output power from 71 down to 65 helped a lot and got the maximum out of the connection (probably less crap pulling my SNR down). And now I can see what the access point is doing and where problems are when they arise 😉

back online

The hard drive crash threw me offline a few days due to strange problems with software raids, Xen and acpi. Turns out that using the latest Xen kernel from debian testing branch on a software raid only works of you don’t set “acpi=off” as a kernel parameter. If acpi is turned off, the script “scripts/local-top/mdadm” in the initrd can’t find the devices needed to mount the software raid … causing the whole boot process to come to a grinding halt.

If I find some time I’ll do some more tests, untill then my server will be running with acpi turned on

btw. the hard disk replacement was easy. after the new drive was popped in it was just a copy the partition table and add the partitions of the new disk to the raid


Server was down for the last 6 hours due to me updating a few stuff that really really didn’t work out too well (grub, xen and kernel at the same time). I gave up and afer a few hours of work managed to revert back to the old configuration so I can recieve mail again till I figure out what went wrong. Server may be down tomorrow a bit too while I try to figure out what exactly broke and try to at least get grub and xen updated.


Doh, I was just writing a bash script to calculate the current network traffic and write it to syslog. nothing special. I’m posting this from my laptop because I was forced to reboot my main PC after accidently writing a fork bomb. I wanted to recursively call myself and detatch the process so I could use the script as a deamon without worring about it if I logout.

AFTER starting the script, I noticed a flaw in my “if” startement to make sure I don’t DOS myself. By the time I got around to “hmm, maybe I should check the pid of that child and kill it so it doesn’t go and suck up my resources” it was too late. duh.

Free vServer during beta phase

I just stumbled accross this website: https://ssl.euserv.de/produkte/vserver/betatest.php where you can order a vServer for free during the beta phase. The beta phase is planned to last till the end of 2009. For more details, have a look at the link (there are a few thing not allowed to use the server for, mainly high traffic and illegal stuff).

An order key is required, just have a look at the tags of this posting …. one of them is a bit “strange” 😉

Moving Exim/Spamassassin/Cyrus -> Debian config

I’ve been putting off moving my mail system to the new server for a few weeks now since the old system was configured from scratch using the original config files and not the debian style config files. The differences in the Exim config are extreme. Debian splits the one large config file into lots of smaller files. This is great if: you never worked with exim before, you aren’t trying to migrate an existing configuration that is in one large file, and you don’t have all kinds of custom stuff like imap, spamassassin, greylisting mixed in. Yeah, not me. Even though I find the “one large file” a whole lot faster and easier to read, the Debian way has one big advantage: it is way easier for external scripts and packages to drop their custom config into exim. They just add a file to the right directory and thats it.

So I decided to go for it and merge my custom stuff into the Debain config. Greylisting worked out-of-the-box, spamassassin needed some minor tweaks, exim was (more or less) easy. The p.i.t.a. with exim is when you know exactly what is missing and where it would be configured, but because of “smart debian scripts” you have to find some config in an unrelated script and put the value there so it gets put in the right placeholder.

Here is a little summary in case I ever do this again and need to see if I forgot something:
– cyrus: copy /var/spool/cyrus/mail/ , /var/lib/cyrus/user/ , use cyradm to add the user.blargh account and /usr/sbin/cyrreconstruct -rf user (don’t forget the sieve filters)
– getmail: nothing special here, just copy config and add cronjob
– spamassassin: alter exim acl to set noscan for auth’d connections and have spamassassin scan everything not “noscan” (because per default local mail isn’t scanned, that includes everything we pick up via getmail)
– exim: check update-exim4.conf.conf for stupid entries, remember to turn on TLS (imap can use the same certificates), since we are using sasl for imap, have smtp auth use the same database (plain_saslauthd_server), turn on TLS by creating a file conf.d/main/00_exim4-config_localmacros with “MAIN_TLS_ENABLE = true” in it

Now that I’m done I found a pretty detailed German website with steps to set up such a system E-Mail-Server mit Debian, Exim und Cyrus. I did the exim router/transports a bit differently to have a bit more control over what goes where when. Still, defiantly worth reading if you are thinking about building such a system.

vmware is odd

I installed the final release of vmware server 2.0 on a server today. Not much has changed since the RC version I had been using. I found some quick fixes and tips in this blog: http://digital.blogsite.org/index.php/2008/10/04/review-vmware-sever-2
The authorization.xml problem was a REAL pain, so I was grateful to find a fix for that. And the tip about the VI Client laying around on the server was priceless (duh, the least they could have done could have been a link in the webinterface). Using the client to access the vmware host is finally not painful anymore. The webinterface is still Ok if I’m not at a computer of mine, or am under linux.

There are still some oddities about vmware that bug me, like the missing option to turn off the DHCP server if you set up networking to use a virtual switch (HostOnly). The VI Client allows me to do some stuff that isn’t possible with the webinterface (like priority for RAM and CPU, or CPU affinity), but it doesn’t allow me to ad virtual machines that are on the server ?!?

The Software is free, so I’m not complaining here. But theese would be a real pain in the rear end if I was using the software in a production environment.

save the forest (or something like that …)

I can’t be the only person who finds it bothersome burning a cd just to install linux. What a waste (it’s not like you use the cd all to often afterwards, except as a coaster). So here are the quick and dirty instructions for making a debian linux install usb stick (adjust /dev/sdb accordingly, failure to do so can pretty must kill any data on a harddisk):

  • wget ftp://ftp2.de.debian.org/debian/dists/etch/main/installer-i386/current/images/hd-media/boot.img.gz
  • wget http://cdimage.debian.org/debian-cd/4.0_r5/i386/iso-cd/debian-40r5-i386-netinst.iso
  • zcat boot.img.gz > /dev/sdb
  • mount /dev/sdb /mnt
  • cp debian-40r5-i386-netinst.iso /mnt
  • umount /mnt

Basic Server Hardening

Ok, here is a list of a few programs I’d advise anyone to use who is running a server on the internet (or thinking of doing so).

  • aide or tripwire (they can check and report if files on your system get changed, configurable levels). If you use tripwire, don’t forget a “tripwire –check -I” after you do any updates.
  • logcheck will check your system logs, and report anything out of the ordinary (“ordinary” is defined by a list of ‘normal’ rules, and anything you add)
  • tiger goes farther than logcheck, it actively checks your system and reports anything strange (files not belonging to packages, users or groups that got added, …)
  • grsecurity adds more security features to your kernel (at least use the basic features and the possibility to turn off module loading after boot)
  • rkhunter, chkrootkit scan the system for signs of rootkits or other malware. just install, make sure they are executed daily by cron, possibly tweak rkhunters config a bit (I had problems with unhide and current kernel versions)

I’m not saying that setting up and tweaking all this software and actually reading the emails they generate will make your server super-duper secure, but they will reduce the risk of running a server open to the internet and alarm you if somthing strange is happening. It is important to read and understand what theese programs mail you. Yes, you will get false positives from time to time. And yes, you will have to adjust the config now and then due to package updates; but I get about 3-4 mails a week, and that is definatly ok considering the amount of data that gets checked.

Bash Scripting

I’ve been doing a bit of bash scripting lately. Anyone who is interrested in bash scripting should also have a look at the “bash support” vim script http://www.vim.org/scripts/script.php?script_id=365. A fair amount of the addons are aimed at a gui usage (like gvim), but even if you are a console user like me, it adds enough features to be worth while. After using it for a few days you get addicted to the neat features, scripting in a vim without it is like typing with your nose. It’s not impossible, but you aren’t having much fun either.

MSI Wind U100

I bought myself a laptop this week.  To be more precise I bought a MSI Wind U100 “Luxury” version in white with 2GB Ram. It arrived today, and I have been spending most of the afternoon setting up Windows. I must say, I’m positively surprised about how good it works and the default setup. It came with 3 partitions, the first (about 3GB) is a rescue system, the second (about 50GB) has windows installed, and the rest (100GB) was an empty partition. I reduced the last partition to 50GB and will be installing Linux in the other half later on (Dual-Boot). It seems most of the community is only interested in installing ubuntu on the msi wind, so let’s see how far I get with a “normal” debian install. Not that ubuntu is bad, I use it often enough as a desktop installation, but this laptop isn’t really your standard hardware or usage here. So debian it will be, and minimized/customized to to what I want efficiently and good looking ;-).

Since there are plenty of reviews floating around the ‘net, I’ll spare you all a rant about how cool the notebook is.

Hackit Server downtime

Sorry for the downtime, wasn’t planned. It was late last night when I set up the knock daemon, I somehow managed to accidently copy and past my terminal which resulted in about a quarter of my /etc/init.d/* scripts getting broken. Unfortunatly I didn’t notice it right away. I did notice it when I rebooted the server (kernel change) and lot’s of daemons didn’t come up (oh unimportant stuff like SSH 🙁 ) Well, that’s what backups are for.

knock daemon with INPUT chain set to default ACCEPT

I know there are plenty of pages floating around the Internet about knock daemons that open ports in a firewall after a predefined series of ports are “knocked”. For some reason ALL the pages I found assumed that a) you want the filter in your INPUT chain, and that the INPUT chain defaulted to DROP or REJECT.
In my case, I’m defiantly not going to have a iptables firewall with a default that drops packets. Every few weeks I try out some new software and can’t be bothered with adjusting my firewall every time. All I need it to do is keep pesky people off my ssh, that’s all.

So here is a short tutorial how to set up s knock daemon with a ACCEPT default for INPUT: