Posted in Me

Fixing Ubuntu 17.04 DNS problems

I recently upgraded my Ubuntu box to 17.04.

Much to my surprise DNS starting behaving strangely, so I checked my DNS server … worked fine if I queried it directly, so I checked if DHCP was giving out the wrong DNS IP … nope, that was fine too. I checked /etc/nsswitch.conf , and that looked fine too so I checked what was ending up in /etc/resolv.conf and was surprised that it contains nameserver  instead of the “real” DNS server.

After a bit of research I found out that Ubuntu switched over to using systemd-resolved, which shoves itself between user land and the DNS servers and (at least in Ubuntu 17.04) has problems with servers that support DNSSEC. Very frustrating when you know everything is OK and worked in the past, just systemd messing with stuff and breaking it.

My workaround was to turn of DNSSEC validation. Not pretty but better than no DNS at all, until Ubuntu get’s their problems sorted out.


Posted in Programming, Tech

How to fetch IP ranges/entries from SPF records in bash

Recently I needed to fetch IP ranges from SPF records. After looking at different python/ruby/perl modules I came to the conclusion that a fancy module (sometimes with wonky dependencies) was overkill just to parse a simple SPF record. So I threw together a simple bash script that is mainly just fetching the SPF record with dig and grep:

It iterates through the options (it currently recognizes a, mx, ip4, ip6, include, and redirect), and then sorts the output by ipv4, then ipv6.

Download URL:

Posted in Programming, Tech

How to compare package version strings in bash

This is a little function I use to compare package version strings. Sometimes they can get complex with multiple different delimiters or strings in them. I cheated a bit by using sort –version-sort for the actual comparison. If you are looking for a pure bash version to compare simpler strings (e.g. compare 1.2.4 with 1.10.2), I’d suggest this stackoverflow posting.

The function takes three parameters (the version strings and the comparison you want to apply) and uses the return code to signal if the result was valid or not. This gives the function a somewhat natural feel, for example compare_version 3.2.0-113.155 “<” 3.2.0-130.145 would return true. Aside from < and > you can also use a few words like bigger/smaller, older/newer or higher/lower for comparing the strings.

List of return codes and meanings:



Temporary theme

A recent update broke my WordPress theme. I’ve used the same theme for almost 10 years and it was starting to be a pain to keep updated and working with newer WordPress versions. So I decided to put up this simple theme until I get a new theme picked out and up and running.

Posted in Security, Server, Tech

Setting up multidomain DKIM with Exim

I was recently setting up SPF, DKIM and DMARC for multiple domains and was having trouble getting Exim to sign emails for the different domains. I found an article here explaining the steps. But I kept getting the following error in my exim logs:

failed to expand dkim_private_key: missing or misplaced { or }

The suggested configuration was the following:

I’m not quite sure why, but Exim was having trouble using the macros in the following macros, so I ended up changing it to the following snippet instead. If you don’t use DKIM_FILE you can omit it. Also you might want to set DKIM_STRICT to true if you published a DMARC policy that will reject or quarantine email failing the DKIM tests (unset, or “false” tells Exim to send the message unsigned if it ran into problems signing the email). The default setting for DKIM_CANON is “relaxed“, so it also can be omitted.

Other than that, just make sure the exim process has permissions to access the dkim directory and certificate files and everything should work nicely.

Posted in Security, Tech

OpenVAS: Using PostgreSQL instead of sqlite

When using OpenVAS in larger environments (e.g. lots of tasks and/or lots of slaves) you may have noticed the manager controlling all the slaves/scans can get sluggish or unresponsive at times. In my experience it is often due to the different processes waiting for an exclusive lock on the sqlite database. Fortunately OpenVAS 8 and above also supports using PostgreSQL as a database backend instead of sqlite. I think OpenVAS 7 also had support built-in, but it was still considered experimental.

Documentation on how to use PostgreSQL as the backend is in the OpenVAS svn repository. In a nutshell it is mainly adding -DBACKEND=POSTGRESQL to your cmake when you compile the manager (my cmake line is cmake -DCMAKE_INSTALL_PREFIX=/ -DCMAKE_BUILD_TYPE=Release -DBACKEND=POSTGRESQL ..). I generally only compile the master with PostgreSQL support and leave the slaves to use sqlite (since they don’t have as many concurrent accesses to their database). The documentation also steps you through the permissions you need to set up in PostgreSQL so it can be used by OpenVAS. Don’t forget to make the system aware of your OpenVAS libraries, in my case since I install OpenVAS to / I put /lib64/ in my /etc/ file and then execute ldconfig.

One issue you may run into is migrating data from sqlite to PostgreSQL. There is a migration script in svn that can migrate the data, but it only works for a few older database versions. I assume OpenVAS 9 will contain an updated version of the script when it is released, but until then I wrote a script that uses the OMP protocol to export/store/import some of the settings. Since it only uses OMP to communicate with the master it is backend agnostic. You can use it to export the sqlite data and import it back into a manger using the PostgreSQL backend. It also means that it can only access data you can export via OMP (so no credential passwords/keys). The script will keep references intact (which tasks uses which target/schedule/…). The list of what it exactly imports/exports is on the github page:

Posted in Ansible, Tech

How to easily switch between ansible versions

Lately I’ve run into issues with different versions of ansible (1.9 handling async better, 2.x having more modules and handling IPv6 better) and having to test playbooks and roles against different versions to make sure they work. TO make life easier I put this little function in my .bashrc to switch back and forth between ansible versions. It checks out the specified version from github if it needs to, and switches over to it (just for that terminal, not the system). Usage is straight forward ansible_switch <branch> , i.e. ansible_switch 2.1  (or whatever branch you want, here is a list of all branches).

It is currently limited to stable branches, but you can change line 6 from stable- to whatever you want (or remove the prefix completely). If you have a github account you also may want to change from https to ssh by using the checkout URL.


Posted in Ansible, Tech

Ansible tasks to reboot a server if required

A quick one today.  The following ansible tasks check if a server needs to be rebooted, reboots it, and then waits for it to come back online. Easy to fire off during a maintenance after updating packages.


Posted in Security, Tech

How to check if you are vulnerable for the DROWN attack (CVE-2016-0800).

CVE-2016-0800, also known as the DROWN attack, is an attack against servers that still support the old SSLv2 protocol. The only reason a server would still offer to use SSLv2 would be for possible compatibility reasons with 20-year-old PCs ( -> there is no reason to use or offer SSLv2 any more). From a configuration side you can disable the v2 protocol by adding  -SSLv2 to the list of protocols being used.

Where and how you configure this depends on the software, but using all -SSLv2 -SSLv3 is fine with most modern servers and clients, Mozilla has a fantastic overview for configuring SSL and TLS.

If you want to check a bunch of your hosts remotely, you can use the sslv2 script included with nmap like this:

Where hostname would be either a FQDN, or an IP, or an IP range. You can swap out  sslv2 with  ssl-enum-ciphers to see all SSL /TLS ciphers and protocols the server offers.


Posted in Security, Tech

Granular OSSEC Emails

Occasionally I see questions on the OSSEC mailing list on how to send a bunch of alerts only to a specific email address. An example for a typical use case would be different departments responsible for different groups of servers and having alerts only go to them. OSSEC has a few options for sending Alerts to specific email addresses, but it only adds those email addresses to the alert (meaning it always goes to the global email address). Sometimes this isn’t desirable.

A workaround is setting the global email recipient to a blackhole email address (something that is aliased to /dev/null on the mail server) and only using the granular settings for delivering mail.

You can then use attributes like the rule ID, group names, or event locations to split up alerts to different recipients. The downside is that by doing this, you will miss alerts with  <options>alert_by_email</options> and a low level, unless you add a few granular email alerts. Rule 1002 (catch-all $BAD_WORDS) is a good candidate you will want to keep on receiving. Rules 501-504 (OSSEC agent/master status alerts) could also be interesting; either add an <email_alert>  for each rule individually, or overwrite the rules adding  <group>ossec,</group> to then, so you can add one <email_alert>  for the group of rules.

We use this system pretty extensively assigning alerts to email groups by <event_location>  and/or <group>

An example for the email block could look like this:


Posted in Me, Tech

A script to diff files/directories on two different servers

Ok,  short one today. This is a straightforward script that simplifies comparing directories on different servers. There is no magic in it, it just rsyncs the directories to a local temp directory and runs diff against them (then deletes the directory afterwards). Mainly intended for config files, I wouldn’t recommend trying to diff gigabytes of binaries with it.


Posted in Internet Stuff, Tech

Renewing “Let’s Encrypt” SSL certificates

Let’s Encrypt provides free DV SSL certificates for everyone and is now in the open beta phase. I’m not going to go into the details of which of the clients are best, since that depends entirely on your use case (I use acme-tiny and a rule in varnish to intercept all calls to /.well-known/acme-challenge/).

Since the certificates are only valid for 90 days, I often see people suggesting to just renew them via cronjob every 2 months. I find this to be really awful advice, if that renewal fails for any reasons (network problems, local problems, problems with let’s encrypt) the next renewal is a month after the certificate expired. It is also pretty inflexible (what if you would rather prefer to renew them after 80 days).

I use openssl to check daily how long the certificate is still valid, and if a threshold has been reached it tries to renew the certificate (I believe the official client has this functionality too). And if the certificate isn’t renewed by a 2nd threshold, it sends an email altering the admin of the problem (for manually intervening and fixing whatever went wrong).

At the end of this posting I’ll add the complete script, but the quickest way to check how long a certificate is still valid is to use openssl x509 -in -checkend. It will return 0 if the file is still valid in x seconds, and 1 if the certificate doesn’t exist or if the certificate will be expired by then. Just multiply the number of days by 86400 and check if the certificate is still valid:

The openssl binary has a few nice options for looking at certificates (both local files and remotely connecting to a server and looking at the provided certificate)
Show information about a local certificate file: openssl x509 -text -noout -in
Connect to a remote server and display the certificate provided: openssl s_client -showcerts -servername -connect IP:PORT | openssl x509 -text -noout  (servername is only required if you are connecting to a server and need to use SNI to request a cert for a specific domain, i.e. a webserver providing multiple domains on port 443 via SNI. It can of course be omitted if you don’t need it.)

This is the full script I use for checking and renewing certs. It basically just loops through a list of domains, checks if any of the date thresholds are met and then renews certificates/send emails.

Posted in Programming, Tech

Convert configuration files to ansible templates

I’ve been playing around with ansible a lot lately, and I noticed that while changing stuff from “installed and configured manually” to “installed and configured by ansible” I was running into quite a few configuration files that needed to be manually turned into templates. It can be quite tedious to replace values in a configuration file with placeholders and put all those placeholders in a .yml file with default values.
Automating this is something I would have typically done in perl, but since I wanted to learn more about using regex in bash I decided to have a go at it in bash using regex and ${BASH_REMATCH}

The script takes a configuration file and spits out an ansible template, as well as the variable definitions you will need to add to your defaults/main.yml or vars/main.yml

The whole script is a bit to long to post here, but the interesting part is:

(You can download the full script here

You can use regular expressions in a [[ ]] with =~ (e.g. if [[ “boot” =~ ^b ]]), and you can access the result of the regular expression by using ( ) to mark what parts of the result to store and access them via $BASH_REMATCH (comparable to how you would do it for other languages). Here I am parsing out anything that looks like a key=value from the configfile (with multiple possible separators) and storing the results in BASH_REMATCH[1] and BASH_REMATCH[2]

Usage of the script is pretty straightforward. you give it a prefix for the variable names (so you don’t end up with multiple roles all using a common variable name like “port”), and either a local or remote file to work with, and it spits out something like this:

There a tons of different configuration file formats out there so this script won’t work perfectly 100% of the time, but it does do quite well and reduces the manually copy&pasting to a minimum.

Posted in Server, Tech

How to prevent changes to a tag via svn hook

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.


Posted in Internet Stuff, Tech

Native tcp/udp sockets in bash cheatsheet

Bash has a nifty feature to open tcp and udp connections and read/write data to that connection. The Advanced Bash Scripting Guide and the bash man page offers some information and examples, and google has some odd examples, but all in all there isn’t much variety of information on the internet on the topic. This feature is enabled in Bash at compile time with the flag –enable-net-redirections

It works by assigning a connection to a file descriptor with exec. Protocol must be udp or tcp, hostname must be either an IP or a FQDN. Use any free file-descriptor (3 or higher usually).

Use &- to close the connection instead of leaving it in the CLOSE_WAIT status.

Basic Example:

You may have noticed that the cat will hang around a while after delivering the content. As long as the connection is established it will sit there and wait for data, which can be quite a while depending on the daemon on the other end. If you want to avoid having to wait, kill or ctrl-c the cat  you can use read with an input timeout.

In the example if read has to wait longer than 2 seconds (-t 2) it will abort reading from the network connection.

If you only want to read a single line you can use head:

Although this will have the same timeout problems as cat if there is no more data but the connection is still established, it is useful and quick if you know exactly what kind of result you are expecting.

You don’t have to read the response right away, it will be buffered until you get around to accessing it (even if the other end terminates the connection).

Posted in Security, Tech

What was and wasn’t fixed in bash after the Shellshock vulnerability (CVE-2014-6271)

By now there are a few websites/blog posts online that explain CVE-2014-6271 (code being executed that was inserted after the function definition) and  CVE-2014-7169 (parser error that also led to code being executed). Both CVEs have been patched. Check out “Quick notes about the bash bug, its impact, and the fixes so far“, “Shellshock proof of concept – Reverse shell“, or “Everything you need to know about the Shellshock Bash bug” if you want more information about the details and how CVE-2014-6271 worked.

Most websites focus on the remote exploitability of the vulnerability via CGI in web servers (understandably since this is the most dangerous aspect since how request headers are passed to scripts), and it didn’t take long after the announcement on the oss-security mailing list for requests starting to hit my webservers trying to exploit the vulnerability (from what I saw either people checking “how many servers does this really affect” or the more malicious “add your server to my DDoS botnet”).

What I would like to focus on is the functionality of exporting function definitions that has been drawn into the spotlight by CVE-2014-6271. You can define a function and make it accessible to the script/child shell that you invoke. This may sound really nifty (and it is … to a certain degree). The problem is that the script has no control over what functions are being imposed upon it. It is therefore possible to override any existing command or function, even builtin functions. No matter how hard you try to keep a sane environment within your script, anyone will be able to manipulate it from the outside (e.g. by overriding unset, set, …).

A small example script that initializes a variable, prints some output and then exits.

If we execute the script we get the following output:

Looks good, the if condition will never be true. Unless of course if we start overriding functions …

Sacrificed the trap function to set i=10. Or slightly more elaborate:

When echo is called we delete our function (so subsequent calls go to the original echo, to keep the code intact), execute an echo with whatever arguments were passed, and set i=10.

Obviously this is a simple example and you could override stuff like set, declare, test, …
Another problem is that this also affects binaries that execute a system() that calls a bash script. The following is a compiled simple C binary that uses system() to execute the bash script from the beginning of this posting.

As you see the environment gets passed along to the shell script. And in my opinion this is where it can start to get ugly. There are probably more bash scripts in your $PATH than you are aware of, do a quick
for dir in ${PATH//:/ }; do egrep "^#\!.*bash" ${dir}/* ;done  and have a look for yourself.
If any of these scripts are called by a setuid/setgid binary that doesn’t sanitize and clean up the environment beforehand, you might have a serious problem on your hands.

Obviously if you have a program that is calling any kind of script, it is your job to make sure the environment is in a sane condition. At the same time I feel that having a more robust and secure way to implement exported functions would take away a lot of the pressure on the parent process ensuring sane conditions it may have itself inherited.

The topic is being extensively discussed by security experts and I expect the problem to be addressed shortly. In my opinion it should never be possible to override builtin functions externally (if a script itself wants to override a builtin, that’s fine with me, or have a switch to explicitly allow it). But since any solution will break the feature of exported functions in existing scripts it is a delicate problem to solve.

Posted in Server, Tech

Downtime, Backups, and IPv6

Sorry for the unexpected downtime that lasted a few days and affected all the services here. One of the hard drives in the raid on my server suddenly showing miserable I/O performance, since the other drive was fine according to tests, SMART and logs I decided to take the slow drive offline to have it replaced. That turned out fatal since the other drive was also kaput and all my virtual machines on that server were now corrupted.

On the bright side the 2nd MX server worked fine and held all the email like it supposed to do until I got the main server back up and running (yaay to “exim -Mvb” to at least be able to read important mails that are in the mail server queue).

Long story short, since these drives have caused me much trouble in the past too, I decided to just get a new server with different hardware and restore everything from backups. Like any sysadmin I have my regular backups and scripts to check that they are working and occasionally test them to ensure I can restore a server from backups, but I’ve never had to really actually restore everything from a backup  on one of my private servers. So when doing so I noticed minor things that aren’t perfect and needed to be changed to make my life easier in the future and decided to share my experience.

Backups. Instead of making one large archive with all the directories backed up (e.g. /etc, /home, /opt, …), split it up into multiple archives of the applications you are backing up (e.g. /etc/apache2, /var/www/). I should have known better since this is common practice at work. It makes your life a lot easier when restoring from a backup if you have one archive with everything in it associated to a specific application, Digging through a backup of /etc and trying to remember all sub-directories needed to get email back up and running was a chore (webinterface, greylisting, spamassassin, exim, dovecot … some of them store data you also want to restore in other parts of the filesystem). You can always tar all the application backups together into one package before uploading it to the backup server if you want.

Use some kind of configuration and/or deployment management software (Puppet, Chef, Ansible, Salt, own scripts, …). Don’t underestimate how relaxing it is to just press a button to get your server(s) back into the defined state you previously had, just drop in the data from the backups (configuration too depending on how you roll) and you are good to go. Getting networking up and running manually took me longer than it should have (VPN networks, routing on the VM host, sysctl settings, NAT, which interfaces are bridged, which are internal, guest network configuration, …), I’ve put that all into Ansible playbooks now so it’s just a press of a button.

IPv6 is still sorely underused in the internet. I always set it up on all my hosts and mainly I see it being used by core services like email servers, dns, package download servers of large distributions, sometimes ntp, that kind of stuff, not so much on normal websites. Ordering extra IPv4 IPs for my guests normally takes an extra day or so until they are assigned to me (and they each cost extra), so I do enjoy having a huge subnet ip IPv6 IPs free of charge to do with as I please. It’s nice to spin up a virtual server, assign it an IPv6 IP and have it online without having to worry about NAT or port forwarding or other stuff you are confronted with when getting a server/services online that reside on an internal IP.
While I like to push people to embrace IPv6 early and get used to it, it is also important to give it the same attention you give IPv4 to make sure that you aren’t opening yourself up to security problems. Make sure you have the same firewall policy for IPv6 as you do for IPv4 and IPv6 brings a few new features with it that IPv4 didn’t have. Unless you explicitly need/use them it is best to shut them off. Below I’ll ad some IPv6 settings you probably want to set per default unless you are explicitly using them.

Posted in Programming

How to install the latest Nmap for Debian/Ubuntu

A quick & dirty script to download the latest version of nmap (sourcecode) and generate a deb and install it (so that it’s correctly in the package management). Yes, I know this is not much more than a glorified configure && make && checkinstall