What was and wasn’t fixed in bash after the Shellshock vulnerability (CVE-2014-6271)

By now there are a few websites/blog posts online that explain CVE-2014-6271 (code being executed that was inserted after the function definition) and  CVE-2014-7169 (parser error that also led to code being executed). Both CVEs have been patched. Check out “Quick notes about the bash bug, its impact, and the fixes so far“, “Shellshock proof of concept – Reverse shell“, or “Everything you need to know about the Shellshock Bash bug” if you want more information about the details and how CVE-2014-6271 worked.

Most websites focus on the remote exploitability of the vulnerability via CGI in web servers (understandably since this is the most dangerous aspect since how request headers are passed to scripts), and it didn’t take long after the announcement on the oss-security mailing list for requests starting to hit my webservers trying to exploit the vulnerability (from what I saw either people checking “how many servers does this really affect” or the more malicious “add your server to my DDoS botnet”).

What I would like to focus on is the functionality of exporting function definitions that has been drawn into the spotlight by CVE-2014-6271. You can define a function and make it accessible to the script/child shell that you invoke. This may sound really nifty (and it is … to a certain degree). The problem is that the script has no control over what functions are being imposed upon it. It is therefore possible to override any existing command or function, even builtin functions. No matter how hard you try to keep a sane environment within your script, anyone will be able to manipulate it from the outside (e.g. by overriding unset, set, …).

A small example script that initializes a variable, prints some output and then exits.

If we execute the script we get the following output:

Looks good, the if condition will never be true. Unless of course if we start overriding functions …

Sacrificed the trap function to set i=10. Or slightly more elaborate:

When echo is called we delete our function (so subsequent calls go to the original echo, to keep the code intact), execute an echo with whatever arguments were passed, and set i=10.

Obviously this is a simple example and you could override stuff like set, declare, test, …
Another problem is that this also affects binaries that execute a system() that calls a bash script. The following is a compiled simple C binary that uses system() to execute the bash script from the beginning of this posting.

As you see the environment gets passed along to the shell script. And in my opinion this is where it can start to get ugly. There are probably more bash scripts in your $PATH than you are aware of, do a quick
for dir in ${PATH//:/ }; do egrep "^#\!.*bash" ${dir}/* ;done  and have a look for yourself.
If any of these scripts are called by a setuid/setgid binary that doesn’t sanitize and clean up the environment beforehand, you might have a serious problem on your hands.

Obviously if you have a program that is calling any kind of script, it is your job to make sure the environment is in a sane condition. At the same time I feel that having a more robust and secure way to implement exported functions would take away a lot of the pressure on the parent process ensuring sane conditions it may have itself inherited.

The topic is being extensively discussed by security experts and I expect the problem to be addressed shortly. In my opinion it should never be possible to override builtin functions externally (if a script itself wants to override a builtin, that’s fine with me, or have a switch to explicitly allow it). But since any solution will break the feature of exported functions in existing scripts it is a delicate problem to solve.

Downtime, Backups, and IPv6

Sorry for the unexpected downtime that lasted a few days and affected all the services here. One of the hard drives in the raid on my server suddenly showing miserable I/O performance, since the other drive was fine according to tests, SMART and logs I decided to take the slow drive offline to have it replaced. That turned out fatal since the other drive was also kaput and all my virtual machines on that server were now corrupted.

On the bright side the 2nd MX server worked fine and held all the email like it supposed to do until I got the main server back up and running (yaay to “exim -Mvb” to at least be able to read important mails that are in the mail server queue).

Long story short, since these drives have caused me much trouble in the past too, I decided to just get a new server with different hardware and restore everything from backups. Like any sysadmin I have my regular backups and scripts to check that they are working and occasionally test them to ensure I can restore a server from backups, but I’ve never had to really actually restore everything from a backup  on one of my private servers. So when doing so I noticed minor things that aren’t perfect and needed to be changed to make my life easier in the future and decided to share my experience.

Backups. Instead of making one large archive with all the directories backed up (e.g. /etc, /home, /opt, …), split it up into multiple archives of the applications you are backing up (e.g. /etc/apache2, /var/www/). I should have known better since this is common practice at work. It makes your life a lot easier when restoring from a backup if you have one archive with everything in it associated to a specific application, Digging through a backup of /etc and trying to remember all sub-directories needed to get email back up and running was a chore (webinterface, greylisting, spamassassin, exim, dovecot … some of them store data you also want to restore in other parts of the filesystem). You can always tar all the application backups together into one package before uploading it to the backup server if you want.

Use some kind of configuration and/or deployment management software (Puppet, Chef, Ansible, Salt, own scripts, …). Don’t underestimate how relaxing it is to just press a button to get your server(s) back into the defined state you previously had, just drop in the data from the backups (configuration too depending on how you roll) and you are good to go. Getting networking up and running manually took me longer than it should have (VPN networks, routing on the VM host, sysctl settings, NAT, which interfaces are bridged, which are internal, guest network configuration, …), I’ve put that all into Ansible playbooks now so it’s just a press of a button.

IPv6 is still sorely underused in the internet. I always set it up on all my hosts and mainly I see it being used by core services like email servers, dns, package download servers of large distributions, sometimes ntp, that kind of stuff, not so much on normal websites. Ordering extra IPv4 IPs for my guests normally takes an extra day or so until they are assigned to me (and they each cost extra), so I do enjoy having a huge subnet ip IPv6 IPs free of charge to do with as I please. It’s nice to spin up a virtual server, assign it an IPv6 IP and have it online without having to worry about NAT or port forwarding or other stuff you are confronted with when getting a server/services online that reside on an internal IP.
While I like to push people to embrace IPv6 early and get used to it, it is also important to give it the same attention you give IPv4 to make sure that you aren’t opening yourself up to security problems. Make sure you have the same firewall policy for IPv6 as you do for IPv4 and IPv6 brings a few new features with it that IPv4 didn’t have. Unless you explicitly need/use them it is best to shut them off. Below I’ll ad some IPv6 settings you probably want to set per default unless you are explicitly using them.

How to install the latest Nmap for Debian/Ubuntu

A quick & dirty script to download the latest version of nmap (sourcecode) and generate a deb and install it (so that it’s correctly in the package management). Yes, I know this is not much more than a glorified configure && make && checkinstall

Bash snippet, verify ctrl+c

Lately I’ve been working on a pair of more elaborate scripts using ncat and openssl to transfer data between hosts. I’ll get around to posting it eventually, but until then a few small snippets that people may find useful.

Today we will catch ctrl+c and ask the user if he really want’s to terminate the script.

The initialize() and cleanup() are my usual function names I have in every script, making sure general settings and variables are defined and that on exit any tempfiles get deleted.
What has been added was a trap for the INT signal (ctrl+c) which calls the verify_quit() function, giving the user 10 seconds to press ctrl+c again to exit (via cleanup()) or return back to wherever we were in the code. There is one unavoidable caveat, the first ctrl+c will kill whatever the script was doing before it jumps into the verify_quit() function.

Simple “try” function for bash

Made a nice little try() function today for simplifying checking/dealing with  return codes from commands. It uses the function text() I posted earlier to colorfy output: How to easily add colored text output in bash scripts. The function accepts 2 parameters, how it should behave if a problem occurs and the command to be executed: try <silent|warn|fatal> command

silent: save the return status in the global variable command_status
warn: if the command has a return code > 0, print a warning and save the return status in the global variable command_status
fatal: if the command has a return code > 0, print an error and exit

Obviously not as versatile as a python try/except, bu streamlines verifying the command return codes.
Example Usage:

Output
Warning: ‘false‘ failed with return code –1
ls: cannot access doesnotexist: No such file or directory
Error: ‘ls -al doesnotexist‘ failed with return code –2

File: error_handling.sh