Posted in Internet Stuff, Tech

Renewing “Let’s Encrypt” SSL certificates

Let’s Encrypt provides free DV SSL certificates for everyone and is now in the open beta phase. I’m not going to go into the details of which of the clients are best, since that depends entirely on your use case (I use acme-tiny and a rule in varnish to intercept all calls to /.well-known/acme-challenge/).

Since the certificates are only valid for 90 days, I often see people suggesting to just renew them via cronjob every 2 months. I find this to be really awful advice, if that renewal fails for any reasons (network problems, local problems, problems with let’s encrypt) the next renewal is a month after the certificate expired. It is also pretty inflexible (what if you would rather prefer to renew them after 80 days).

I use openssl to check daily how long the certificate is still valid, and if a threshold has been reached it tries to renew the certificate (I believe the official client has this functionality too). And if the certificate isn’t renewed by a 2nd threshold, it sends an email altering the admin of the problem (for manually intervening and fixing whatever went wrong).

At the end of this posting I’ll add the complete script, but the quickest way to check how long a certificate is still valid is to use openssl x509 -in -checkend. It will return 0 if the file is still valid in x seconds, and 1 if the certificate doesn’t exist or if the certificate will be expired by then. Just multiply the number of days by 86400 and check if the certificate is still valid:

The openssl binary has a few nice options for looking at certificates (both local files and remotely connecting to a server and looking at the provided certificate)
Show information about a local certificate file: openssl x509 -text -noout -in
Connect to a remote server and display the certificate provided: openssl s_client -showcerts -servername foo.bar -connect IP:PORT | openssl x509 -text -noout  (servername foo.bar is only required if you are connecting to a server and need to use SNI to request a cert for a specific domain, i.e. a webserver providing multiple domains on port 443 via SNI. It can of course be omitted if you don’t need it.)

This is the full script I use for checking and renewing certs. It basically just loops through a list of domains, checks if any of the date thresholds are met and then renews certificates/send emails.

Posted in Internet Stuff, Tech

Native tcp/udp sockets in bash cheatsheet

Bash has a nifty feature to open tcp and udp connections and read/write data to that connection. The Advanced Bash Scripting Guide and the bash man page offers some information and examples, and google has some odd examples, but all in all there isn’t much variety of information on the internet on the topic. This feature is enabled in Bash at compile time with the flag –enable-net-redirections

It works by assigning a connection to a file descriptor with exec. Protocol must be udp or tcp, hostname must be either an IP or a FQDN. Use any free file-descriptor (3 or higher usually).

Use &- to close the connection instead of leaving it in the CLOSE_WAIT status.

Basic Example:

You may have noticed that the cat will hang around a while after delivering the content. As long as the connection is established it will sit there and wait for data, which can be quite a while depending on the daemon on the other end. If you want to avoid having to wait, kill or ctrl-c the cat  you can use read with an input timeout.

In the example if read has to wait longer than 2 seconds (-t 2) it will abort reading from the network connection.

If you only want to read a single line you can use head:

Although this will have the same timeout problems as cat if there is no more data but the connection is still established, it is useful and quick if you know exactly what kind of result you are expecting.

You don’t have to read the response right away, it will be buffered until you get around to accessing it (even if the other end terminates the connection).

Posted in Internet Stuff

Virtualbox to 4.2.12 problems with Ubuntu

After updating Virtualbox to 4.2.12 and updating the guest additions on my Ubuntu 12.10 the guest suddenly started having problems with the graphics driver (X.org wouldn’t start correctly, only starts in “rescue” mode). I did a bit of googling, and it seems other users are also affected (mainly 12.04 and 12.10 Ubuntu), and it seems to affect people who upgraded from virtualbox guest addditions 4.2.10 -> 4.2.12

Here the quick and dirty workaround to the problem: download the 4.2.10 guest additions and use them.

Will need to reboot or make sure the new modules are loaded and restart X afterwards.

Posted in Internet Stuff, Security, Tech

Script to start minion in tmux

Minion is a security project from Mozilla (link). It provides a user-friendly web interface to various security scanner tools. There is a webcast demonstrating the software (link).

The software requires a few services to run, and since I like having one script take care of starting everything with the right parameters, I threw together a simple shell script that sets up a tmux session with the services started in windows with the names of the services.

Posted in Internet Stuff, Programming

How to break down CIDR subnets in Bash

I was playing around with subnets in bash recently and needed an elegant/easy way to split up a subnet into smaller subnets. First I used 2 functions I found on stackoverflow.com to convert an IP addresse to and from an integer. After that it was just a bit of math in bash to split up any networks too big.
Any network larger than $maxSubnet gets split up.
Here the useful code:

Output of script:

 

Posted in Internet Stuff, Programming

How to get the intersecting area of two polygons in MySQL

I was playing around with spatial features of MySQL this weekend and stumbled into a problem where I was looking for the area of two rectangles that overlap.  MySQL provides a function to check if they overlap, but no function to extract the region that overlaps.

I’ve never written a stored routine in MySQL before, so I decided it would be a good exercise to try making one. As you can see the function is pretty straightforward and it assumes you are working with rectangles, but other than that it does what it is supposed to.
You pass the function 2 polygons (e.g. Intersection(a.poly,b.poly)), and it returns the intersecting area as a new polygon.

Example comparing some rectangles in 2 tables using the function:

Result:

 

Posted in Internet Stuff, Security

Updated OSSEC Web UI 0.3 files for OSSEC 2.6

OSSEC is an open source HIDS (Host-based Intrusion Detection System), and a pretty darn good one too. It also has a simple web front-end to view what’s going on, search through alerts and stuff like that  (called OSSEC Web UI, I’ll just call it “WUI” here). Unfortunately the code is a bit outdated (the last official update was from 2008 as far as I can tell) and it doesn’t support newer features of OSSEC like polling data from a database. Something I’d like to tackle if I find the time 😉

The latest version of OSSEC is 2.6, and due to some small changes to the format of the logs WUI no longer works out-of-the-box. I had a look at the code this weekend and am providing patches and downloads of the files needed to change to get everything running again with OSSEC 2.6.

List of changes:

  • Works with the OSSEC 2.6 alert log file format
  • Changed Rule ID Link to better work with the new documentation wiki
  • Added “user” field to alert output
  • Widened the layout by a few pixels (to 1000px) and changed the CSS / alert layout to make the individual alerts better readable
  • Moved some of the hardcoded formatting to CSS

Download Download changed files
Download Patch download

 

Posted in Internet Stuff

Playing around with Exim, Spamassassin, DKIM and SPF

I was playing around with my mail server today and decided to have a look at DKIM and SPF. Jump to the bottom for some useful links. You will have to be able to change TXT entries for your domain/subdomains to do any of this. Sending a mail to check-auth@verifier.port25.com will generate an auto-reply with the result of your mail servers settings. I also noticed that Exim wasn’t listening to the IPv6 interface and promptly fixed that, and have received my first mail via IPv6.

Some spamassassin scores for your local/user settings:

SPF:
Syntax: http://www.openspf.org/SPF_Record_Syntax
Tools & Information: http://www.openspf.org/
Wizard: http://www.spfwizard.net/

DKIM:
Exim walkthrough: http://mikepultz.com/2010/02/using-dkim-in-exim/
Domain specific settings: http://www.debian-administration.org/users/lee/weblog/41
Tools: http://dkimcore.org/tools/
Wizard: https://www.port25.com/dkim-wizard/

Posted in Internet Stuff, Server

New Server (with IPv6)

I got a new server and I finished moving everything to the new hardware this week. If you notice anything behaving strange just drop me a note. I also set up IPv6 on all my VMs so will be playing around with that a bit. It’s a shame that hardly any providers give customers IPv6 addresses, migrating from IPv4 to IPv6 will take forever if none of the users can use them (and in the meantime IPv4 addresses are getting more and more expensive for me).

The new server is from hetzner since I have been very satisfied with them in the past. Impressive hardware features for low prices and (what really makes me keep choosing them) fantastic technical support. I’ve been with a few companies in the past, but none have had as good support as hetzner. Sooner or later hardware will die, and having a hosting provider that responds promptly and knowledgeably to request is priceless.

Posted in Internet Stuff, Programming, Tech

How to build an efficient GeoIP SQL table

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

Posted in Internet Stuff

How to add file versions/history to iFolder

I’ve been using iFolder for about a month now and am pretty satisfied with the performance and features, but there is one feature I dearly miss: having access to older versions of a file.  To work around this problem I simply created a SVN repository for every iFolder user.  This allows the users to “check in” files whenever they want to save a specific version of a file, they can tell SVN to ignore directories that they don’t need or want in the SVN system. And you can easily browse through the history of a file, pulling up versions from specific dates, or seeing what changed when in the file (obviously not for binary data).

I am fully aware that SVN may be a bit overkill and that I am only using a very small subset of the SVN features, but on the other hand it is very easy to set up, and there are clients for all the major OS out there that integrate seamlessly into the file managers. So even if I don’t need all the bells and whistles it does what I need it to do without any worries.

As with all workarounds this of course brings some limitations and implications with it.

  • First of all you will only have the history of a file on systems with a svn client, so no history if you are accessing the files via webinterface.
  • Situations where multiple people have access to the same files must be set up in the same consellation in SVN to ensure all users have access to file histories
  • Due to the usage of two systems (iFolder and SVN) the files will be stored in both systems
Depending on your setup and needs the implications may turn out to be a dealbreaker. But if not it may be a viable “addon” to add some features you otherwise missed. I’m not going to go into “how to setup SVN”, since anything I write would be tuned to a specific installation. Just use google, there are plenty of Howtos out there, just find the one that meets your system.
Posted in Internet Stuff, Server

How to install ruby 1.9.2 on Ubuntu 10.04

The current LTS version of Ubuntu is 10.04 and the most current version of ruby it ships with is 1.9.1. Unfurtunately 1.9.1 wasn’t that great of a release and anyone using the 1.9 branch really should use the stable 1.9.2.

After doing a bit of researching I found some information on how the best approach to get ruby installed is. Downloading the source, compiling it and registering the installed version with the package manager.

The following little bash script takes care of installing ruby 1.9.2 on a ubuntu or debian based system (or any other version if you change the $Version variable in the script). The script just consolidates information found online and wraps it up into a nice bashscript

 

Posted in Internet Stuff, Server

Using iFolder as an open source Dropbox alternative

I’ve had a few posting here in my blog over the years about trying to keep data synced over various computers. Nowadays it has gotten easier since it seems everyone is chruning out their own online storage. Classics like Dropbox or SugarSync, and then we have the big companies  Google, Microsoft and Ubuntu trying their luck. Basically they all provide you with 2-5Gb free storage, and more space and features for more money. But they all have one downside, whether encrypted or not, you are leaving your own files to the mercy of an external company. There is not really any voodoo in the whole system, basically you just have some space online, a bit of encryption, and some way to ensure that all clients of a user are synced, nonetheless building your own service can be a hassle as I have found out.

Currently I’m using SVN as a workaround. I just sometimes forget to commit, which leads to me having old data and no access to updated files. Instead of starting from scratch and build strange stuff with rsync I had a look if there were any open source solutions out there. And I was pleased to find a few. Unfortunately they are almost all in the “I have an idea and a few mockups” stage, maybe even some beta version that does something, but nothing really “production” level. Another downside was that most of them either supported Linux or Windows, but often not both.

One open source solution I stumbled across is iFolder,  seasoned software supported by Novell, with clients for Windows, Linux and Mac. And if all else fails you can use the web interface to access your files. Admin and User web interface are both easy to use. I don’t remember how I stumbled across it, but I do know that it’s not one of the main products you see named when searching for alternatives to Dropbox.

If you have SuSE as an OS, then installing the software is immensely simplified due to SuSE belonging to Novell, just follow the SuSE instructions on the website. I prefer debian based installations and found Ubuntu instructions at help.ubuntu.com that were written mainly for Ubuntu, but should work with any debian deviate.

I’m not going to repeat anything from the installation instructions, they already did a fine job of documenting everything. One thing is important however: the Client 3.8.0.0 just plain doesn’t work. unfortunately it is the one linked from the website. Go to the sourceforge repository, and download the newer 3.8.03 client. That one works like a charm.

Posted in Internet Stuff, Server

How to get Teamspeak 3 running on a current Linux

Teamspeak is know for lagging a bit behind with development.

The last few days I have been upgrading my servers to current distributions, today the Voice servers were on the list to get Debian 6 / Ubuntu 11.04. And again I ran into problems with Teamspeak, turns out they won’t work with libmysqlclient 16 libraries and require the good old 15 version (which isn’t available out-of-the-box in the latest Debian and Ubuntu release).

So anybody running into the same problem (do a ldd libts3db_mysql.so to check), can hop on over to http://packages.debian.org/lenny/libmysqlclient15off and download the package for your architecture and install it with dpkg -i

Posted in Internet Stuff, Security

What plugins is that website running?

While having a look at nikito yesterday I stumbled accross cms-explorer. It’s an interesting little program that checks the themes/modules/plugins installed in common CMS systems (Drupal, WordPress, Joomla! and Mambo), with automatic exploration for Drupal and WordPress. It also has some nice bonus features like providing a list of known issues for plugins found by accessing the OSVDB.org database.

Example output:

Running it against my own webspace revealed a possible SQL injection I was unaware of. *) Fixed that, will probably replace that plugin completely this week, anything that has stuff so obviously bad in it is generally not all too sane.

*) I normally look at plugins before I install them, must have missed this one. @ PHP programmers: anyone who passes on the content of a $_REQUEST directly to a SQL query without any sanity checking deserves to be flogged with his own code.

Posted in Internet Stuff, Programming

Disemvoweling

Talk about weird words … ok, according to Wikipedia disemvoweling is the term for replacing or removing vowels from words. Commonly used as a tool for moderating.  I’m pretty sure everyone has run across  certain disemvoweled  words on the internet like f*ck or sh*t. Anyway I went and made a pure html/javascript page that does just that, removes any vowels from an inputted text. The usefullness can certainly be argued, it was more for me to brush up on my javascript and css skills.

http://www.dopefish.de/projects/disemvoweling/

Posted in Internet Stuff, Tech

Virtualbox update

Anyone running Ubuntu as a Virtualbox guest is advised to update to Virtualbox 4.0.6 (+ the extensions) that was released today.  Don’t forget to recompile the guest additions after upgrading to 4.0.6. Besides the usual stuff in the changelog, the update fixes a problem with screen resolution in Ubuntu 11.04. Since the Ubuntu update is just around the corner updating Virtualbox beforehand will prevent a bit of hassle.

Posted in Internet Stuff, Programming

captcha cracking

This is a pretty old posting from 2009 I just recently discovered in my “drafts” directory. Nowadays there are probably easier and more elegant ways of defeating a captcha, but for old times sake, here is my simple approach.
———————–

Eclectic and Marko were so kind as to “provide” me a captcha to play around with. Took me a few days of poking around and googling but in the end it was easier than I had thought. As long as there aren’t and logic errors in the code (e.g. bad or no session handling) you probably won’t get around some kind of OCR. As OCR software I decided to use gocr because it is free, runs under linux, and it is fairly easy to train to specific needs. Because I knew which libraries were being used to create the captcha images, it was possible for me to build a testing area. This just speeds things up a bit, the process would have worked just as well off the original website. First off: the spambot in action -> http://captcha.dopefish.de/spambot.php, and the website it accesses: http://captcha.dopefish.de/

Now I’ll describe the steps I took to defeat the captcha. Look at what happens on failed and successful inputs, first write a script that works if you enter the solution manually. I used the following 2 php functions for getting and posting stuff (and keeping the session intact)

Now train a gocr database for the images. Obviously it get’s better the more you train it.
Since curl is taking care of  session handling, we can use the get_url() function for downloading the captcha image. I pipe it through this shell command to make it easier for gocr to read:

It turnes this:

into this:

Since the valid captcha result is always the same length, we can check if gocr matched all the chars. If it looks good we can use post_url() to continue our session and throw all the fields at the form and submit it. See, wasn’t that hard. Most of the time is spent training gocr and converting the image into something easier to read. It doesn’t solve 100% of the images, more like 80-90%, but still better than nothing ;-).