How to check if a IP (ipv4) address is valid in pure Bash

Here is a small bash function to check if a IP is valid (4 octets, each octet < 256). I find it somewhat elegant since instead of using a lot of case/if/then constructs or a crazy long regex it splits the IP into each octet (and stores them in an array, and then uses a combination of regex and bit shifting to check each octet.

The function will return 0 if the IP is valid, and 1 or higher if it encountered an error (you can check with the $? variable directly after calling the function)
Example:

How to use Cluster SSH (cssh) and Mosh together

A colleague told me about mosh today, just saying “it’s an alternative to SSH that addresses many of the problems” doesn’t do it credit. Go look at the page and have a look, don’t worry it’s a nice page and I’ll still be here waiting when you come back. I do a lot of work on servers across the globe, and believe me, although subtle, the difference between a 10ms connection and a 100+ ms connection is definitely noticeable 😉 That lag all but disappears with mosh. Mosh doesn’t support X11/SSH-Agent forwarding yet, but it’s on their roadmap.

Back to topic, I also use cssh for working on multiple servers simultaneously. I’d recommend making a copy of your .cshrc file (cp -a ~/.csshrc ~/.csshrc_mosh) and working on the copy. Edit the  ~/.csshrc_mosh and change the lines ssh= and ssh_args= (diff of my files):

You will have to remove any parameters in ssh_args= and move them to your ~/.ssh/config due to the difference in command line parameters between mosh and ssh. Now all you have to do is call cssh with the new config file, e.g. with an alias to make things easier alias mcssh=”cssh -C ${HOME}/.csshrc_mosh” And your done.

New Virtualbox version, script for easy update of extension pack

VirtualBox 4.1.10 was released yesterday with a few nice things in the changelog. Updating virtualbox itself is easy, just download the package and update it. Since I seem to stumble over the update of the extension pack every time (on my headless system) I thought I’d write a small script this time so I don’t have to rethink it next time (automatically downloads and installs the current extpack):

 

How to add locking to a shell script (the easy way)

I haven’t posted anything with bash here for a while, so today I’ll throw in a little snippet to use flock to make sure a script is only running once.  This is very handy in cron jobs that you want to run often, but there shouldn’t be multiple instances of the script running at the same time.
Since it is small and easy I’d recommend adding it to any code you don’t want running multiple times since “that script” you just wrote, that runs 10 minutes now, might turn into a monster in 6 months and run 45 minutes when things change (data grows, more stuff to do).  Better safe than sorry.

Basically we rely on flock to do the heavy lifting and we just add some logic around it:

man flock will show you more details to the parameters used and even some examples. Basically it will use trap to make sure the lock is released if the script ends in any way. 200 is a random file descriptor I chose for this example, it can be anything numeric. flock -xn means it will attempt to acquire an exclusive lock, and if that fails it will exit with an error.

Putting this somewhere at the top of your script will simply end the script if it finds an existing lock. flock has a few other options like -wait or nor using -n that allow you to not exit but wait for the lock to end (with wait a variable amount of seconds). And thus with a bit of creativity enabling you to only lock specific parts of the code (e.g. database calls, file changes, …) and handling failed lock attempts more gracefully than an exit.

How to build an efficient GeoIP SQL table

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

How to install ruby 1.9.2 on Ubuntu 10.04

The current LTS version of Ubuntu is 10.04 and the most current version of ruby it ships with is 1.9.1. Unfurtunately 1.9.1 wasn’t that great of a release and anyone using the 1.9 branch really should use the stable 1.9.2.

After doing a bit of researching I found some information on how the best approach to get ruby installed is. Downloading the source, compiling it and registering the installed version with the package manager.

The following little bash script takes care of installing ruby 1.9.2 on a ubuntu or debian based system (or any other version if you change the $Version variable in the script). The script just consolidates information found online and wraps it up into a nice bashscript

 

How to easily add colored text output in bash scripts

Here is small snippet that can give your shell scripts some nice output: colortext.sh As with the debug.sh script, just download it to the same directory as your own script and add it with

It contains one simple function called text with the syntax text “text to be output”. Color can be red, green, yellow, blue or grey. The function does not automatically add a linebreak to the putput, so pop a \n in there if you need it. I prefer using it together with printf for clean and easy color output.

Here are some examples of how the function can be used, and below the corresponding output:

Output:

normal text
blue text, yellow text
Status of script: [ERROR]
Status of script: [OK]

How to add debugging to shellscripts

Debugging bash scripts is pretty straightforward, throwing around a couple echo and set -x quickly gives you what you need. But what if you want to add a nice breakpoint,  debugging to lots of paces in the code or turn all debugging on or off at once? Then this little script I wrote is the right thing for you: debug.sh just download it to the same directory as your script and include it with the following line:

It contains 4 simple functions that will make your bash coding easier.
debug and breakpoint both print the argument with a timestamp to STDERR
You can turn off all the functions by adding a DEBUG=false into your code

Example:

Output:

Wireshark remote capturing

yeah, this is real simple stuff, not really worth writing a script for it. but on the other hand it saves me from remembering how to do it every time I need it (which isn’t often). So here is a little script to setup remote capturing with wireshark.
All it basically does is ssh to the remote host and tcpdump sucking the output via stdout through the ssh connection to a local pipe, that is then used by wireshark to display the stream. Because of this you may want to make sure you aren’t capturing your own ssh data when doing this 😉

XEN 3.4 with ipv6 routing

Yes, there are a few postings out there about getting ipv6 routing running with XEN. But I’ll throw this online anyway since there are a few changes I had to make for it to work on my server. This text is intended for people who know their way around Linux and XEN so it will be a bit technical and won’t spell out every single step you have to make.

Most of the changes are based off scripts and information from BenV and wnagele (latter is interesting for me since I am also running XEN on a hetzner server). Have a look at the two links if anything is unclear. Now let’s start the fun 🙂

First of all we need IPv6 up and running on the host (dom0). Add the IP and gateway to your /etc/network/interfaces
This is what mine looks like:
iface eth0 inet6 static
address 2a01:4f8:100:1123::2
netmask 64
gateway 2a01:4f8:100:1120::1
pre-up ip -6 route add 2a01:4f8:100:1120::1 dev eth0

Check if the IP address is responding to the outside world (e.g. with wiberg.nu/iptools.php), if everything looks ok, proceed …
Now we need to enable a few things to get routing and neighbor discovery running on the host (dom0). Edit your /etc/sysctl.conf and add/change these 2 entries (and while you are at it, set them with “sysctl -w” too):
net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.proxy_ndp=1

So, your host should by now be online with ipv6 and soon be able to route packets to it’s guests. By default XEN will only take care of IPv4 when a guest is created, so here is a small patchfile that adds support for IPv6: xen-ipv6-vif-route.patch. The patch changes vif-route and vif-common.sh, while these files may be in different places depending on your distribution, /etc/xen/scripts/ is where they can commonly be found. Download the patch to the directory with the scripts to be changed and execute a “patch -p0 < xen-ipv6-vif-route.patch” (vif-common.sh gets a few new IPv6 functions, and iptables now won’t try to change stuff for IPv6 IPs. vif-route changes are: ndp is enabled for the vif device and the route/neighbor IPv6 settings are set)

So, now that the scripts know how to setup all our IPv6 needs, we need to add the IPv6 IP to our guest settings (.cfg file typically found in /etc/xen/). What we want to change is the “vif” setting. Add the IPv6 IP of the guest to the IPv4 IP (just the IP without the trailing /network, space separated form the IPv4 IP):
vif = [ 'mac=B1:A3:3F:25:11:B8, ip=2a01:4f8:100:1123::5 188.40.34.101' ]

Now you can create the guest(domU) and add the IPv6 IP to the /etc/network/interfaces of the guest if you haven’t so already (it uses the host (dom0) as the gateway).

iface eth0 inet6 static
address 2a01:4f8:100:1123::5
netmask 64
gateway 2a01:4f8:100:1123::2

Restart the networking on the guest (or reboot it) and you should now be able to ping the guest from the internet. See, easy wasn’t it 🙂

Script of the day – clean up stale .ssh/known_hosts

This little script takes an IP or hostname as a parameter, and if there is an offending key in the .ssh/known_hosts it removes it and replaces it with the current valid one useful if you are moving/reinstalling a large amount of servers …

Checking a list of IPs against RBL

This is more a reminder to myself than anything else … this is small snippet that takes a list of IPs and does a whois on all that aren’t in a RBL

Lets say we have al list of IPs in a file “iplist.txt”:
12.172.121.171
24.149.208.68
38.105.100.9
58.185.207.86
61.201.51.66
64.78.164.169

Snippet that checks the IPs (can of course be easily changed to check IPs that are IN a RBL)

bash: using the content of a variable as variable name

Since the implementation of Arrays in Bash is somewhat lacking compared to higher level programming languages (only one-dimensional), and hash lists require a bit of work to set up, you may run into a situation where you have a small list of key/value pairs that are both variable and you need to store.
There are various solutions for the problem, e.g. creating two arrays (one for the keys, one for the values, and combining them by using the same index values for the entries), or using the functions from the link above to build a hash list. For me the easiest way to solve the problem, if I only have a few variables and don’t want to bloat the code, is to (mis)use declare. declare is intended for setting the type of a variable (constant, array, integer,…), but has the nice side affect that you can use variables in the key name, and you can set the value of the variable.

usage:
declare ${Key}=${Value}

Example:
$File_Config is variable holding the name of a configfile, the content of the file could look like this:
foo:one
bar:three
foobar:bignumber

Snippet:

after the snippet has read the configfile, you can use $Configuration_foo, $Configuration_bar and $Configuration_foobar in your script. The keynames could also have came from a mysql query, array, command line args, …

Mythtv and IR Blaster

We use a set-top-box as the video source for mythtv. And while this works perfectly fine, we have to manually make sure the right channel is set for what mythtv wants to record. This can turn out to being a pain at time, especially when multiple things are planned to be recorded on different channels. So I went and ordered a IR transmitter from irblaster.info to give mythtv the possibility to change channels itself.

Since a bunch of the infos I found online were a bit out dated, here are a few tips for anyone using current versions of the software (I’ve got mythtv-backend 0.22 and lirc 0.8.4a running on a mythbuntu based system when I wrote this)

Hardware is easy to set up, just plug it into a free serial port.
1st we will head over to http://lirc.sourceforge.net/remotes/ and grab the file for our reciever set-top-box if we don’t already have them.
Then go edit /etc/lirc/hardware.conf and add the transmitter settings, these were mine:
TRANSMITTER="Skymaster_XL10"
TRANSMITTER_MODULES="lirc_dev lirc_serial"
TRANSMITTER_DRIVER=""
TRANSMITTER_DEVICE="/dev/lirc1"
TRANSMITTER_LIRCD_CONF="skymaster/XL10"
TRANSMITTER_LIRCD_ARGS=""

now include the reciever settings to /etc/lirc/lircd.conf
I edited my /etc/init.d/lirc to execute the following line before starting the daemon (was needed to ensure the lirc_serial module can be loaded):
/bin/setserial /dev/ttyS0 uart none
Restart lircd and you can then start testing if transmitting codes work with irsend works. here is a little wrapper script I wrote around irsend for mythtv to use. You may have to twiddle around with the sleeps and change the $Remote to your reciever. If the sleeps are too long (-> the script takes to long to finish switching channels), then mythtv will timeout and not display live tv.

If everything is working fine, then go ahead and tell mythtv to use the script to change channels. This is done in the tuner card setup of mythtv-backend (mythtv-setup).

bash scripting … switching into directory of the script

Sometimes it is useful to switch into the directory of the script e.g. when we need to call or include further files and don’t want to go through the hassle of searching for the script in the file system. Especially when symlinks are involved everything get a bit more interesting. This little snippet switches into the directory of the script, using readlink to dereference symlinks if the script is called via a symlink.

Bash scripting, traps and sleep

Today I ran into any old problem: you have a script that should do something when it recieves a signal (e.g. if someone sends it USR1 it should write to a log/syslog), but the script uses a long sleep because it normally only checks/calculates stuff every x min. If you send it a kill -USR1 $pid it will normally execute the trap AFTER the sleep is done, not so great. I figured of the following solution today: put the sleep in a while loop that checks if the full time was slept, and inside the loop a sleep that sleeps the X seconds remaing in the background followed by a wait.

If the script now recieves a USR1 it can kill the wait, execute the trap and will continue the remaining sleep on the next iteration of the loop.

Reboot Script for Linksys WAP200 access points

Since the Linksys WAP200 has a tendency to hang and not let any users connect anymore, this little script to reboot an accesspoint (webinterface still works fine). Replace user and password with correct values.

bash foo

Instead of writing tons and tons of small scripts < 10 lines that all depend on each other I have the habbit of writing them as functions and throwing them into one file (easier to edit, less worries about them depending on each other). To be able to use the functions from the bash shell, just add $(basename $0) “$@” to the END of the file, and add symlinks function_name -> file.

Example:
$ ln -s foo.sh foobar

$ cat foo.sh


rrdtool tricks / tips

There are various different ways to generate rrdtool images. And most people will probably be generating elaborate ones out of perl or php scripts. The ones I am currently generating are built in a shell script (bash) and I was really having a problem with quotes and white space here for a while. If you generate the query and save it in a variable, the quotes go to hell and your white space (e.g. a space in a COMMENT) starts causing problems. my solution may not be the best, but it works without any trouble: Throw the whole command line into a temporary file and then use eval to create the graph. The following is not a complete statement, just a few lines as an example.

self0wned

Doh, I was just writing a bash script to calculate the current network traffic and write it to syslog. nothing special. I’m posting this from my laptop because I was forced to reboot my main PC after accidently writing a fork bomb. I wanted to recursively call myself and detatch the process so I could use the script as a deamon without worring about it if I logout.

AFTER starting the script, I noticed a flaw in my “if” startement to make sure I don’t DOS myself. By the time I got around to “hmm, maybe I should check the pid of that child and kill it so it doesn’t go and suck up my resources” it was too late. duh.