bash – Page 5 – Ryan Schulze

How to check if a IP (ipv4) address is valid in pure Bash

July 25, 2012August 26, 2013Ryanbash, ipv4

Here is a small bash function to check if a IP is valid (4 octets, each octet < 256). I find it somewhat elegant since instead of using a lot of case/if/then constructs or a crazy long regex it splits the IP into each octet (and stores them in an array, and then uses a combination of regex and bit shifting to check each octet.

is_valid_ipv4() {
  local -a octets=( ${1//\./ } )
  local RETURNVALUE=0

  # return an error if the IP doesn't have exactly 4 octets
  [[ ${#octets[@]} -ne 4 ]] &amp;&amp; return 1

  for octet in ${octets[@]}
  do
    if [[ ${octet} =~ ^[0-9]{1,3}$ ]]
    then # shift number by 8 bits, anything larger than 255 will be > 0
      ((RETURNVALUE += octet>>8 ))
    else # octet wasn't numeric, return error
      return 1
    fi
  done
  return ${RETURNVALUE}
}

is_valid_ipv4() {

local -a octets=( ${1//\./ } )

local RETURNVALUE=0

# return an error if the IP doesn't have exactly 4 octets

[[ ${#octets[@]} -ne 4 ]] && return 1

for octet in ${octets[@]}

if [[ ${octet} =~ ^[0-9]{1,3}$ ]]

then # shift number by 8 bits, anything larger than 255 will be > 0

((RETURNVALUE += octet>>8 ))

else # octet wasn't numeric, return error

return 1

done

return ${RETURNVALUE}

}

The function will return 0 if the IP is valid, and 1 or higher if it encountered an error (you can check with the $? variable directly after calling the function)
Example:

is_valid_ipv4 ${ip}
if [[ $? -gt 0 ]]
then 
	echo "invalid IP"
else
	echo "valid IP"
fi

is_valid_ipv4 ${ip}

if [[ $? -gt 0 ]]

then

echo "invalid IP"

else

echo "valid IP"

How to use Cluster SSH (cssh) and Mosh together

May 7, 2012May 7, 2012Ryanbash, cssh, mosh

A colleague told me about mosh today, just saying “it’s an alternative to SSH that addresses many of the problems” doesn’t do it credit. Go look at the page and have a look, don’t worry it’s a nice page and I’ll still be here waiting when you come back. I do a lot of work on servers across the globe, and believe me, although subtle, the difference between a 10ms connection and a 100+ ms connection is definitely noticeable 😉 That lag all but disappears with mosh. Mosh doesn’t support X11/SSH-Agent forwarding yet, but it’s on their roadmap.

Back to topic, I also use cssh for working on multiple servers simultaneously. I’d recommend making a copy of your .cshrc file (cp -a ~/.csshrc ~/.csshrc_mosh) and working on the copy. Edit the ~/.csshrc_mosh and change the lines ssh= and ssh_args= (diff of my files):

$ diff .csshrc .csshrc_mosh 
27,28c27,28
&lt; ssh=/usr/bin/ssh
&lt; ssh_args= -x -o ConnectTimeout=5
---
&gt; ssh=/usr/bin/mosh
&gt; ssh_args=

$ diff .csshrc .csshrc_mosh

27,28c27,28

< ssh=/usr/bin/ssh

< ssh_args= -x -o ConnectTimeout=5

---

> ssh=/usr/bin/mosh

> ssh_args=

You will have to remove any parameters in ssh_args= and move them to your ~/.ssh/config due to the difference in command line parameters between mosh and ssh. Now all you have to do is call cssh with the new config file, e.g. with an alias to make things easier alias mcssh=”cssh -C ${HOME}/.csshrc_mosh” And your done.

New Virtualbox version, script for easy update of extension pack

March 14, 2012March 14, 2012Ryan2 Commentsbash, extpack, virtualbox

VirtualBox 4.1.10 was released yesterday with a few nice things in the changelog. Updating virtualbox itself is easy, just download the package and update it. Since I seem to stumble over the update of the extension pack every time (on my headless system) I thought I’d write a small script this time so I don’t have to rethink it next time (automatically downloads and installs the current extpack):

#!/bin/bash
URL="$(wget -qO - https://www.virtualbox.org/wiki/Downloads|grep "Oracle VM VirtualBox Extension Pack"|sed "s/.\+\(http:\/\/download.virtualbox.org.\+.vbox-extpack\).\+/\1/")"
File=${URL##*/}

if [[ "${URL}" != "" &amp;&amp; "${File}" != "" ]]
then
  echo "Downloading and installing ${File}" &amp;&amp; \
  wget -qO ${File} ${URL} &amp;&amp; \
  VBoxManage extpack install --replace ${File}
fi

#!/bin/bash

URL="$(wget -qO - https://www.virtualbox.org/wiki/Downloads|grep "Oracle VM VirtualBox Extension Pack"|sed "s/.\+$http:\/\/download.virtualbox.org.\+.vbox-extpack$.\+/\1/")"

File=${URL##*/}

if [[ "${URL}" != "" && "${File}" != "" ]]

then

echo "Downloading and installing ${File}" && \

wget -qO ${File} ${URL} && \

VBoxManage extpack install --replace ${File}

How to add locking to a shell script (the easy way)

February 14, 2012February 18, 2012Ryanbash, flock

I haven’t posted anything with bash here for a while, so today I’ll throw in a little snippet to use flock to make sure a script is only running once. This is very handy in cron jobs that you want to run often, but there shouldn’t be multiple instances of the script running at the same time.
Since it is small and easy I’d recommend adding it to any code you don’t want running multiple times since “that script” you just wrote, that runs 10 minutes now, might turn into a monster in 6 months and run 45 minutes when things change (data grows, more stuff to do). Better safe than sorry.

Basically we rely on flock to do the heavy lifting and we just add some logic around it:

trap cleanup INT TERM EXIT # call cleanup() if script exits

cleanup() {
	  flock -u 200
}

exec 200&gt;/var/lock/my_script.lock

flock -xn 200 || exit 1 # exit if a lock exists

trap cleanup INT TERM EXIT # call cleanup() if script exits

cleanup() {

flock -u 200

}

exec 200>/var/lock/my_script.lock

flock -xn 200 || exit 1 # exit if a lock exists

man flock will show you more details to the parameters used and even some examples. Basically it will use trap to make sure the lock is released if the script ends in any way. 200 is a random file descriptor I chose for this example, it can be anything numeric. flock -xn means it will attempt to acquire an exclusive lock, and if that fails it will exit with an error.

Putting this somewhere at the top of your script will simply end the script if it finds an existing lock. flock has a few other options like -wait or nor using -n that allow you to not exit but wait for the lock to end (with wait a variable amount of seconds). And thus with a bit of creativity enabling you to only lock specific parts of the code (e.g. database calls, file changes, …) and handling failed lock attempts more gracefully than an exit.

How to build an efficient GeoIP SQL table

August 22, 2011April 10, 2012Ryan1 Commentbash, GeoIP, MySQL

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================

set -o nounset                              # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi

if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} &amp;&amp; rm ${tmpfile}
fi

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"

echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip &gt; geoip.sql

echo -e "\ndone generating geoip.sql"

#!/bin/bash

#===============================================================================

# FILE: generate_geoip_sql.sh

# USAGE: ./generate_geoip_sql.sh

# DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com

# USAGE: import sql into a database

# REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE

# EXAMPLE:

# SELECT country_code,country_name

# FROM geo_ip

# WHERE MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))

# SELECT country_code,country_name

# FROM geo_ip

# WHERE INET_ATON('1.2.3.4')

# BETWEEN ip_from AND ip_to;

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

#===============================================================================

set -o nounset # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]

then

echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "

read answer

if [[ "$(echo $answer|tr "N" "n")" != "n" ]]

then

rm ${CSVFile}

if [[ ! -e ${CSVFile} ]]

then

tmpfile=$(mktemp)

wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip

unzip ${tmpfile} && rm ${tmpfile}

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"

mysql -D temp -e "drop table if exists geo_ip"

echo '

CREATE TABLE geo_ip

(

id INT UNSIGNED NOT NULL auto_increment,

ip_poly POLYGON NOT NULL,

ip_from INT UNSIGNED NOT NULL,

ip_to INT UNSIGNED NOT NULL,

country_code CHAR(2) NOT NULL,

country_name CHAR(50) NOT NULL,

PRIMARY KEY (id),

SPATIAL INDEX (ip_poly)

) ENGINE=MyISAM DEFAULT CHARSET=latin1;

LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"

INTO TABLE geo_ip

FIELDS

TERMINATED BY ","

ENCLOSED BY "\""

LINES

TERMINATED BY "\n"

(

@ip_from_string, @ip_to_string,

@ip_from, @ip_to,

@country_code, @country_string

)

SET

id := NULL,

ip_from := @ip_from,

ip_to := @ip_to,

ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(

/* clockwise, 4 points and back to 0 */

POINT(@ip_from, -1), /* 0, top left */

POINT(@ip_to, -1), /* 1, top right */

POINT(@ip_to, 1), /* 2, bottom right */

POINT(@ip_from, 1), /* 3, bottom left */

POINT(@ip_from, -1) /* 0, back to start */

))),

country_code := @country_code,

country_name := @country_string

;

' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip > geoip.sql

echo -e "\ndone generating geoip.sql"