Tech – Page 4 – Ryan Schulze

Bash function for easily watching logs and colorizing the output

October 10, 2017April 1, 2022Ryanbash

Another useful bash function I have on my servers. It’s a wrapper around tail -F and ccze . It will look for a log file (prepends /var/log/ to the patch if it can’t find it), and pipes it into ccze for colorizing the output. Handy if you find yourself watching logs. I mostly use it for dhcp/tftp/mail where I don’t have a huge amount of traffic (i.e. can watch it in real time) and am expecting an event/log entry.

logwatch() {
  local logfile=
  for path in '' /var/log/; do
    if [[ -r "${path}${1}" ]]; then
      tail -n50 -F "${path}${1}" | ccze ${2:+--plugin} ${2} --mode ansi --convert-date
      return
    fi
  done
}

logwatch() {

local logfile=

for path in '' /var/log/; do

if [[ -r "${path}${1}" ]]; then

tail -n50 -F "${path}${1}" | ccze ${2:+--plugin} ${2} --mode ansi --convert-date

return

done

}

Usage:

1 2	logwatch kern.log logwatch /var/log/apache2/access.log

Using regex comparision in bash and BASH_REMATCH

September 8, 2017September 8, 2017Ryanbash

Bash supports regular expressions in comparisons via the =~ operator. But what is rarely used or documented is that you can use the ${BASH_REMATCH[n]} array to access successful matches (back-references to capture groups). So if you use parentheses for grouping () in your regex, you can access the content of that group.

Here is an example where I am parsing date placeholders in a text with an optional offset (e.g. |YYYY.MM.DD|+2 ). Storing the format and offset in separate groups:

while read -r line; do
	while [[ ${line} =~ \|([YMD\\/\ .-]+)\|(\+*[0-9]*) ]]; do
		dateformat=${BASH_REMATCH[1]}
		dateformat=${dateformat/YYYY/%Y}
		dateformat=${dateformat/MMMM/%B}
		dateformat=${dateformat/MM/%m}
		dateformat=${dateformat/DD/%d}
		offset='now'
		[[ ! -z ${BASH_REMATCH[2]} ]] && offset="${BASH_REMATCH[2]} days"
		line=${line/|${BASH_REMATCH[1]}|${BASH_REMATCH[2]}/$(date "+${dateformat}" --date="${offset}")}
	done
	echo "${line}"
done < input

while read -r line; do

while [[ ${line} =~ \|([YMD\\/\ .-]+)\|(\+*[0-9]*) ]]; do

dateformat=${BASH_REMATCH[1]}

dateformat=${dateformat/YYYY/%Y}

dateformat=${dateformat/MMMM/%B}

dateformat=${dateformat/MM/%m}

dateformat=${dateformat/DD/%d}

offset='now'

[[ ! -z ${BASH_REMATCH[2]} ]] && offset="${BASH_REMATCH[2]} days"

line=${line/|${BASH_REMATCH[1]}|${BASH_REMATCH[2]}/$(date "+${dateformat}" --date="${offset}")}

done

echo "${line}"

done < input

|YYYY.MM.DD|

|YYYY.MM.DD|+7

|YYYY-MM-DD|

|YYYY-MM-DD|+14

|MMMM YYYY|

|YYYY/MM|

|MM/YYYY|

This is a sentence containing a timestamp (|YYYY.MM.DD|+7) with an offset.

This is another sentence containing multiple timstamps between |YYYY.MM.DD| and |YYYY.MM.DD|+7.

2017.09.08

2017.09.15

2017-09-08

2017-09-22

September 2017

2017/09

09/2017

This is a sentence containing a timestamp (2017.09.15) with an offset.

This is another sentence containing multiple timstamps between 2017.09.08 and 2017.09.15.

Multiply floats by 10,100, … in bash

August 22, 2017September 8, 2017Ryanbash

A short one today. Bash can only handle integer numbers and not floats, so when someone searches the internet on how to use math on floats in bash the solution they find is usually “use bc” and looks something like this:

$ f=12.3456
$ bc -l <<< "${f} * 10"
123.4560

$ f=12.3456

$ bc -l <<< "${f} * 10"

123.4560

Or if they want the result to be an integer:

$ f=12.3456
$ bc -l <<< "scale=0; ${f} * 10 /1"
123

$ f=12.3456

$ bc -l <<< "scale=0; ${f} * 10 /1"

123

It’s a fine solution, and readable (which can mean a lot for people maintaining scripts). But if all you want to do is multiply by 10,100,1000, … you can achieve this faster with a bit of string manipulation:

$ f=12.3456
$ _sub="${f#*.}"
$ echo "${f%.*}${_sub:0:1}.${_sub:1}"

$ f=12.3456

$ _sub="${f#*.}"

$ echo "${f%.*}${_sub:0:1}.${_sub:1}"

It just splits the number into two strings, and assembles it again with the decimal shifted. Have a look at substring_removal and substring_expansion for more examples on how to modify strings in bash. I’d highly suggest either sticking this in a separate function, or commenting the code since it isn’t necessarily obvious what is going on

Since it is all pure bash and doesn’t need to spawn external commands, it quicker (not that bc is slow, but if you are doing a lot of calculations, it can add up). I know what you are thinking “if your goal is speed, you shouldn’t be using bash”, that doesn’t mean we can’t write efficient code.

API for Troy Hunts passwords list

August 14, 2017August 14, 2017RyanAPI, passwords

TL;DR version: https://github.com/ryanschulze/password-check-api

So NIST updated their recommendations on passwords/authentication a few weeks ago. And while a lot of the reporting was about how password complexity was removed in favor of password length, one point I found intriguing was the suggestion to check if a users password falls into one of these categories:

Passwords obtained from previous breach corpuses.
Dictionary words.
Repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).
Context-specific words, such as the name of the service, the username, and derivatives thereof.

Troy Hunt, the guy behind https://haveibeenpwned.com, deals with a lot of data breaches and made 320 Million passwords from breaches available (at the time of this posting) to help people with checking if passwords that were part of a data breach.

I threw together a small API that can make the data from Troy Hunt easily query-able (or any list of SHA1 hashes for that matter). This can be useful if you have multiple systems that want to query the data, or if you want the data on a separate system.

It’s nothing special, a MySQL backend, a Webserver and an API application using the Slim framework. It’s also stupid fast because there is nothing fancy or special about it. Since it uses a well documented framework it is also easy to change/extend/adjust to your specific requirements.

Bash function to run commands against ansible hosts

July 10, 2017August 14, 2017Ryanbash

I haven’t posted anything ansible related in a while, so here is a nifty little function I regularly use when I want to execute something on all (or a subset) of ansible hosts. It’s just a wrapper around ansible host -m script -a scriptname.sh but adds –tree so that the output is stored and can easily be parsed by jq

ansible_run() {
	local ansible_directory="${HOME}/svn/ansible/"
	local command=${1}
	local target=${2:-all}
	local dir="${HOME}/ansible_output/${RANDOM:0:5}"
	local startdir=
	local script=

	startdir="$(pwd)"
	script="$(mktemp)"
	mkdir -p "${dir}"
	if [[ $? -eq 0 ]] ; then
		{
			echo "#!/usr/bin/env bash"
			echo "${command}"
			echo
		} > "${script}"
		cd "${ansible_directory}" && time ansible "${target}" -m script -a "${script}" --tree "${dir}"
		rm "${script}"
		echo -e "\n${dir}\n"
	fi
	rmdir --ignore-fail-on-non-empty "${dir}"
	cd "${startdir}" || exit
}

ansible_run() {

local ansible_directory="${HOME}/svn/ansible/"

local command=${1}

local target=${2:-all}

local dir="${HOME}/ansible_output/${RANDOM:0:5}"

local startdir=

local script=

startdir="$(pwd)"

script="$(mktemp)"

mkdir -p "${dir}"

if [[ $? -eq 0 ]] ; then

{

echo "#!/usr/bin/env bash"

echo "${command}"

echo

} > "${script}"

cd "${ansible_directory}" && time ansible "${target}" -m script -a "${script}" --tree "${dir}"

rm "${script}"

echo -e "\n${dir}\n"

rmdir --ignore-fail-on-non-empty "${dir}"

cd "${startdir}" || exit

}

Usage example:

$ ansible_run "iptables -S | grep -E '^-P (INPUT|FORWARD)'"
...
/home/user/ansible_output/6599

$ cd ${HOME}/ansible_output/6599
$ jq -r .stdout ./*

$ ansible_run "iptables -S | grep -E '^-P (INPUT|FORWARD)'"

...

/home/user/ansible_output/6599

$ cd ${HOME}/ansible_output/6599

$ jq -r .stdout ./*