Finding changes in an ansible run output

As you can tell I work a bit with ansible. Once you get a few longer playbooks, or have to deploy to hundreds of hosts it can get a bit tedious to search through the output to find “ok, what changed?”

ARA can be a big help in such situations, but good old sed will also do nicely in a pinch:  sed -n '/^--- before/,/^changed/p' ansible_output  for bonus points you can pipe the output to colordiff 

Ansible 2.7 oddities

One of the changes with ansible 2.7 is how you are supposed to pass a list of packages to a package module. instead of passing a list via with_items (which then get’s squashed into one call anyway), you just pass the whole list.
Apt as an example:

ansible <= 2.6

ansible <= 2.7

What the documentation doesn’t touch is how to pass multiple lists.
The error output suggests using ['{{ list1 }}', '{{ list2 }}'], but ansible throws a fit if you do that. Turns out you can also concatenate lists with the + sign. So the solution looks like this:

Selfhosting email, and sending email to Microsoft.

About once a year one of my outgoing email servers will magically pop up on Microsofts blacklist of email servers. The exact status in SNDS is “Blocked due to user complaints or other evidence of spamming“, which is a bit ironic since the weekly volume of emails that go to Microsoft controlled domains seldom is higher than 5 and they are all personal emails.
The error message the MTA recieves is “550 5.7.1 Unfortunately, messages from [xxx.xxx.xxx.xxx] weren’t sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.” in case anyone ends up here via google.

I’m not going to bitch and moan or attempt to guess why the server ends up on the list since it isn’t something I can influence
 (just submit a support request and wait), instead I’ll post some tips and pointers to useful tools you should be using:

SPFSender Policy Framework
Specify which IPs are allowed too send mail for your domains, also determine what should happen with mail that is recieved from unauthorized IPs.

DKIMDomainKeys Identified Mail
Outgoing mail servers for a domain can sign the emails and thereby allow recipients to verify that the email really came from a valid mail server and is not from a spoofed sender.

DMARCDomain-based Message Authentication, Reporting & Conformance
DMARC allows you to specify policies for domains or subdomains in regards to “what should happen if an email fails DKIM or SPF” and where/if to send reports for Emails recieved by other mail servers.

SNDSSmart Network Data Service
Allows you to monitor the IPs of your mailservers as viewed by the Outlook.com system.

JMRPJunk Email Reporting Program
Forwards the full message with headers of any email marked as “junk” or “phishing”  by Outlook.com users

MX Toolbox
MXtoolbox is a great website for testing your mail server settings, they also have a free_monitoring service for one IP,  so you can get alerted if your IP shows up on a Blacklist.

Last but not least I use the following small script to monitor the status of my servers according to Microsoft. I call the script via a daily cronjob, and if a server is blacklisted, it sends an alert. Go to Automated Data Access first to generate an API key.

And if all else fails and your IP does end up blacklisted in SNDS, you can go here to submit a ticket to get back off (although I’d suggest checking your mail server logs and the status of the IP on other RBLs first, just to make sure).

http://go.microsoft.com/fwlink/?LinkID=614866

 

Odroid XU4 case

It’s been quiet here lately. I moved back to Germany and have taken on new responsibilities at work. We are slowly getting settled in, although i still have to get the network at the new apartment up and running the way I want it. I expect to post here more often once I have more time for side projects.

Odroid Xu4

One thing I recently acquired and really like, is a new case for my Odroid XU4.
The small 5v Noctua fan fits very nicely in it and is virtually silent, together with the large heatsink it stays cool enough and seldom has to throttle the CPU.
I prefer the odroid over a raspberry pi for “small server” tasks since it has a Gbit interface (that isn’t shared on the USB bus) and supports eMMC modules (no one enjoys waiting on I/O).

Updating foscam firmware via API

Foscam just released a new round of firmware updates. The suggested process to deploy firmware updates involves using the web interface. I’m hoping newer models are less of a hassle, but older models require the user to install an .exe and use internet explorer 10 or hope that your browser is old enough to allow you to enable NPAPI.

That pretty much rules out Linux, Mac, and up-to-date browsers on Windows.

But fear not, the devices have an API (for some reason they deleted any references/links to it from their forum, but I found a version here).

The API comes with its own share of odd quirks, but to update the firmware without a browser you can use the following CURL command

curl --output - --connect-timeout 15 --max-time 180 -H 'Expect:' --form 'image=@FOSIPC_A_patch_ver_whatever.bin' --request POST 'http://192.168.0.123/cgi-bin/CGIProxy.fcgi?cmd=fwUpgrade&usr=admin&pwd=password' 

Replace the username and password with an account with admin privileges, the IP with the IP of your device, and the .bin file with the latest patch for your device.

Note: If your admin username or password contain any non alphanumeric characters, you need to Urlencode them. There are multiple ways to do this (perl, python, jq), but I’ll assume you have curl installed and can do
echo -n '$uper$secure' | curl -Gso /dev/null -w %{url_effective} --data-urlencode @- "" | cut -c 3-

Privacy Policy

Soon new laws will go into effect in the EU regarding data privacy. I assume anyone reading this has already noticed everyone updating their privacy policies. The German version of the laws exempt private websites, but the exact definition of what “private” means is a bit unclear.

To err on the side of caution we also updated our privacy policy, our impressum, and took the necessary steps behind the scenes to be complaint with the laws. The following links/forms are also in the policy itself, but for easier access:

 

Submit a request to rectify data:

 

Blog cleanup

Today I cleaned the blog up a little. I removed the (hard to read) main menu at the top and moved the relevant topics over to the sidebar into the “More information” box. Makes information easier to find and the top of the page is a bit slimmer.

I also added a new page with an overview of a few of the more interesting public and private (web) services I’m running. This makes it easier for me to update the page when I add or remove a service, and easier for you all to see what I’m running, how to reach it, and what kind of uptime can be expected.

And last but not least I switched out the font to something that is slightly better readable with this layout and color scheme and tweaked the CSS a little bit.

Organizing your Steam library

I’ve had a steam account since 2003, so unsurprisingly I’ve accumulated a bunch of games over time thanks to summer sales, winter sales, humble bundles, …
As a way to give users a means to sort and categorize games, Steam introduced categories a while back. You can add arbitrary categories, and have a game in multiple categories, and categories are synced to your account so they appear on multiple devices. The downside is that you have to set categories for each game manually.

Here is where a nifty tools come in hand: Depressurizer (download)

It provides a user friendly interface for organizing your categories, making it easy to mass categories games. But it’s most useful feature is the ability to auto-categorize games. It can do so via various criteria like genres, Steam flags (e.g. “Single-Player” or “Steam Cloud”), Steam tags, Developer & Publisher info, How Long to Beat times, release year, and/or Steam review user scores.

Once automatically categorized it is easy to find games on your library that are Local co-op, or maybe you are feeling like playing a Dungeon Crawler game today, or just want to browse games you have with an Overwhelmingly Positive rating.

I use a unique prefix for the different auto-categories so it is easy to tell them apart in steam. It also doesn’t mess with existing categories, so it’s easy to manage both categories from depressurizer and your own manual ones.

And this is an example of how the end result looks like
in Steam

 

 

 

 

 

 

 

 

 

A short list of cheap but useful network devices to have handy.

Here is a snort list of useful, off the shelf, network devices that sell for around $50. Not necessarily stuff that you would want use long-term in production, but more for the one-off projects or for troubleshooting. 

First off is a 5-Port Gigabit switch from Netgear (the GS105Ev2).
It is interesting because it supports port mirroring and only costs about $35. That’s a really good price. It obviously isn’t going to replace a real network tap and you will lose traffic if the upstream plus downstream you are copying are together more than 1Gbit of traffic (but that’s a problem you always have when mirroring ports 1:1). It is dirt cheap, fanless and small enough to throw in a bag, can’t go wrong with this one. Need a temporary tap to monitor traffic? Just pop it inline and you are good to go.

 

If you are more of a DIY kind of person, I’d suggest the ESPRESSObin.
It sells for about $50 and is a single board computer with Gigabit network interfaces and a SATA interface. It is based on ARM architecture and runs Linux.
Since it has multiple Gbit interfaces and you can hook a HDD up to it you have a wide array of possibilities you could use it for. A lot of network traffic monitoring situations come to mind, where you would want to keep copies of the traffic in question for later analysis.

 

Another little useful device is the EdgeRouter X from Ubiquiti.
As the name suggests it’s a router and sells for about $55 The nice thing about this device is that it runs Linux (or more specifically a fork of Vyatta), so you can dig into it to get it to do things not available on the GUI.
5 Gbit ports, RIP and OSPF routing protocols, usual SOHO features like Firewall, VPN, QoS. Behaves a lot like a switch with some routing functionality. Unfortunately I couldn’t get port mirroring to work reliably but aside from that it’s still an impressive little router.

 

Strict-Transport-Security

I finally got around to setting the Strict-Transport-Security header on all my HTTPS websites. The current age is set to 1 month and I’ll gradually up that to 6 months if no problems pop up.

In case anyone is curious, I terminate SSL on a haproxy, which then sends requests to various backends (containers on the same physical server). This makes it really easy to deal with Let’s Encrypt (just need one agent/script installed on the haproxy host that deals with all incoming /.well-known/acme-challenge/ requests), and I can keep all HTTPS settings central and don’t have to worry about configuration creep or any settings falling through the cracks for oddball subdomains.

How to use ansible to scan for Spectre/Meltdown vulnerable hosts

First of all head on over to github and download a spectre-meltdown-checker that supports JSON output. Now all we need is a ansible playbook that calls that script:

Important is to adjust the path to spectre-meltdown-checker.sh  in the script:  task (the path is relative to wherever your playboook file is). Adapt to your needs however you want. It is basically just feeding the output of the script into the from_json  filter, storing it in a variable and then iterating over the result via with_items.

Example output:


(vulnerable to  CVE-2017-5715 since Intel retracted their microcode updates and haven’t released new ones yet)

Adding CVE information to Nmap scans

Last week I stumbled across a nice nmap script that adds CVE information from https://vulners.com/ to the results of nmap scans. Since it relies on version information from services it requires you scan the host with -sV

Example:  nmap -sV --script vulners host 

The github repository is https://github.com/vulnersCom/nmap-vulners/ 
You can add the script to your nmap installation with dir=/usr/share/nmap/scripts/ ; curl -ksL https://raw.githubusercontent.com/vulnersCom/nmap-vulners/master/vulners.nse >> ${dir}vulners.nse && chmod 0644 ${dir}vulners.nse 
(or wherever your nmap script directory is, e.g. /usr/local/share/nmap/scripts/ ).

How to colorize manpages

I’m surprised I’ve never posted this here before. Turning manpages from monochrome to color is super easy.

There are a few LESS_TERMCAP_*  environment variables you can adjust. Here is a list of useful ones to change

I prefer to only set them for man, so I put this little function in my ~/.bashrc 

 

Bash function for easily watching logs and colorizing the output

Another useful bash function I have on my servers. It’s a wrapper around tail -F  and ccze . It will look for a log file (prepends /var/log/ to the patch if it can’t find it), and pipes it into ccze for colorizing the output. Handy if you find yourself watching logs. I mostly use it for dhcp/tftp/mail where I don’t have a huge amount of traffic (i.e. can watch it in real time) and am expecting an event/log entry.

Usage:

Using regex comparision in bash and BASH_REMATCH

Bash supports regular expressions in comparisons via the =~ operator.  But what is rarely used or documented is that you can use the ${BASH_REMATCH[n]}  array to access successful matches (back-references to capture groups). So if you use parentheses for grouping ()  in your regex, you can access the content of that group.

Here is an example where I am parsing date placeholders in a text with an optional offset (e.g. |YYYY.MM.DD|+2 ). Storing the format and offset in separate groups:

 

 

 

Multiply floats by 10,100, … in bash

A short one today. Bash can only handle integer numbers and not floats, so when someone searches the internet on how to use math on floats in bash the solution they find is usually “use bc” and looks something like this:

Or if they want the result to be an integer:

It’s a fine solution, and readable (which can mean a lot for people maintaining scripts). But if all you want to do is multiply by 10,100,1000, … you can achieve this faster with a bit of string manipulation:

It just splits the number into two strings, and assembles it again with the decimal shifted. Have a look at substring_removal and substring_expansion for more examples on how to modify strings in bash. I’d highly suggest either sticking this in a separate function, or commenting the code since it isn’t necessarily obvious what is going on

Since it is all pure bash and doesn’t need to spawn external commands, it quicker (not that bc  is slow, but if you are doing a lot of calculations, it can add up). I know what you are thinking “if your goal is speed, you shouldn’t be using bash”, that doesn’t mean we can’t write efficient code.

API for Troy Hunts passwords list

TL;DR version: https://github.com/ryanschulze/password-check-api

So NIST updated their recommendations on passwords/authentication a few weeks ago. And while a lot of the reporting was about how password complexity was removed in favor of password length, one point I found intriguing was the suggestion to check if a users password falls into one of these categories:

  • Passwords obtained from previous breach corpuses.
  • Dictionary words.
  • Repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).
  • Context-specific words, such as the name of the service, the username, and derivatives thereof.

Troy Hunt, the guy behind https://haveibeenpwned.com, deals with a lot of data breaches and made 320 Million passwords from breaches available (at the time of this posting) to help people with checking if passwords that were part of a data breach.

I threw together a small API that can make the data from Troy Hunt easily query-able (or any list of SHA1 hashes for that matter). This can be useful if you have multiple systems that want to query the data, or if you want the data on a separate system.

It’s nothing special, a MySQL backend, a Webserver and an API application using the Slim framework. It’s also stupid fast because there is nothing fancy or special about it. Since it uses a well documented framework it is also easy to change/extend/adjust to your specific requirements.

Running an OSSEC server with lots of agents

The default maximum amount of agents an OSSEC server supports is 256. That’s not very much. In order to increase that number, you need to recompile the source code. Instructions for that are in the official documentation

Depending on the defaults of your Linux Distribution, you may start noticing dropped UDP packets (depending on the amount of agents connecting and how much data they are sending to the server). An easy way to check is with  cat /proc/net/snmp | grep Udp\:|column -t  (check if RcvbufErrors or InErrors are increasing).

The following tweaks to the incoming network buffers resolved the situation for me, and can be a good starting point if you are having similar problems and looking at which settings to tweak. Focus on the rmem  buffers first.

 

Bash function to run commands against ansible hosts

I haven’t posted anything ansible related in a while, so here is a nifty little function I regularly use when I want to execute something on all (or a subset) of ansible hosts. It’s just a wrapper around ansible host -m script -a scriptname.sh  but adds –tree so that the output is stored and can easily be parsed by jq 

Usage example: