Analyze car traffic with Elasticsearch and Kibana

There are currently a lot of construction sites on the my way to work. Therefor i wanted to know when the traffic jams start to occur and whats the best time to get to work.

Google Maps show current traffic on their routing service and “Hey!” there is a API for it: https://developers.google.com/maps/documentation/directions/. You just have to apply for API key there and they give you 2500 queries a day for free.

I started querying the data with curl. Since its a REST API thats really easy:

That will output a lot of JSON data to your screen. And since elasticsearch is schema-less and also uses JSON getting the data into elasticsearch is also really easy. Just pipe the data into another curl command:

That will already works, but to build up time series a timestamp is needed within each document. A small script was needed to reformat the document before pushing it to the elasticsearch index:

Next step was to run this script every 10 minutes in a cronjob:

And to create a simple visualization in kibana based on the routes.legs.duration_in_traffic.value field:

kibana-google-traffic

 

SMS notifications in icinga2 with sms77.de

The sms77.de service could be used to send SMS notifications from icinga2.

First create a service account at sms77.de and install the sms77send command which comes with the SMS::SMS77 perl module:

You may want to send a test message to your mobile:

To be able to call sms77send from icinga2 you need to place the following wrapper script in /etc/icinga2/scripts/sms77-service-notification.sh:

Place the following command definition in /etc/icinga2/conf.d/commands.conf:

Place a template for the notification in /etc/icinga2/conf.d/templates.conf:

And the notification in /etc/icinga2/conf.d/notifications.conf:

To activate SMS notifications for a host add the following lines to your host definition in /etc/icings2/conf.d/hosts.conf:

And your mobile phone number to your contact in /etc/icinga2/conf.d/users.conf (eg. within object User icingaadmin):

Structured JSON Logging with Log4perl

With the help of the Log::Log4perl::Layout::JSON module Log4perl is able to output structured logs.

Example Log4perl Configuration with JSON output to syslog:

Example Code:

Example Output:

The Log4perl MDC feature could be used to attach additional information to all log events produced by your program. For a CGI script it may be usefull to attach the client ip or session information to log events.

Attach additional fields to the MDC:

MDC Example Output: