Talk: Using Gitlab CI/Registry for automated Docker builds

On 1. Sept. 2016 i held a talk about using Gitlab with the built in CI and Registry for an automated Docker workflow at the local Docker Bamberg Meetup.

Agenda:

  1. About Gitlab
    1. What is Gitlab?
    2. Community of Enterprise?
    3. Installation options
  2. Setup Gitlab environment
    1. Overview
    2. Starting it up
    3. Setting up the Gitlab server
    4. Setting up the Runner
  3. Setup your personal user account
  4. Setup a project
    1. Create the project in gitlab
    2. Add a build job to the CI
  5. Run your Docker image
  6. Build versioned releases

Find the slides at:

Analyze car traffic with Elasticsearch and Kibana

There are currently a lot of construction sites on the my way to work. Therefor i wanted to know when the traffic jams start to occur and whats the best time to get to work.

Google Maps show current traffic on their routing service and “Hey!” there is a API for it: https://developers.google.com/maps/documentation/directions/. You just have to apply for API key there and they give you 2500 queries a day for free.

I started querying the data with curl. Since its a REST API thats really easy:

That will output a lot of JSON data to your screen. And since elasticsearch is schema-less and also uses JSON getting the data into elasticsearch is also really easy. Just pipe the data into another curl command:

That will already works, but to build up time series a timestamp is needed within each document. A small script was needed to reformat the document before pushing it to the elasticsearch index:

Next step was to run this script every 10 minutes in a cronjob:

And to create a simple visualization in kibana based on the routes.legs.duration_in_traffic.value field:

kibana-google-traffic

 

Test-Driven Infrastructure Talk at Docker-Bamberg

On 28th April I held a talk about Test-Driven Infrastructure at the local Docker-Bamberg Meetup.

Agenda:

  • Concepts
    • Puppet
    • Test-Driven Infrastructure
    • Behavior-Driven Development
  • Toolchain
    • A toolchain for TDI
  • Hands on
    • Build a simple webserver for serving a Docker Bamberg weblog
  • Goodies
    • Continuous Integration, Monitoring, Documentation, Checklists, Change Evidence build in!

Slide are available at:

https://markusbenning.de/slides-docker-bamberg/test-driven-infrastructure.html

Setup an iptables honeypot with fail2ban

The following example shows how to setup an connection honeypot with the fail2ban daemon. It works by logging connection attempts to unused ports with the iptables LOG target and taking ban actions on the source IPs with fail2ban.

The configuration has been tested on a Debian Wheezy box but should also work for other distributions.

On a Debian box, install the fail2ban daemon with:

Create a action definition in /etc/fail2ban/action.d/iptables-honeypot.local:

The start and stop actions will be executed everytime fail2ban starts/stops and will insert the honeypot rules. Adjust the honeyports and honeydev settings for your system. The honeyports line should only list unused ports.

Next create a filter to match the log lines caused by connections to one of the honeyports in /etc/fail2ban/filter.d/iptables-honeypot.local:

How add a honeypot jail like the following to your jail configuration in /etc/fail2ban/jail.local:

Activate the changes by restarting fail2ban:

Be careful to not lock out yourself. Make sure to whitelist your own subnets with the ignoreip setting in [DEFAULT]. You may also want to start with a softer ban action than iptables-allports first.

Use posttls-finger to monitor your DANE configuration in icinga2

First you need to install the posttls-finger command. This command is included in postfix versions >=2.11. On Debian you may just rebuild the packages from unstable for your distribution.

Then download the check_posttls_finger script and make it executable:

Add a command definition to icinga2 by creating /etc/icinga2/conf.d/check_posttls_finger.conf with the following content:

Also add an service definition to /etc/icinga2/conf.d/services.conf:

Now configure the domains your want to monitor in your host definitions. For example to monitor markusbenning.de:

Use delv to monitor your DNSSEC configuration in icinga2

First you need to install delv. delv is a new diagnostic tool like dig, but with improved DNSSEC support (read more). It comes with bind 9.10 and newer. If you’re already using bind >9.10 then it should be already installed. Otherwise you can grab the latest bind tarball, compile it and use the compiled delv binary:

Then download the check_delv nagios plugin script:

Add a command definition to icinga2 by creating /etc/icinga2/conf.d/check_delv.conf with the following content:

Also add an service definition to /etc/icinga2/conf.d/services.conf:

Now configure the domains your want to monitor in your host definitions. For example to monitor markusbenning.de:

Checking your IP against RBLs in icinga2

To make sure that your IP is listed on any RBL you can implement a daily check in icinga2.

The check can be implement with the check_rbl script:

https://trac.id.ethz.ch/projects/nagios_plugins/wiki/check_rbl

The script has a few perl module dependencies. To install them on a debian system execute:

Then download the script and make it executable:

Also download a copy of the configuration file:

Edit the configuration file and add/remove RBLs as needed. When writting this, the list still included the retired AHBL blacklist. To disable it comment the following line:

Now its time to start a test run:

The script will display all checked RBLs and exit with a Nagios status line:

Now add the command/service definitions to your icinga2 configuration and apply the rbl_address to your hosts definition.

Create /etc/icinga2/conf.d/check_rbl.conf with the following content:

Add the following service description to your /etc/icinga2/conf.d/services.conf:

And a rbl_address variable to all hosts you want to check:

Restart the icinga2 service and see the results in icinga-web.

SMS notifications in icinga2 with sms77.de

The sms77.de service could be used to send SMS notifications from icinga2.

First create a service account at sms77.de and install the sms77send command which comes with the SMS::SMS77 perl module:

You may want to send a test message to your mobile:

To be able to call sms77send from icinga2 you need to place the following wrapper script in /etc/icinga2/scripts/sms77-service-notification.sh:

Place the following command definition in /etc/icinga2/conf.d/commands.conf:

Place a template for the notification in /etc/icinga2/conf.d/templates.conf:

And the notification in /etc/icinga2/conf.d/notifications.conf:

To activate SMS notifications for a host add the following lines to your host definition in /etc/icings2/conf.d/hosts.conf:

And your mobile phone number to your contact in /etc/icinga2/conf.d/users.conf (eg. within object User icingaadmin):

Structured JSON Logging with Log4perl

With the help of the Log::Log4perl::Layout::JSON module Log4perl is able to output structured logs.

Example Log4perl Configuration with JSON output to syslog:

Example Code:

Example Output:

The Log4perl MDC feature could be used to attach additional information to all log events produced by your program. For a CGI script it may be usefull to attach the client ip or session information to log events.

Attach additional fields to the MDC:

MDC Example Output:

Structured JSON Logging with Amavis

Amavis is able to output an structured JSON log format. First make sure you syslog daemon allows log messages longer than 1500 bytes. For example in rsyslogd you can use the following configuration line to raise the maximum message size to 32k:

To activate JSON logging in amavisd you need to adjust the $logline_maxlen setting since amavis will otherwise split long log lines at this threshold.

The log format in amavis is defined by the $log_templ template. For JSON logging you have to use the report_json macro:

For more macros take a look at the amavis documentation:

README.customize.txt