Markus Benning – with full speed and blue lights

Talk: Using Gitlab CI/Registry for automated Docker builds

On 1. Sept. 2016 i held a talk about using Gitlab with the built in CI and Registry for an automated Docker workflow at the local Docker Bamberg Meetup.

Agenda:

About Gitlab
1. What is Gitlab?
2. Community of Enterprise?
3. Installation options
Setup Gitlab environment
1. Overview
2. Starting it up
3. Setting up the Gitlab server
4. Setting up the Runner
Setup your personal user account
Setup a project
1. Create the project in gitlab
2. Add a build job to the CI
Run your Docker image
Build versioned releases

Find the slides at:

Online at https://markusbenning.de/slides-gitlab-ci-docker-builds/
Project Repository at https://github.com/benningm/slides-gitlab-ci-docker-builds

Analyze car traffic with Elasticsearch and Kibana

There are currently a lot of construction sites on the my way to work. Therefor i wanted to know when the traffic jams start to occur and whats the best time to get to work.

Google Maps show current traffic on their routing service and “Hey!” there is a API for it: https://developers.google.com/maps/documentation/directions/. You just have to apply for API key there and they give you 2500 queries a day for free.

I started querying the data with curl. Since its a REST API thats really easy:

$ curl -X GET 'https://maps.googleapis.com/maps/api/directions/json?key=<your-key>&origin=<home>&destination=<work>&mode=driving&departure_time=now

1	$ curl -X GET 'https://maps.googleapis.com/maps/api/directions/json?key=<your-key>&origin=<home>&destination=<work>&mode=driving&departure_time=now

That will output a lot of JSON data to your screen. And since elasticsearch is schema-less and also uses JSON getting the data into elasticsearch is also really easy. Just pipe the data into another curl command:

<previous-curl> | curl 'http://localhost:9200/traffic/work/' -d '@-'

1	<previous-curl> \| curl 'http://localhost:9200/traffic/work/' -d '@-'

That will already works, but to build up time series a timestamp is needed within each document. A small script was needed to reformat the document before pushing it to the elasticsearch index:

#!/usr/bin/env perl

use strict;
use warnings;

use LWP::UserAgent;
use JSON;
use Time::Piece;
use Search::Elasticsearch;
use URI;

sub usage {
  print "usage: $0 <from> <to>\n"
}

if( scalar(@ARGV) != 2 ) {
  usage;
  exit 1;
}

my $from = $ARGV[0];
my $to = $ARGV[1];

my $url = URI->new('https://maps.googleapis.com/maps/api/directions/json');
$url->query_form(
  key => '<your-key>',
  mode => 'driving',
  departure_time => 'now',
  origin => $from,
  destination => $to,
);

my $ua = LWP::UserAgent->new;

my $response = $ua->get($url);
if ( ! $response->is_success ) {
  die('could not retrieve data: '.$response->status_line);
}
my $data = from_json( $response->content );

my $es = Search::Elasticsearch->new( nodes => [ 'localhost:9200' ] );

$es->index(
  index   => 'traffic',
  type    => 'google_traffic',
  body    => {
    '@timestamp' => Time::Piece->gmtime->datetime,
    from => $from,
    to => $to,
    %$data,
  }
);

#!/usr/bin/env perl

use strict;

use warnings;

use LWP::UserAgent;

use JSON;

use Time::Piece;

use Search::Elasticsearch;

use URI;

sub usage {

print "usage: $0 <from> <to>\n"

}

if( scalar(@ARGV) != 2 ) {

usage;

exit 1;

}

my $from = $ARGV[0];

my $to = $ARGV[1];

my $url = URI->new('https://maps.googleapis.com/maps/api/directions/json');

$url->query_form(

key => '<your-key>',

mode => 'driving',

departure_time => 'now',

origin => $from,

destination => $to,

);

my $ua = LWP::UserAgent->new;

my $response = $ua->get($url);

if ( ! $response->is_success ) {

die('could not retrieve data: '.$response->status_line);

}

my $data = from_json( $response->content );

my $es = Search::Elasticsearch->new( nodes => [ 'localhost:9200' ] );

$es->index(

index => 'traffic',

type => 'google_traffic',

body => {

'@timestamp' => Time::Piece->gmtime->datetime,

from => $from,

to => $to,

%$data,

}

);

Next step was to run this script every 10 minutes in a cronjob:

*/10 * * * * perl <path>/google-traffic '<from>' '<to>'

1	/10 * * * perl <path>/google-traffic '<from>' '<to>'

And to create a simple visualization in kibana based on the routes.legs.duration_in_traffic.value field:

Test-Driven Infrastructure Talk at Docker-Bamberg

On 28th April I held a talk about Test-Driven Infrastructure at the local Docker-Bamberg Meetup.

Agenda:

Concepts
- Puppet
- Test-Driven Infrastructure
- Behavior-Driven Development
Toolchain
- A toolchain for TDI
Hands on
- Build a simple webserver for serving a Docker Bamberg weblog
Goodies
- Continuous Integration, Monitoring, Documentation, Checklists, Change Evidence build in!

Slide are available at:

https://markusbenning.de/slides-docker-bamberg/test-driven-infrastructure.html

mtpolicyd 1.22 has been released

Version 1.22 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

What has changed:

Added support for LDAP
Support for an global LDAP connection has been added to mtpolicyd. The new plugin LdapUserConfig uses this connection to read parameters from a LDAP server.

mtpolicyd 1.21 has been released

Version 1.21 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

What has changed:

New feature vhost_by_policy_context
New option vhost_by_policy_context will if activated tell mtpolicyd to select the VirtualHost based on the policy_context.For example in postfix main.cf use advanced syntax:

check_policy_service { inet:localhost:12345,
    policy_context=reputation }
  ...
  check_policy_service { inet:localhost:12345,
    policy_context=accounting }

check_policy_service { inet:localhost:12345,

policy_context=reputation }

...

check_policy_service { inet:localhost:12345,

policy_context=accounting }

In mtpolicyd.conf:

port="127.0.0.1:12345"# only 1 port
vhost_by_policy_context=1
<VirtualHost 12345>
  name=reputation
  # ...plugins ...
</VirtualHost>
<VirtualHost 12346>
  name=accounting
  # ...plugins ...
</VirtualHost>

port="127.0.0.1:12345"# only 1 port

vhost_by_policy_context=1

name=reputation

# ...plugins ...

</VirtualHost>

name=accounting

# ...plugins ...

</VirtualHost>

The policy_context feature will be available in postfix 3.1 and later.

New plugin SMTPVerify
The SMTPVerify plugin implements address verification at a remote SMTP server with MAIL FROM and RCPT TO commands. It support the following checks:
- check if the remote SMTP server would accept mail for a address.
  Apply actions or scores if a permanent or temporary error is returnedIf the
- remote server support the SIZE extension the SIZE will be passed to the remote SMTP server. This way it could be checked if the message exceeds the message size limit or the quota limit of the recipient.
- Check if the remote SMTP server announces support for STARTTLS
- Check if there is a TLSA record for the remote SMTP server
- Check if there is OPENPGPKEY for the recipient

mtpolicyd 1.20 has been released

Version 1.20 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

What has changed:

fix SQL connection handling after child fork
Closing the connection after child fork did not cause a reconnect on all DBI versions. Instead do a reconnect by overwriting the previous connection.
improve request logging
mtpolicyd now logs the plugin that caused the result.The new log format is:

Perl

<vhost>: instance=<instance>, type=<type>, t=<time>ms, plugin=<plugin>, result=<result>

1

<vhost>: instance=<instance>, type=<type>, t=<time>ms, plugin=<plugin>, result=<result>

mtpolicyd 1.19 has been released

Version 1.19 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

New Features:

escape control characters in logs
The logging method in Mail::MtPolicyd now extends the Net::Server
log method with a mechanism to escape control and special characters.

mtpolicyd 1.16 has been released

Version 1.16 has been released.

Get the sources from: https://mtpolicyd.org/download.html or from CPAN.

New Features:

Improved SPF support
The SPF plugins supports now checks on helo and uses postmaster@<heloname> as sender for null-sender mails as defined in RFC. (thx to Scott Kitterman for pointing this out)
Support for Spamassassins AWL reputation
The SaAwlLookup and SaAwlAction plugins can be used to take actions based on the senders reputation stored in Spamassassins AWL database.

Setup an iptables honeypot with fail2ban

The following example shows how to setup an connection honeypot with the fail2ban daemon. It works by logging connection attempts to unused ports with the iptables LOG target and taking ban actions on the source IPs with fail2ban.

The configuration has been tested on a Debian Wheezy box but should also work for other distributions.

On a Debian box, install the fail2ban daemon with:

apt-get install fail2ban

1	apt-get install fail2ban

Create a action definition in /etc/fail2ban/action.d/iptables-honeypot.local:

[Definition]
actionstart = iptables -A INPUT -p tcp --syn -m multiport -i <honeydev> --dports <honeyports> -j LOG --log-prefix "HONEYPOT CONNECTION: "
actionstop = iptables -D INPUT -p tcp --syn -m multiport -i <honeydev> --dports <honeyports> -j LOG --log-prefix "HONEYPOT CONNECTION: "

actioncheck =
actionban =
actionunban =

[Init]
honeyports = 23,111,137:139,161,162,194,389,445,636,1080,1433,3306,3128
honeydev = eth0

[Definition]

actionstart = iptables -A INPUT -p tcp --syn -m multiport -i <honeydev> --dports <honeyports> -j LOG --log-prefix "HONEYPOT CONNECTION: "

actionstop = iptables -D INPUT -p tcp --syn -m multiport -i <honeydev> --dports <honeyports> -j LOG --log-prefix "HONEYPOT CONNECTION: "

actioncheck =

actionban =

actionunban =

[Init]

honeyports = 23,111,137:139,161,162,194,389,445,636,1080,1433,3306,3128

honeydev = eth0

The start and stop actions will be executed everytime fail2ban starts/stops and will insert the honeypot rules. Adjust the honeyports and honeydev settings for your system. The honeyports line should only list unused ports.

Next create a filter to match the log lines caused by connections to one of the honeyports in /etc/fail2ban/filter.d/iptables-honeypot.local:

[INCLUDES]
before = common.conf

[Definition]
_daemon = kernel
failregex = ^%(__prefix_line)s.*HONEYPOT CONNECTION: .*SRC=<HOST>
ignoreregex =

[INCLUDES]

before = common.conf

[Definition]

_daemon = kernel

failregex = ^%(__prefix_line)s.*HONEYPOT CONNECTION: .*SRC=<HOST>

ignoreregex =

How add a honeypot jail like the following to your jail configuration in /etc/fail2ban/jail.local:

[iptables-honeypot]
enabled  = true
maxretry = 1
banaction = iptables-allports
filter   = iptables-honeypot
logpath  = /var/log/kern.log
port     = all
action   = iptables-honeypot
           %(action_)s

[iptables-honeypot]

enabled = true

maxretry = 1

banaction = iptables-allports

filter = iptables-honeypot

logpath = /var/log/kern.log

port = all

action = iptables-honeypot

%(action_)s

Activate the changes by restarting fail2ban:

/etc/init.d/fail2ban restart

1	/etc/init.d/fail2ban restart

Be careful to not lock out yourself. Make sure to whitelist your own subnets with the ignoreip setting in [DEFAULT]. You may also want to start with a softer ban action than iptables-allports first.

Use posttls-finger to monitor your DANE configuration in icinga2

First you need to install the posttls-finger command. This command is included in postfix versions >=2.11. On Debian you may just rebuild the packages from unstable for your distribution.

Then download the check_posttls_finger script and make it executable:

curl -o /usr/local/bin/check_delv https://raw.githubusercontent.com/benningm/nagios-plugins/master/check_posttls_finger
chmod 755 /usr/local/bin/check_posttls_finger

1 2	curl -o /usr/local/bin/check_delv https://raw.githubusercontent.com/benningm/nagios-plugins/master/check_posttls_finger chmod 755 /usr/local/bin/check_posttls_finger

Add a command definition to icinga2 by creating /etc/icinga2/conf.d/check_posttls_finger.conf with the following content:

object CheckCommand "posttls_finger" {
    import "plugin-check-command"
    command = [ "/usr/local/bin/check_posttls_finger" ]
    arguments = {
        "--domain" = "$dns_lookup$"
        "--wrap-resolvconf" = {
            set_if = "$dns_wrap_resolvconf$"
        }
    }
    vars.dns_wrap_resolvconf = false
}

object CheckCommand "posttls_finger" {

import "plugin-check-command"

command = [ "/usr/local/bin/check_posttls_finger" ]

arguments = {

"--domain" = "$dns_lookup$"

"--wrap-resolvconf" = {

set_if = "$dns_wrap_resolvconf$"

}

vars.dns_wrap_resolvconf = false

}

Also add an service definition to /etc/icinga2/conf.d/services.conf:

apply Service "dane-" for (zone => config in host.vars.zones) {
  import "generic-service"
  check_command = "posttls_finger"
  check_interval = 2h
  vars += config
  assign where host.vars.zones
}

apply Service "dane-" for (zone => config in host.vars.zones) {

import "generic-service"

check_command = "posttls_finger"

check_interval = 2h

vars += config

assign where host.vars.zones

}

Now configure the domains your want to monitor in your host definitions. For example to monitor markusbenning.de:

vars.zones["markusbenning.de"] = {
  dns_lookup = "markusbenning.de"
}

vars.zones["markusbenning.de"] = {

dns_lookup = "markusbenning.de"

}