Markus Benning – with full speed and blue lights

7. September 20177. September 2017

Building a esp8266 based weather sensor

In a previous post i wrote about integration of a netatmo weather station into a telegraf/influxdb/grafana stack. The second part of the project was to build a weather sensor for my basement and to integrate it into this stack.

The used esp8266 is a cheap low-power 32bit microcontroller often used for IoT devices. I used a Sparkfun “The Thing” board because it already has a LiPo-charger and a connector for an external antenna on board. But the board has no build in serial/usb connector. You have to connect a USB-RS232 temporary for programming the chip. I used a simple FTDI breakout.

The sensor chip is a DHT22. A digital sensor for temperature and humidity. Since it uses a digital protocol wiring is really easy. Just connect it to 3V3 and GND for power and to a GPIO pin for data.

On the software side i used ESPEasy which is a complete firmware for the esp with everything included. It just must be compiled and falshed. Then everything can be configured by browser. I patched it a little bit to also calculate the value for absolute humidity.

I’m using the build-in “OpenHAB MQTT” connector to upload values to an mosquitto mqtt broker. This format could be read with telegraf with the following configuration:

[[inputs.mqtt_consumer]]
 servers = ["mosquitto:1883"]
 qos = 0
 topics = [
 "/<devicename>/#",
 ]
 persistent_session = false
 client_id = ""
 data_format = "value"
 data_type = "float"
 username = "<user>"
 password = "<secret>"

[[inputs.mqtt_consumer]]

servers = ["mosquitto:1883"]

qos = 0

topics = [

"/<devicename>/#",

]

persistent_session = false

client_id = ""

data_format = "value"

data_type = "float"

username = "<user>"

password = "<secret>"

6. September 20177. September 2017

Calculate absolute humidity from relative humidity and temperature

The short story in ruby is:

a = 13.2471*(2.718 ** (17.67*t/(t+243.5)))*r/(273.15+t)

1	a = 13.2471(2.718 * (17.67t/(t+243.5)))r/(273.15+t)

And in C:

float a = (6.112 * pow(2.71828,((17.67 * t)/(t + 243.5))) * r * 2.1674) / (273.15 + t);

1	float a = (6.112 * pow(2.71828,((17.67 * t)/(t + 243.5))) * r * 2.1674) / (273.15 + t);

More details:

Mander P. (2012): How to convert relative humidity to absolute humidity

29. August 201730. August 2017

Integrate Netatmo Weather station into TIG stack

To integrate my netatmo weather station with custom esp8266 based weather sensors i decided to setup a TIG stack.

The components of the stack are

Telegraf – collects data from different sources and pushes them to influxdb
Influxdb – a metric database
Grafana – a visualization and dashboard application

The netatmo weather stations have no direct interface to capture the collected data, but there is an API server provided by Netatmo to retrieve data from it.

To access the API you must log into the developer website with your netatmo account and register an application:

https://dev.netatmo.com/

After you created an application on their website an client_id and client_secret is assigned to this application. You need this information together with your account credentials to access your stations data within the API.

I wrote an integration script to read the data from the API and output it as JSON suitable for telegrafs exec input:

https://github.com/benningm/docker-telegraf-netatmo

The script could be used directly or as a docker container:

https://hub.docker.com/r/benningm/telegraf-netatmo/

See the github page or docker hub page for usage instruction.

21. August 201722. August 2017

Prevent your credentials from being committed to git

Sometimes you want to add local credentials to a configuration file without adding them to version control or you just want to make sure credentials dont get added to version control at all. Git filters can be used to filter out credentials.

A filter is just a simple pipe command which reads content from STDIN and outputs the filtered content to STDOUT.

An example filter script in ruby:

#!/usr/bin/env ruby

ARGF.each do |line|
 line.gsub!(/^(\s*user(?:name)?\s*[:=]\s*)(.*)$/i, '\1username')
 line.gsub!(/^(\s*(?:password|token)\s*[:=]\s*)(.*)$/i, '\1secret')
 puts line
end

#!/usr/bin/env ruby

ARGF.each do |line|

line.gsub!(/^(\s*user(?:name)?\s*[:=]\s*)(.*)$/i, '\1username')

line.gsub!(/^(\s*(?:password|token)\s*[:=]\s*)(.*)$/i, '\1secret')

puts line

end

And the same example in perl:

#!/usr/bin/env perl

use strict;
use warnings;

while (<>) {
 s/^(\s*user(?:name)?\s*[:=]\s*)(.*)$/$1username/i;
 s/^(\s*(?:password|token)\s*[:=]\s*)(.*)$/$1secret/i;

print $_;
}

#!/usr/bin/env perl

use strict;

use warnings;

while (<>) {

s/^(\s*user(?:name)?\s*[:=]\s*)(.*)$/$1username/i;

s/^(\s*(?:password|token)\s*[:=]\s*)(.*)$/$1secret/i;

print $_;

}

It will replace most username fields in ini and yaml style files with ‘username’ and passwords with ‘secret’.

The next step is to define the a filter with this script in your global git config:

git config --global filter.credentials.clean <git-clean-credentials-script>

1	git config --global filter.credentials.clean <git-clean-credentials-script>

This will add a section with your filter script to the .gitconfig file:

[filter "credentials"]
 clean = <git-clean-credentials-script>

1 2	[filter "credentials"] clean = <git-clean-credentials-script>

The next step is to apply this filter in your git projects. This is done with git attributes. To filter yaml config files add the following line to .gitattributes in the root directory of your git repository:

config/*.yml filter=credentials

1	config/*.yml filter=credentials

Create the file if it does not yet exist. And commit it to the repository. If the filter is not configured in someone else git client it will be ignored.

20. August 201720. August 2017

Tidy HTML files in Gitlab CI pipeline

The tidy tool could be use to check the syntax of HTML files. To use it in your Gitlab CI pipeline add a tidy.config file to your projects root folder:

quiet: yes
language: en
doctype: html5
drop-empty-elements: no

quiet: yes

language: en

doctype: html5

drop-empty-elements: no

And the following job to .gitlab-ci.yml:

test-html-tidy:
 stage: test
 tags:
 - docker-runner
 image: debian:stretch
 script:
 - apt-get update && apt-get install -y tidy
 - find . -iregex ".*\.html?$" -print0 | xargs -0 -n1 -t tidy -config tidy.config -utf8 -errors

test-html-tidy:

stage: test

tags:

- docker-runner

image: debian:stretch

script:

- apt-get update && apt-get install -y tidy

- find . -iregex ".*\.html?$" -print0 | xargs -0 -n1 -t tidy -config tidy.config -utf8 -errors

Make sure your Pipeline has a test stage and a docker runner is configured with the tag docker-runner.

For available configuration options check the tidy(1) man page.

19. August 201722. August 2017

Git undo cheatsheet

This is a cheat sheet for undoing changes in git.

Local Changes

$ vim example.txt

1	$ vim example.txt

Undo with:

$ git checkout -- example.txt # fetch file from repository
$ git stash [save <description> # save for later
$ git reset --hard # reset whole repository

$ git checkout -- example.txt # fetch file from repository

$ git stash [save <description> # save for later

$ git reset --hard # reset whole repository

Staged Changes

$ git add example.txt

1	$ git add example.txt

Undo with:

$ git reset HEAD example.txt

1	$ git reset HEAD example.txt

Committed Changes

$ git commit

1	$ git commit

Undo with:

$ git revert HEAD  # with revert commit
$ git commit --amend # redo last commit with next commit
$ git reset HEAD^ --hard # remove last commit

$ git revert HEAD # with revert commit

$ git commit --amend # redo last commit with next commit

$ git reset HEAD^ --hard # remove last commit

Remote Changes

$ git push

1	$ git push

Undo with:

$ git revert HEAD ; git push # revert in new commit
$ git reset HEAD^ --hard && git push origin -f # remove commit and force push

1 2	$ git revert HEAD ; git push # revert in new commit $ git reset HEAD^ --hard && git push origin -f # remove commit and force push

17. August 201718. August 2017

Yubikey for 2FA with Gitlab

Settings for “Two-Factor Authentication” in Gitlab are located at:

User -> Settings -> Account

Press the green button: “Enable two-factor authentication”

Gitlab will require you to register a 2FA app for fallback first before a U2F device could be registred. I used the FreeOTP App which is available for Android and iOS.

Press the QR code button in the app and scan the QR code displayed in the settings dialog. This will add a new item for your gitlab account to the app.
Tap on the new item to generate a code and enter this code into the ‘Pin code’ form.
The next screen will show recovery keys for the 2FA. Store them in a save place.

Now you will be asked for a one-time PIN everytime you log into gitlab.

The next step is to register the yubikey as a U2F device.

Open the two-factor authentication dialog again. (now its called “Manage two-factor authentication”)
Press “Setup new U2F device”
The browser should ask you tap your U2F device. Do it.
Enter a name for the device and press “Register U2F device”

Now you should be able to login to gitlab with your yubikey as second factor.

16. August 201718. August 2017

Using system postfix as mail relay for docker containers

If you configured postfix as a local MTA it will only listen on local network devices. To make it accessible from your docker container add the docker0 interface to inet_interfaces in main.cf:

inet_interfaces = 127.0.0.1, [::1], 172.17.0.1

1	inet_interfaces = 127.0.0.1, [::1], 172.17.0.1

Since the docker interface is started when the docker service starts you have to make sure docker is started before postfix. With systemd you can do this by creating the file /etc/systemd/system/postfix.service.d/after-docker.conf with the content:

[Unit]
After=docker.service

1 2	[Unit] After=docker.service

If you want to send outgoing mails with a destination other than the local system you can allow this by adding the docker subnet to mynetworks in main.cf:

mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 172.17.0.1/16

1	mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 172.17.0.1/16

2. January 2017

mtpolicy package included in Debian/unstable

Thanks to Scott Kittermann mtpolicyd found its way into the official Debian repository:

https://packages.debian.org/sid/mtpolicyd

2. September 201623. August 2017

Talk: Using Gitlab CI/Registry for automated Docker builds

On 1. Sept. 2016 i held a talk about using Gitlab with the built in CI and Registry for an automated Docker workflow at the local Docker Bamberg Meetup.

Agenda:

About Gitlab
1. What is Gitlab?
2. Community of Enterprise?
3. Installation options
Setup Gitlab environment
1. Overview
2. Starting it up
3. Setting up the Gitlab server
4. Setting up the Runner
Setup your personal user account
Setup a project
1. Create the project in gitlab
2. Add a build job to the CI
Run your Docker image
Build versioned releases

Find the slides at:

Online at https://markusbenning.de/slides-gitlab-ci-docker-builds/
Project Repository at https://github.com/benningm/slides-gitlab-ci-docker-builds