Building a esp8266 based weather sensor

In a previous post i wrote about integration of a netatmo weather station into a telegraf/influxdb/grafana stack. The second part of the project was to build a weather sensor for my basement and to integrate it into this stack.

The used esp8266 is a cheap low-power 32bit microcontroller often used for IoT devices. I used a Sparkfun “The Thing” board because it already has a LiPo-charger and a connector for an external antenna on board. But the board has no build in serial/usb connector. You have to connect a USB-RS232 temporary for programming the chip. I used a simple FTDI breakout.

The sensor chip is a DHT22. A digital sensor for temperature and humidity. Since it uses a digital protocol wiring is really easy. Just connect it to 3V3 and GND for power and to a GPIO pin for data.

On the software side i used ESPEasy which is a complete firmware for the esp with everything included. It just must be compiled and falshed. Then everything can be configured by browser. I patched it a little bit to also calculate the value for absolute humidity.

I’m using the build-in “OpenHAB MQTT” connector to upload values to an mosquitto mqtt broker. This format could be read with telegraf with the following configuration:

The finished board.

Prototype on a breadboard.

Calculate absolute humidity from relative humidity and temperature

The short story in ruby is:

And in C:

More details:

Integrate Netatmo Weather station into TIG stack

To integrate my netatmo weather station with custom esp8266 based weather sensors i decided to setup a TIG stack.

The components of the stack are

  • Telegraf – collects data from different sources and pushes them to influxdb
  • Influxdb – a metric database
  • Grafana – a visualization and dashboard application

The netatmo weather stations have no direct interface to capture the collected data, but there is an API server provided by Netatmo to retrieve data from it.

To access the API you must log into the developer website with your netatmo account and register an application:

https://dev.netatmo.com/

After you created an application on their website an client_id and client_secret is assigned to this application. You need this information together with your account credentials to access your stations data within the API.

I wrote an integration script to read the data from the API and output it as JSON suitable for telegrafs exec input:

https://github.com/benningm/docker-telegraf-netatmo

The script could be used directly or as a docker container:

https://hub.docker.com/r/benningm/telegraf-netatmo/

See the github page or docker hub page for usage instruction.

Prevent your credentials from being committed to git

Sometimes you want to add local credentials to a configuration file without adding them to version control or you just want to make sure credentials dont get added to version control at all. Git filters can be used to filter out credentials.

A filter is just a simple pipe command which reads content from STDIN and outputs the filtered content to STDOUT.

An example filter script in ruby:

And the same example in perl:

It will replace most username fields in ini and yaml style files with ‘username’ and passwords with ‘secret’.

The next step is to define the a filter with this script in your global git config:

This will add a section with your filter script to the .gitconfig file:

The next step is to apply this filter in your git projects. This is done with git attributes. To filter yaml config files add the following line to .gitattributes in the root directory of your git repository:

Create the file if it does not yet exist. And commit it to the repository. If the filter is not configured in someone else git client it will be ignored.

Tidy HTML files in Gitlab CI pipeline

The tidy tool could be use to check the syntax of HTML files. To use it in your Gitlab CI pipeline add a tidy.config file to your projects root folder:

And the following job to .gitlab-ci.yml:

Make sure your Pipeline has a test stage and a docker runner is configured with the tag docker-runner.

For available configuration options check the tidy(1) man page.

Git undo cheatsheet

This is a cheat sheet for undoing changes in git.

Local Changes

Undo with:

Staged Changes

Undo with:

Committed Changes

Undo with:

Remote Changes

Undo with:

Yubikey for 2FA with Gitlab

Settings for “Two-Factor Authentication” in Gitlab are located at:

User -> Settings -> Account

Press the green button: “Enable two-factor authentication”

Gitlab will require you to register a 2FA app for fallback first before a U2F device could be registred. I used the FreeOTP App which is available for Android and iOS.

  • Press the QR code button in the app and scan the QR code displayed in the settings dialog. This will add a new item for your gitlab account to the app.
  • Tap on the new item to generate a code and enter this code into the ‘Pin code’ form.
  • The next screen will show recovery keys for the 2FA. Store them in a save place.

Now you will be asked for a one-time PIN everytime you log into gitlab.

The next step is to register the yubikey as a U2F device.

  • Open the two-factor authentication dialog again. (now its called “Manage two-factor authentication”)
  • Press “Setup new U2F device”
  • The browser should ask you tap your U2F device. Do it.
  • Enter a name for the device and press “Register U2F device”

Now you should be able to login to gitlab with your yubikey as second factor.

Using system postfix as mail relay for docker containers

If you configured postfix as a local MTA it will only listen on local network devices. To make it accessible from your docker container add the docker0 interface to inet_interfaces in main.cf:

Since the docker interface is started when the docker service starts you have to make sure docker is started before postfix. With systemd you can do this by creating the file /etc/systemd/system/postfix.service.d/after-docker.conf with the content:

If you want to send outgoing mails with a destination other than the local system you can allow this by adding the docker subnet to mynetworks in main.cf:

Talk: Using Gitlab CI/Registry for automated Docker builds

On 1. Sept. 2016 i held a talk about using Gitlab with the built in CI and Registry for an automated Docker workflow at the local Docker Bamberg Meetup.

Agenda:

  1. About Gitlab
    1. What is Gitlab?
    2. Community of Enterprise?
    3. Installation options
  2. Setup Gitlab environment
    1. Overview
    2. Starting it up
    3. Setting up the Gitlab server
    4. Setting up the Runner
  3. Setup your personal user account
  4. Setup a project
    1. Create the project in gitlab
    2. Add a build job to the CI
  5. Run your Docker image
  6. Build versioned releases

Find the slides at: