Mqtt on mhemeryck

evok2mqtt ansible

Mon, 09 Aug 2021 00:00:00 +0000

As part of my home automation series, I did write about a small interfacing library that I did write, called evok2mqtt.

The purpose of this service is to translate between two different protocols: websockets, as exposed by evok and mqtt, which is the primary protocol I did use in communicating with home assistant. If you’d like to get more details about the setup itself, check the blog post series first.

The focus of this post is the automated provisioning of a unipi unit using ansible.

Rationale

Earlier, I did discuss all of the software running on each of my unipi units, where each part is responsible for a layer in the overall process:

the linux-based OS: for capturing and sending I/O events
evok: for exposing this I/O information in various formats and protocols (of which I did use websockets)
evok2mqtt: for translating between websockets and MQTT

Arguably, it would be better to have fewer of these layers, but the current situation is one of many different moving parts.

Next to that, I do have multiple of these unipi nodes, each interfacing different types of I/O, yet fundamentally they share a common setup.

Occasionally, there is a need to update or reprovision the unipi nodes, e.g. in the case of regular software or in the unlucky event that and SD-card crashed.

Considering all of these factors, having a way to automatically provision the unipi units in a predefined state is very helpful to have.

Automated provisioning

Automated provisioning refers to tooling that can handle the tedious tasks related to setting up a client or server automatically. Many different kinds of tooling can be used, e.g.:

scripting like bash:
declarative-style recipes like ansible and puppet
immutable builds like packer or docker
infrastructure-as-code like terraform

Scripting is one of the oldest and most widespread approaches to automation. In scripting, you define a list of commands that each run in sequence after each other, generally from the top of the script to the bottom. Scripting is procedural, meaning you define how a specific action is to be carried out, like a recipe. The advantage of scripting is that it is simple and ubiquitous. On the other side of the equation are all issues related to state manipulation. As a script runs through its sequence of steps, it changes the state to end up in a desired target state. In theory, you can write your scripts in such a way that they can adapt for any kind of initial or in-between state. Also, you could attempt to write some parts of your scripts in an idempotent way, meaning that they perform the same action regardless of their input. At larger scale however, scripting quickly becomes unwieldy of this state manipulation it needs to take into account and idempotency is simply not always possible.

Declarative tools take a different approach, not by defining how the provisioning should take place, but rather what the end-result needs to be. Particularly in the case of provisioning, this makes for an interesting programming paradigm since ideally after running the provisioning tool, it will always result in the same end state, regardless of any initial state the node was in.

Immutable build tools also operate in this fashion, the difference being that the provisioning process itself consists of picking a pre-built image (virtual machine image or container image) and deploying that entirely. Doing updates in this fashion just means killing the entire image and replacing it with another one.

Infrastructure as code augments this approach by extending it to any kind of resource. Cloud providers make all of their virtualised infrastructure components available via APIs. Infrastructure as code provides a common code-based framework to declaratively define your desired cloud architecture.

All of these have their place, but for now, I did focus on using ansible as a declarative provisioning tool:

scripting is too much of a pain, considering the state manipulation issues mentioned before
immutable build tools seem really interesting, but might not fit my use case where I would manually need to swap out built images¹
infrastructure as code: this does not apply here, since I want to run the tooling on the unipi unit itself²
I recently did learn about ansible and also a good use case to apply it

Some basic ansible-lingo before we dive into the details:

ansible is the tool you run on your local machine to perform actions on a remote system
an action on a remote machine is a task
a set of tasks can be combined into a role
a set of roles can be further combined into a playbook
… further higher levels of grouping also still exist (collections?), but I didn’t give that much extra thought
ansible-playbook is the tool to orchestrate the execution of a playbook

For my purposes, I did write a playbook.

Initial setup

Ansible has some basic initial conditions for it to be run on the target host:

the target host needs to be reachable via SSH (ideally key-based)
the target host should run some version of python (as ansible itself is based on python)
depending on the permissions you need to execute, the connecting user needs to have root access

Fortunately, unipi does provides such an image that already satisfies these constraints (most modern linux-based OS images would do, actually): the Neuron OpenSource OS.

Practical next steps include:

flash the SD card for the unipi with this image
ensure an internet connection, preferably wired, although a headless raspberry pi wifi install would also work

From here on, you can start running the playbook!

Running `evok2mqtt-ansible`

The playbook can be found on github evok2mqtt-ansible.

From the README file:

The first step is to clone the repository:

git clone https://github.com/mhemeryck/evok2mqtt-ansible.git

Next, we need some extra requirements which can be installed using ansible-galaxy (a tool to pull in other people’s playbooks).

ansible-galaxy install -r requirements.yaml

Adapt the hosts.yaml file for your own purposes.

Run the playbook with

ansible-playbook site.yaml -i hosts.yaml

A closer look

Taking a step back; what does this playbook actually do? Let’s check the folder structure:

.
├── hosts.yaml
├── LICENSE.txt
├── README.md
├── requirements.yaml
├── roles
│   ├── account
│   │   └── tasks
│   │   └── main.yaml
│   ├── evok2mqtt
│   │   ├── handlers
│   │   │   └── main.yaml
│   │   ├── tasks
│   │   │   └── main.yaml
│   │   └── templates
│   │   └── evok2mqtt.service.j2
│   ├── hostname
│   │   ├── tasks
│   │   │   └── main.yaml
│   │   └── vars
│   │   └── hostnames.yaml
│   ├── security
│   │   ├── handlers
│   │   │   └── main.yaml
│   │   └── tasks
│   │   └── main.yaml
│   └── wifi
│   ├── handlers
│   │   └── main.yaml
│   ├── tasks
│   │   └── main.yaml
│   └── templates
│   └── wpa_supplicant.conf.j2
└── site.yaml

The playbook consists of:

a site.yaml file which is the main entry point which invokes the different roles
a hosts.yaml file which lists the groups of target hosts to run the commands against (you would definitely need to adapt this to your needs)
a requirements.yaml file which defines external roles and plugins to be used
5 different folders, each related to a specific role; a set of tasks that are available for execution

Discussing each of the roles themselves might be a bit too much (you can just was well read the source for that), but I want to discuss the main site.yaml file which brings all of the roles together:

---
- hosts: testpi
 vars:
 ansible_python_interpreter: /usr/bin/python3 # Force py3
 ansible_ssh_password: unipi.technology # not really a secret
 ansible_ssh_user: unipi
 mqtt_host: shuttle.lan
 username: mhemeryck
 wifi_ssd: luctor_IoT
 wifi_passphrase: "{{ lookup('community.general.passwordstore', 'home/luctor') }}"
 roles:
 - account
 - hostname
 - wifi
 - evok2mqtt
- hosts: testpi
 vars:
 ansible_python_interpreter: /usr/bin/python3 # Force py3
 username: mhemeryck
 roles:
 - security

The main file defines two sets of roles, referring to the different roles that are made available. The first set is the biggest one, the second one can only run after the first has completed. The reason for this split is that when the initial image is made available, it does include a default unipi user with a fixed password. The first set of roles will run as this user (as this is the only user available at that time). Afterwards, we can run a second security-related role as a newly available hardened user and then remove that unipi user – you obviously can’t remove the unipi user while you are executing tasks with it. You can see that the first set of roles does include the ansible_ssh_user and ansible_ssh_password: these refer to those standard credentials.

In terms of credentials, I did include those ssh user / password as variables in plain text, as this publicly available knowledge. For the wifi_passphrase however, I did use my password manager pass and the related ansible pass plugin!

A quick look at the different roles:

account: this provisions my own user with proper root access and copies in my public SSH keys from github
hostname: this role changes the host name to a sensible one. The mapping for the host name is determined from a variables file³.
wifi: this sets up wpa_supplicant with proper WIFI credentials for my home network.
evok2mqtt: the main job; it will do a full system upgrade, install evok, install evok2mqt, create a system unit file and make sure it runs as a daemon
security: some cleanup tasks, like disabling password-based SSH logins (key-based only) and removing the default unipi user.

After running these tasks, the unipi unit should in theory be in my ideal end state!

Closing thoughts

Given the central place my unipi units play in my home automations setup, having any failures on them is, however rare, not fun. With this in mind and also a general interest in learning infrastructure automation, I did set out to automate those tasks with ansible.

The current downside of this approach is that the full system upgrade will mean that the unipi is occupied for a large portion of time. Consequently, I haven’t yet set out to use it on my live environment. However, any breaking failures on my unipi units could now be quickly mitigated by flashing an SD-card and running this playbook against, which is a huge improvement over the previous situation.

Alternative future solutions could consist of changing the software required to run on the unipi units (would simplify the install process) and / or fully immutable (packer) builds, which completely prepare the image upfront.

in the cloud server world, this is of course an entirely different situation. ↩︎
a platform that could mimic infrastructure-as-code for local development would be really cool, though! ↩︎
I did not include my own file, since this could contain sensitive information … ↩︎

Service

Tue, 03 Aug 2021 00:00:00 +0000

Up to this point, I did already lay out all of the tidbits concerning the electricity, the I/O hardware units, even some custom software to provide an event-based API to address all of the various components in the overall system. The final fundamental piece to bring everything together is what I call the service layer, and specifically home assistant.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Architecture

To recap, have a look at this drawing from the earlier home automation architecture post.

In summary:

a central MQTT broker: this component takes in events on a topic from the various publishers and forwards them to any of the subscribers for that topic
all the unipi units: can take in any MQTT event to handle as commands and pushes out events as state updates.
home assistant: has logic to trigger specific events on other events incoming

Note that for this publish / subscribe system, all of the clients can (and will) function as both publisher and subscriber.

home assistant

I already did mention home assistant, but what is it really?

Home assistant is an open source software platform that was created with the vision of being able to integrate all sorts of IoT solutions.

It is bundled with a huge amount of integrations for various different vendors; check out the home assistant integrations page for an overview. The way the platform is structured is via a number of entities, e.g. a generic switch, light, etc … Each integration can then implement such entities. The home assistant community would typically encourage new contributors to put the interfacing logic (e.g. an API client) in its own, open source library, to be called from the custom integration. The added value of this approach is twofold:

the community rapidly gets more of these open source libraries
the home assistant installation only needs to pull in the source libraries it needs, instead of having of a big blob of code.

The advantage of running your own platform is that you are more in control of your own hardware, less reliant on cloud solutions.

Though not required, I did find it interesting to learn about more about the way home assistant itself is structured, check the home assistant architecture dev docs.

The integration I have been mostly using is home assistant MQTT integration. At this point, this means that all of the I/O that is provided by the unipi units via an MQTT interface can now be readily represented as entities directly in home assistant!

Scenario

Let’s make it more tangible by presenting the scenario from the home automation architecture post again, where a push button triggers a light.

In terms of the home assistant MQTT integration, the notion of command and state topics are quite relevant:

command topic: this is a specific MQTT topic, e.g. /{device}/{io}/set which accepts and incoming command to update the I/O. This scenario is typically coming from home assistant to the IO module.
state topic: this topic is specific for reporting state updates back to any listeners, e.g. /{device}/{io}/state. This is for the opposite scenario where the IO module itself updates its state back again to home assistant.

With this pair of topics for each entity, a typical flow combining command and state topics means:

home assistant requesting a state update on the command topic of the IO module
the IO module updates its value (e.g. toggles a relay for a light)
the IO module sends out an update on the corresponding state topic
home assistant then updates its matching entity according to the state topic

In relation to the flow diagram above:

on the push of the button, an MQTT event is pushed out to the state topic
the MQTT broker pushes the state update to home assistant
home assistant updates the matching home assistant MQTT switch entity
home assistant triggers an automation which connects a state update for the push button entity to a matching light entity
to update the light entity, an event is published on that home assistant MQTT light entity
the MQTT broker pushes the command to the subscribed I/O module
the I/O module takes in the command to update its state and toggles the light accordingly
after that, the I/O module triggers the relay, it pushes back an update on its state topic
the MQTT broker forwards the state update again to home assistant
home assistant sees the state update for that light entity and updates its internal state

Containers and services

Home assistant has multiple ways of installing and running the platform, see the home assistant installation page.

First of all, you should be aware that the overall platform consists of multiple layers and the meaning of each of those has changed over time. I would refer to the following diagram from the home assistant architecture dev docs.

core: at the center, there’s the main home assistant software.
supervisor: this component manages the core software, but it also has the capability to run other services besides home assistant, yet integrating them in a single UI
operating system: the way I understand this, this is a minimal linux-based OS just to be able to run the supervisor for you. The idea is that at this level, you can just download a disk image, flash an SD-card and fire it up in a raspberry pi.

The core itself is actually a containerized version of the main home assistant software, which in turn is a python application. Check out the home assistant core docker container on docker hub. The rationale behind containerization is that you specify an image that contains an isolated environment for your application to run in. Containers are different from virtual machines in that multiple containers would share the same underlying OS kernel, which is better in terms of performance.

Containerization has become dominant in current-day software development and infrastructure management; I use it on a daily basis. A lot can be said on the topic of containerization and IT infrastructure, but I would like to keep this out of the discussion here. If you would be considering gaining more knowledge on this, before anything, understand that containers are there to support a particular model of software development, i.e. service-based development. This typically means stateless, web-based, services.

Apart from the container itself, you typically need something like a supervisor or container orchestrator to run the containers. The orchestrator is responsible for e.g. starting the containers, making sure the correct amount of containers are running at the same time (redundancy, scaling), moving containers between hosts (in a multi-node setup), resource allocation, external networking, …

As mentioned, home assistant has its own orchestrator; the supervisor. An orchestrator I often use for quick-and-dirty local development is docker-compose. The current de-facto industry standard is the google-backed kubernetes, abbreviated to k8s. The lightweight version of this is k3s. This probably not the end of the list, others like hashicorp nomad also exist.

All of these have their own merits and disadvantages:

orchestrator	pros	cons
home assistant supervisor	built into the eco-system of home assistant	container orchestration is hard to do right, why roll your own?
	allows extra functionality in the same UI	to me, this indicates a bad design, where a supervisor is responsible for more than what it’s intended
	supposedly more suited for embedded platforms (raspi 3)	low-memory footprint k8s alternatives exist, like the k3s mentioned before.
docker-compose	simple syntax	only supports a subset of features
	easy to get up an running
k8s	the de-facto standard	standard for web-based services and IT infrastructure management, might be overkill for IoT
	a lot of features, e.g. zero-downtime deployments!	lots of concepts to grasp

Over the course of time, I think I have actually ran home assistant in all of these configurations (even including directly from source). Like everything in IT, there is no real right or wrong solution, it depends on your specific situation. My current approach is based on k3s – but this is better left as the topic of a future post.

Sample setup

All of the theoretical babble about containers and orchestration is just to come to this point where I can actually show something to run. For this sample setup, I will build part of the service layer using only docker-compose and a set of related home assistant configuration files.

The final folder structure will look like this:

├── automations.yaml
├── blueprints
│   ├── automation
│   │   └── homeassistant
│   │   ├── motion_light.yaml
│   │   └── notify_leaving_zone.yaml
│   └── script
│   └── homeassistant
│   └── confirmable_notification.yaml
├── configuration.yaml
├── deps
├── docker-compose.yaml
├── groups.yaml
├── home-assistant.log
├── scenes.yaml
├── scripts.yaml
├── secrets.yaml
├── tts
└── ui-lovelace.yaml

For those interested, have a look at hass.zip for all of the source files.

Most of these files were actually generated automatically when home assistant started the first time. The format for all of the most relevant files is YAML, which stands for “Yet Another Markup Language”. YAML is fairly widespread nowadays as a format for configuration files. To me, one of the most obvious advantages of the format is its conciseness and readability. Indentation matters though, so be sure you use a proper (plain text) editor when editing in this format.

I will go over the most important files in here:

docker-compose.yaml
configuration.yaml
ui-lovelace.yaml

`docker-compose.yaml`

This the main file that will start all of the required containers.

version: "3.8"
services:
 homeassistant:
 image: homeassistant/home-assistant:2021.6.3
 ports:
 - "8123:8123"
 volumes:
 - .:/config
 links:
 - mosquitto
 - db
 mosquitto:
 image: eclipse-mosquitto:1.6.14
 ports:
 - "1883:1883"
 db:
 image: postgres:13.3-alpine
 environment:
 POSTGRES_DB: hass
 POSTGRES_USER: hass
 POSTGRES_PASSWORD: hass
 volumes:
 - hass-db:/var/lib/postgresql/data

volumes:
 hass-db:

This file defines 3 services, homeassistant (the main home assistant service), mosquitto (an MQTT broker) and db (a database – although this isn’t strictly required). Some notes on the keys used in the configuration:

ports: this is a mapping of the ports within the container to the outside world, e.g. "8123:8123" means “maps the port 8123 inside the container to the host on port 812 “. Port 8123 is the port home assistant runs on by default.
volumes: this is mapping of the folder structure inside the container to the outside host. For the homeassistant service, this means to map the configuration folder to the location where docker-compose is executed. This also means it’s easy to put your own configuration inside the container this way. Volumes needn’t be mapped to an explicit host folder though, see the example for the db service where there’s just a named volume called hass-db, managed by docker.
links: links take care of the networking between containers. A link from homeassistant to mosquitto means that the home assistant instance will be able to reach that other service with the name mosquitto. docker-compose takes care of creating the proper docker network for that. It also means that if the homeassistant service is started, the other 2 services are also automatically started.

At this point, you can actually already just fire up home assistant using docker-compose up!

$ docker-compose up
Starting hass_mosquitto_1 ... done
Starting hass_db_1 ... done
Starting hass_homeassistant_1 ... done
Attaching to hass_mosquitto_1, hass_db_1, hass_homeassistant_1
db_1 |
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1 |
db_1 | 2021-06-10 19:19:56.893 UTC [1] LOG: starting PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
db_1 | 2021-06-10 19:19:56.893 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2021-06-10 19:19:56.893 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2021-06-10 19:19:56.902 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2021-06-10 19:19:56.912 UTC [21] LOG: database system was interrupted; last known up at 2021-06-08 20:37:13 UTC
mosquitto_1 | 1623352796: mosquitto version 1.6.14 starting
mosquitto_1 | 1623352796: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto_1 | 1623352796: Opening ipv4 listen socket on port 1883.
mosquitto_1 | 1623352796: Opening ipv6 listen socket on port 1883.
mosquitto_1 | 1623352796: mosquitto version 1.6.14 running
db_1 | 2021-06-10 19:19:57.219 UTC [21] LOG: database system was not properly shut down; automatic recovery in progress
db_1 | 2021-06-10 19:19:57.227 UTC [21] LOG: redo starts at 0/1920BF0
db_1 | 2021-06-10 19:19:57.231 UTC [21] LOG: invalid record length at 0/1931478: wanted 24, got 0
db_1 | 2021-06-10 19:19:57.231 UTC [21] LOG: redo done at 0/1931450
homeassistant_1 | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
db_1 | 2021-06-10 19:19:57.298 UTC [1] LOG: database system is ready to accept connections
homeassistant_1 | [s6-init] ensuring user provided files have correct perms...exited 0.
homeassistant_1 | [fix-attrs.d] applying ownership & permissions fixes...
homeassistant_1 | [fix-attrs.d] done.
homeassistant_1 | [cont-init.d] executing container initialization scripts...
homeassistant_1 | [cont-init.d] udev.sh: executing...
homeassistant_1 | [cont-init.d] udev.sh: exited 0.
homeassistant_1 | [cont-init.d] done.
homeassistant_1 | [services.d] starting services
homeassistant_1 | [services.d] done.

This should actually have created the other configuration files for you.

`configuration.yaml`

The next file to discuss is the main home assistant configuration file.

Here is my final adapted version:

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Text to speech
tts:
 - platform: google_translate

group: !include groups.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

lovelace:
 mode: yaml

recorder:
 db_url: postgresql://hass:hass@db/hass

mqtt:
 broker: mosquitto

# Entities representing push buttons
switch:
 - platform: mqtt
 name: study room
 command_topic: tesla/input/2_01/set
 state_topic: tesla/input/2_01/state

# Entities representing lights
light:
 - platform: mqtt
 name: study room
 command_topic: edison/relay/2_05/set
 state_topic: edison/relay/2_05/state
 optimistic: true # only needed because of demo now

# Automation to link the two together
automation:
 - alias: Toggle study room
 trigger:
 platform: state
 entity_id: switch.study_room
 from: "off"
 to: "on"
 action:
 service: light.toggle
 entity_id: light.study_room

The top part of the file contains mostly boilerplate code that was added by homeassistant itself, e.g. the default_config and tts. The group, script and scene blocks were also added automatically and use the !include keyword, which home assistant would interpret as including those other yaml files within this main configuration file. When you have a large number of entities to add, this would be the typical way to structure them. For the discussion here, I did not do that though.

The recorder part is about the home assistant database integration. This contains the connection URL (including the credentials). If you compare this again with the docker-compose.yaml file, you will see those credentials which were set from the environment key for the db postgres service. In a proper setup, you’d use the secrets system, but as the database integration isn’t that important here, I did not do that.

The mqtt integration is quite relevant here. This should point to your MQTT broker. Checking docker-compose.yaml, this service was called mosquitto and should be reachable with just thus name from inside the docker container network¹.

The following entry is a configuration for a push button using the home assistant MQTT switch integration. It minimally needs the integration platform, the name (which is relevant for later referring to the entity) and a command and state topic, the meaning of which was described earlier.

Next would be the matching light which we would like to have triggered whenever the push button is toggled, based on the home assistant MQTT light integration. It also has the MQTT platform, name (which matches now, but needn’t really), command and state topic. I also configured the light to be in optimistic mode. This means the that the state of the light internally in home assistant is updated regardless of changes on the state topic. I did this because this is a dummy setup and this light is not linked to an actual device that will acknowledge commands with a matching update on the state topic. In my real setup, this is obviously omitted.

At this point, both push button and light are known to home assistant, but they are not linked. The automation takes care of that. The general outline of an home assistant automation is:

| trigger | when the switch state changes |
| condition | and it's nighttime |
| action | then toggle the light |

The condition part is optional for the automation, and for the example given here quite nonsensical indeed. Important here is that we only allow toggles when the push buttons gets toggled from off to on, not the other way around – otherwise if would follow the push buttons state, which means the light would only be on for the duration the button is pressed. The light.toggle is an action that is built-in to home assistant and does just take: if the light was off, it will turn it on, and vice versa.

Check the home assistant configuration for more extensive documentation and further links.

`ui-lovelace.yaml`

This file contains the visual outline for home assistant.

views:
 - title: Home
 icon: mdi:home
 cards:
 - type: entities
 title: Entities
 entities:
 - switch.study_room
 - light.study_room
 - automation.toggle_study_room

This just makes the demo a bit easier to look at.

Demo

Let’s see it in action!

The top left terminal just shows the docker-compose output, containing the aggregated log output of the different services. The right hand part shows the simplified home assistant UI, containing (top to bottom) the switch state, the light state and the automation. The middle left terminal uses the mosquitto_sub tool to read all events from the broker; the left one subscribes to tesla/# meaning any topic below tesla/ (for the push button) and the right for edison/ (the light).

The bottom left terminal is used to simulate the push of a button by sending out events directly to the push button’s state topic, using another tool called mosquitto_pub

The commands themselves are:

mosquitto_pub -h localhost -t tesla/input/2_01/state -m ON
mosquitto_pub -h localhost -t tesla/input/2_01/state -m OFF

Each of the lines represent a push to host “h” localhost, on topic “t” tesla/input/2_01/state with message “m” ON or OFF. By rapidly firing them off shortly after each other, this simulates the push of a button (first on, then off again).

When the push button state updates are pushed:

home assistant updates the internal state of the push button entity
the automation kicks in, and updates the matching light entity is toggled
the light entity sends out the update on the command topic

Apart from controlling the light with the push button, it can also be controlled directly from home assistant; when toggling the light entity directly in home assistant, it just sends out the command on the MQTT topic directly.

Closing thoughts

This concludes my approach to using (abusing?) home assistant and the MQTT integration as the central part of my home automation setup.

I did use MQTT as the standard to centralize all my components on, have a look at the other posts presented in the home automation overview post of how to get there. Examples of other components that use MQTT in the same fashion are window contacts, PIR, smoke and water detectors, motorized blinds, etc.

The advantage of home assistant though is that it does not need to all be the same protocol, so it is quite easy to add other integrations in the mix as well. A notable example of this is an automation where I link one of these push buttons to a Philips Hue light; as far as home assistant is concerned, both of these are light entities.

A disadvantage of using home assistant this way and by extent the overall approach is the centralized nature of it. Home assistant needs to be online all the time for everything to work. Initially, I did run home assistant using docker-compose, which was quite stable. However, doing updates was never fun, since it meant that one container needs to go down before the next comes up, leaving in a short (but stressful) period of downtime. Using k3s as the supervisor certainly fixed that. Given that the majority of my interfacing hardware is actually all on MQTT, I might at some point add another service in the mix that mimics the automation engine as a backup.

A similar argument can be made about the mosquitto broker. Currently, I actually run that as a dedicated service on the host itself. Alternative solutions such as emqx exist though, which would run MQTT in a high-availability mode (multiple, load balanced instances).

Currently though, the system works quite fine and everyone at home has gotten so accustomed to it, they only really notice when not being at home. The backlog of improvements and extra bells and whistles is still quite long, so I’ll hope to write more on this later!

the mosquitto version used in the docker-compose.yaml version was explicitly pinned to a version < 2.x, which did not enforce authentication, hence why the broker configuration just needs a host name to connect to. ↩︎

Software

Tue, 27 Jul 2021 00:00:00 +0000

After connecting everything on the different “physical” layers, it is time to have a look at how to connect all on the software side of things. This post shall describe my current setup, specifically the different software components and the protocols they rely on for the unipi hardware.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Particularly relevant is the hardware post, since I my software builds on the unipi neuron platform.

Software

The following diagram shows the different software layers, going from the kernel which polls the IO-boards to eventually pushing out / pulling in MQTT events.

Kernel: polling SPI

Unipi provides their own neuron open source OS, which is a modified version of raspbian including custom kernel drivers to poll the I/O boards over SPI. The kernel driver makes the data available via the modbus protocol. The modbus protocol is a widely accepted, open industry standard for interfacing with PLCs. I believe you would typically use it in a setup where you have one main controller and a series of follower controllers - which is pretty much the situation you would have using SPI. Note that the unipi units also have an RS-485 connection, commonly used for daisy chaining multiple controllers.

Apart from the modbus interface, unipi also provides a sysfs interface, which essentially maps the I/O states and controls to a number of files in a fixed file system structure. I have used this sysfs interface in another project, unipitt, but since I am currently not using this one, I will not discuss this further.

Evok: modbus to web APIs

Unipi also provides an open source library called evok that will periodically poll the underlying modbus interface and make it available via all sorts of common web API formats and protocols: JSON / REST, JSONRPC, SOAP, … See the evok API docs for an extensive overview of all possible interfaces.

The most interesting interface provided by evok is the evok websockets interface, since this is the only interface that can also trigger actions for particular events. This is a definite requirement here, since you’d need to be able toggle the relays fast in case of an incoming request.

For the receiving of messages (I/O board → outside) you would need to register a number of callback functions to the websocket. The arguments to the callback function then hold information about the specific I/O action that triggered them. For the sending of messages (outside → I/O board), you can just send a structured message directly.

Here’s a python snippet directly taken from the evok websockets interface docs:

import json

import websocket

url = "ws://your.unipi.ip.address/ws"


def on_message(ws, message):
 obj = json.loads(message)
 dev = obj["dev"]
 circuit = obj["circuit"]
 value = obj["value"]
 print(message)


def on_error(ws, error):
 print(error)


def on_close(ws):
 print("Connection closed")


# receiving messages
ws = websocket.WebSocketApp(
 url,
 on_message=on_message,
 on_error=on_error,
 on_close=on_close,
)
ws.run_forever()

# sending messages
ws = websocket.WebSocket()
ws.connect(url)
ws.send('{"cmd":"set","dev":"relay","circuit":"3","value":"1"}')
ws.close()

`evok2mqtt`: websocket to MQTT

The final layer is to translate the websocket “events” from and to MQTT events. This translation is required since this is the standard message-based system interface that was chosen to interface with home assistant. In order to implement this, I did create a small application called evok2mqtt. The application is written in python3 and uses the websockets library for interfacing with the evok websocket and paho-mqtt to do the same thing for MQTT. This application runs alongside evok on the unipi neuron itself. Note that the application could in theory also run elsewhere on the network as the websocket and the MQTT broker can just be addressed over the network. Given that the program itself was actually quite simple and fun to write, I will outline how it works with some example code snippets.

To express actions that need to occur on the receiving of messages, the MQTT library works pretty similar to the websockets library using callback functions, see e.g. this evok2mqtt on_message snippet

def on_message(client, userdata, message):
 """Callback for MQTT events"""
 logger.info(
 f"Incoming MQTT message for topic {message.topic} with payload {message.payload}",
 extra={"kind": LOG_KIND.MQTT},
 )
 match = MQTT_COMMAND_TOPIC_REGEX.match(message.topic)
 if match is None:
 return

 device_name = match.group("device_name")
 dev = match.group("dev")
 circuit = match.group("circuit")
 if device_name != _settings().DEVICE_NAME:
 logger.warning(
 "Handling incoming message for device %s, expected %s",
 device_name,
 _settings().DEVICE_NAME,
 extra={"kind": LOG_KIND.MQTT},
 )

 # Update state topic
 client.publish(
 MQTT_TOPIC_FORMAT.format(
 device_name=device_name,
 dev=dev,
 circuit=circuit,
 hass_action=HASS_ACTION.STATE,
 ),
 message.payload,
 )

 # Send to websocket
 value = 1 if message.payload == _settings().MQTT_PAYLOAD_ON else 0
 logger.info(
 f"Push to output {dev}, {circuit}, {value}", extra={"kind": LOG_KIND.WEBSOCKET}
 )
 asyncio.run(_ws_trigger(dev, circuit, value))

This snippet essentially details all actions that occur for incoming events from the MQTT broker (originating from home assistant):

the message is logged
the message topic is checked to ensure whether we should process this message further¹
the state of the I/O is updated to acknowledge to home assistant that the update went OK
a call is issued to the websocket to do the update on the I/O

For the opposite action, a websocket event that needs to be translated to MQTT, a similar callback function is implemented.

async def _ws_loop():
 """Main loop polling incoming events from websockets"""
 logger.info(
 "Connecting to %s",
 _settings().WEBSOCKET_URI,
 extra={"kind": LOG_KIND.WEBSOCKET},
 )
 async with websockets.connect(_settings().WEBSOCKET_URI) as websocket:
 while True:
 payload = await websocket.recv()
 await _ws_process(payload)

This snippet sets up the websocket loop that will continually check the websocket for newly incoming messages. Note that this uses and async-await on the websocket, a feature provided in python>3.5’s asyncio library. The key thing to understand about asyncio here is that even though this is an infinite loop, during the waiting times for new data to come in from the websocket (or more generally, during the waiting time for any I/O), the thread scheduler can yield control to another thread to execute concurrently.

The function _ws_process that processes the incoming payload is next:

async def _ws_process(payload):
 """Process incoming websocket payload, push to MQTT"""
 obj = json.loads(payload)[0]
 logger.debug(
 "Incoming message for websocket %s", obj, extra={"kind": LOG_KIND.WEBSOCKET}
 )

 topic = MQTT_TOPIC_FORMAT.format(
 device_name=_settings().DEVICE_NAME,
 dev=obj["dev"],
 circuit=obj["circuit"],
 hass_action=HASS_ACTION.STATE,
 )
 payload = (
 _settings().MQTT_PAYLOAD_ON
 if obj["value"] == 1
 else _settings().MQTT_PAYLOAD_OFF
 )
 _mqtt_client().publish(topic, payload=payload)
 logger.info(
 "MQTT publish %s to topic %s",
 payload,
 topic,
 extra={"kind": LOG_KIND.WEBSOCKET},
 )

The operations it does, are quite straightforward:

load the object representation from the incoming payload (as JSON).
construct the MQTT topic based on the object name and circuit.
construct the MQTT payload from the object payload
publish the MQTT payload to the topic

There you have it: translation from-and-to websockets / MQTT. Note that this translation covers both inputs (push buttons) as well as outputs (relays).

Closing thoughts

At this point, with all of these layers of software in between, the unipi neuron unit provides a standard MQTT-based interface to home assistant.

The main downside I’ve felt at this point is that the reaction speed from the evok interface is sometimes a bit too slow to react quickly enough to push button toggles. On rare occasions, this can result in push buttons events being missed – which most of the times can be easily fixed by pushing the buttons a bit longer.

Also, having multiple of these translation layers makes the overall solution a bit more complex and will surely also add to the latency.

I do intend at some point to have another look at this setup. One option is to look at the (open source) implementation of evok and replace the async parts there with the more modern asyncio library. However, if performance is key, I will more likely have a look again at the sysfs interface and my unipitt project, which uses golang under the hood.

Nonetheless, this current setup has been working quite OK for me thus far.

Update from unipi

A while after this post went live, I did get some friendly feedback on my unipi forum post.

Some selected parts of the notes I think I should share²:

evok might get an update, including MQTT support!

The Evok is planned (well, maybe “wished” is a better word:)) to be rewritten to Python 3 and to a multi-threaded application. Along with that, major changes to the API will happen. We will abandon some of the less used APIs (RPC and SOAP probably) and definitely add native MQTT support since it is a driving force in the industry.

Missed state changes should be possible to detect with the digital input counter:

The missed state changes of the DI can be detected by checking the state of the DI counter. The counter is implemented in the HW and can count as fast as 10kHz signals.

The low-level communication I did outline using SPI and modbus is slightly different from what I did present above:

And a bit of explanation of the low level communication:

The HW boards are connected over SPI

The protocol on the SPI level is modified ModbusRTU

The Unipi kernel module (part of the unipi-kernel-modules package) exposes this communication channel as /dev/unipispi device

The Unipi ModbusTCP server (part of the unipi-modbus-tools) exposes this as a ModbusTCP server running on TCP/502

The rest is correct. Evok polls the ModbusTCP server as fast as it can. And creates a “system image” of the HW state.

The IO boards might also get a firmware-level upgrade!

And in the firmware of the IOs we plan to introduce a “latching” state of the DIs. Those will be registers and coils which will hold the leading edge of the DIs until they are read from the software. That will prevent the losing of the edge altogether.

I really appreciate this kind of feedback from the unipi team. It is nice to see how they are continuously improving their product (e.g. the idea of including native MQTT support). More importantly, I think this company really understands well the value of their product as an open platform and is actively working on community-building. Like I did mention in some of my earlier posts in this series, this openness is one of the reasons I am really glad about their products!

this could actually also be solved using configuration on the MQTT client setup, by only registering callbacks to certain topics. ↩︎
mostly literal, some formatting changes mine. ↩︎

Architecture

Tue, 22 Jun 2021 00:00:00 +0000

Prior to actually building anything, it is important to think on a higher level what the various components are and how they will interact with each other, i.e. the architecture. Over the process of doing my own setup, I came across a large number of different possible configurations, each with their own benefits and downsides. For the first part of this post, I will focus on the final layout.

While discussing the architecture, it is important to realize there are a number of different layers of abstraction to reason about these. This will be the topic of the second part of this post.

To make everything more tangible, I did add a third section detailing toggling a light with push button.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts.

Architecture

The following diagram gives an high-level overview of the overall architecture.

Its main characteristics are:

distributed communication:
- communication is (mostly) done via an event-based system, MQTT
centralized wiring:
- all I/O (push buttons, relays for lights, …) is connected to modules which fit in a centralized electric cabinet
- there is a central service with automations connecting the different modules (home assistant)

Let’s elaborate a bit more on the constituent components.

MQTT broker

In the center of the drawing is a mosquitto MQTT broker. MQTT is a publish-subscribe protocol:

there is one central instance, called the broker, which manages different topics
different clients can publish updates, called events, to the topics
in addition, all clients can also subscribe to topics to receive the events

MQTT is a popular choice for IoT since its event-based model fits well for a lot of problems (e.g. a push button toggle). Another advantage is that the each of the different clients only needs to maintain a connection to the broker and the topics they want to be subscribed to. This makes for a very loosely coupled and scalable architecture: it is quite easy to add new components. A disadvantage however is that all connections rely on the availability of the central broker.¹

MQTT clients

The principal other components of the setup are all MQTT clients which both subscribe and publish to specified topics, specifically:

subscribe: they listen to topics which instruct them to do some behavior (“command topics”)
publish: they publish events to topics that indicates their state (“state topics”)

The clients in this setup can provide (a combination of):

sensors: push buttons, motion sensors, smoke detectors, …
actors: lights, covers, sirens, …
logic: send out events on a certain condition, e.g. on another event (an automation)

unipi

For the sensors and actors I did use hardware from unipi. In essence the unipi neuron units are DIN-rail mountable units housing a raspberry pi and a couple of I/O boards depending on the model you have. The role of the unipi units is thus to provide an MQTT event-based interface for all home automation I/O: reading push buttons, toggling lights, controlling shades, reading smoke and movement detectors, etc.

home assistant

Whereas the I/O modules themselves are distributed both physically and logically, I did want to centralize the controlling logic in a dedicated service. Cue in home assistant.

home assistant is an open source software project which aims to integrate all sorts of smart home devices. Given my choice of MQTT as an interfacing protocol throughout my setup, I heavily rely on the home assistant MQTT integration.

From a high level, all sensors and actors which are available as MQTT topics can be represented as entities within the home assistant configuration. The home assistant instance itself connects to the central MQTT broker and listens to those MQTT topics for which it has entities. Automations, which are also part of the configuration, can then link actions from one entity to another. An obvious example is the push of a button that switches a light on. The rationale behind using automations to link inputs with outputs, is that nothing is really hard-wired in this setup!

Note that home assistant supports a wide range of integrations beyond MQTT. The thermostat in the schematic for instance has an integration based on REST / HTTP.

Layers of abstraction

Planning for a DIY wired open home automation setup requires some other specific thinking in terms of the different layers of abstraction. The following diagram focuses on these different layers.

From low to high level:

wiring: the physical wiring which connects to electrical components such as push buttons and light fixtures
hardware: the physical hardware component that will interface with the electrical components (I/O).
software: a client library interfacing with the hardware and which can translate low-level hardware events to protocol events
network: layer which connects the different components together using the protocols
service: the services are those parts which actually talk to each other over the network

Notice that not all components need to have all of the layers. Services like home assistant and the mosquitto broker act completely on the network itself.

Why does this matter? For the majority of commercially available IoT devices, you would get something that takes care of most of the lower levels. Wiring mostly isn’t an issue (it is typically wireless) and the hardware and software are part of the package that you buy. In this wired layout with custom components however, all of these layers need to be addressed.

As an example, have a look at the wiring I did for my cover control:

Most of my wiring was done in a star topology. This means for example that you do need a single cable run for each of the individual (group of) lights you want to control. The upside is that this is very flexible towards the future, since physical rewiring is still possible from the electrical cabinet. The (visible) downside is that the amount of wires piles up rapidly and you will need to think of some kind of system to keep a sane overview of this.

Bus-based systems

Alternatively, the wiring could have been done with a bus-based system like KNX or DALI. The advantage of systems like these is that the wiring would have been a lot easier and they can also provide some of the functionalities on other parts of the stack. For the case of KNX for instance, each of the components (push buttons, light control, …) are provisioned individually, resulting in a fully distributed, more fault-tolerant system. Note that there is a home assistant KNX integration, so even in this case, it could fit the proposed architecture (but obviously less reliant on MQTT).

At this point, I did not choose for a bus-based system though. The first reason was one of cost, notably in the case of KNX. Also, I did want the possibility to easily (re)program my setup using general purpose programming languages, whereas KNX relies on (costly, vendor-restricted) visual programming tools. All of my light sockets are wired with 5-wire cable, so at some point I might have another look at using DALI for controlling my lights.

Toggling a light switch …

To illustrate the theory, let’s just walk through all the steps of toggling a push button:

a push button is pushed down, making contact (off-on) and then released again (on-off)
the unipi digital input unit sees this change and translates this to an MQTT state change event which is pushed to the central mosquitto broker
the home assistant instance has a subscription on the broker for the input button state change and updates its internal entity state
the home assistant instance has an automation configured for the push button entity and triggers the matching light entity to an action
the light entity update pushes out another event to the light action topic to turn on
another unipi unit which is connected to the physical light and has a subscription to the light action topic pulls in the update
the unipi unit toggles a relay, switching on the light
finally, after the light has turned on, a state update is also pushed out from the unipi unit, which updates the internal state on home assistant

Closing thoughts

The focus of this post was to outline the main components and links in my home automation setup, principally based on MQTT and home assistant as the brains behind it all.

The main advantage of this setup is the flexibility if offers since everything is configurable from within home assistant. On the other side of the equation, once home assistant breaks down, the entire system can break down, which poses high reliability constraints on such central components.

I did not yet dive down into the details of the different components i.e. the sensors and actuators, most notably the unipi platform. Also, the different applications themselves, such as push buttons, lighting fixtures, DALI, shades, an alarm system, … all merit their own discussion. Finally, I did also point the different layers of abstraction to consider when laying out a wired home automation system.

This should conclude the “theoretical” part of the home automation series. Check the home automation overview post for the more “practical” upcoming posts.

although even this could be mitigated using a highly available MQTT broker. EMQX seems to be an example of this, but I haven’t tried it myself yet. ↩︎

MQTT over TLS

Mon, 12 Nov 2018 00:00:00 +0000

MQTT is a popular lightweight protocol for use in home automation. If features a central broker where each of the clients can communicate with in a pub / sub fashion.

Since I’ve been toying around with some home automation setups, I was curious how to secure it with TLS.

Mosquitto

The most commonly used broker implementation I have come across is mosquitto and is quite straightforward to setup on most *nix flavors. In my setup, I wanted to run it on Raspberry Pi, which I could leave running at my local home network. Since it runs raspbian, installing it was as simple as:

~ sudo apt-get install mosquitto

TLS setup: keys and certificates

Doing the TLS setup requires a number of key and certificate files. The official mosquitto-tls man page fortunately lists all of them and how to generate them.

Note that for the final setup, not all of these files are required; these are the files we do need:

ca.crt: Certificate Authority certificate: the “central” authority certifying ownership of the public key
server.key: server private key, linked to the certificate.
server.crt: certificate from the server (MQTT broker), signing the server key pair, by the CA.

File layout on the broker

On my setup, most of these folders were already available after install, so I figured to actually use them.

~ tree /etc/mosquitto
/etc/mosquitto
├── ca_certificates
│   ├── ca.crt
│   └── README
├── certs
│   ├── README
│   ├── server.crt
│   └── server.key
├── conf.d
│   ├── README
│   └── tls.conf
└── mosquitto.conf

`mosquitto.conf`

This is pretty the same config that was also provided by the built-in; just to show:

~ cat /etc/mosquitto/mosquitto.conf
# Place your local configuration in /etc/mosquitto/conf.d/
#
# A full description of the configuration file is at
# /usr/share/doc/mosquitto/examples/mosquitto.conf.example
pid_file /var/run/mosquitto.pid
persistence true
persistence_location /var/lib/mosquitto/
log_dest file /var/log/mosquitto/mosquitto.log
# This include assures that our own custom config gets loaded
include_dir /etc/mosquitto/conf.d

`tls.conf`

Actual TLS specific config, pointing to all the files that were added.

~ cat /etc/mosquitto/conf.d/tls.conf
port 8883
cafile /etc/mosquitto/ca_certificates/ca.crt
keyfile /etc/mosquitto/certs/server.key
certfile /etc/mosquitto/certs/server.crt
tls_version tlsv1

Reload the setup

The package installer also conveniently creates the required (systemd) service for us. Simply restart the service now:

sudo systemctl restart mosquitto.service

With the broker running the new config, testing can easily be done with the mosquitto built-in mosquitto_pub / mosquitto_sub commands. Obviously, all clients that want to use this config now need the CA file.

On a client: subscribe:

~ mosquitto_sub --cafile ca.crt -h raspberrypi.lan -t test

Note: the host name should correspond to the host name in the certificate.

Client publish

On a client: publish

~ mosquitto_pub --cafile ca.crt -h raspberrypi.lan -t test -m "hi there"

In the window running the subscribe, the message should now appear.

Here, the clients do not verify themselves though, they just use the certificate to encrypt traffic over TLS.

Mqtt on mhemeryck

evok2mqtt ansible

Rationale

Automated provisioning

Initial setup

Running evok2mqtt-ansible

A closer look

Closing thoughts

Service

Architecture

home assistant

Scenario

Containers and services

Sample setup

docker-compose.yaml

configuration.yaml

ui-lovelace.yaml

Demo

Closing thoughts

Software

Software

Kernel: polling SPI

Evok: modbus to web APIs

evok2mqtt: websocket to MQTT

Closing thoughts

Update from unipi

Architecture

Architecture

MQTT broker

MQTT clients

unipi

home assistant

Layers of abstraction

Bus-based systems

Toggling a light switch …

Closing thoughts

MQTT over TLS

Mosquitto

TLS setup: keys and certificates

File layout on the broker

mosquitto.conf

tls.conf

Reload the setup

Client subscribe

Client publish

Running `evok2mqtt-ansible`

`docker-compose.yaml`

`configuration.yaml`

`ui-lovelace.yaml`

`evok2mqtt`: websocket to MQTT

`mosquitto.conf`

`tls.conf`