Home Automation on mhemeryck

modbusbackup

Tue, 16 Nov 2021 00:00:00 +0000

Another home automation post! Please check out my earlier home automation series first to get an idea what I am talking about.

Recently, I figured to rework some part of it to be able to deal better with certain kind of failures, while at the same time still being able to offer all the automation abilities as before.

Recap

First, let’s recap the architecture that was already in place:

inputs: a unipi unit is used to read the digital inputs (e.g. push buttons); updates are pushed as MQTT events
MQTT broker: a centralized broker takes in the updates and pushes them to all subscribers
home assistant: sees updates on the MQTT inputs and performs specific actions as automations on outputs push out another MQTT events
outputs: a unipi unit takes in MQTT updates and triggers a corresponding relay change (e.g. a light).

For more details, refer to the architecture post.

Although this approach has worked fairly well for me, it inherently holds some potential issues.

Problem: centralized points of failure

Firstly, on the software level, all connections need to run through home assistant and the MQTT broker. An advantage of this is that everything is configurable from a centralized place, in software. However, this also requires a higher level of reliability on those software instances. As an example, initially, doing any updates to this software component was a bit more involved since any downtime here, effectively translated into non-functioning light push buttons. Downtime is fortunately a lot less common since I did start using k3s for orchestrating the deployment. Nevertheless, it is (currently) not possible to run home assistant in an “high availability” (HA) mode. HA mode means that multiple home assistant instances can run at the same time, where one instance sits idle as a fail-over for the main running instance.

In addition, on the network level, the links between the components all depend on the local network. Any downtime at this level of the stack again translates into the non-availability of my light push buttons.

This solution would need to be more robust against issues on both software and network layers. At the same time, I would still like to keep centralized control from home assistant.

Solution: direct link

I did start digging around and got some helpful feedback on the unipi forum from this post on connecting 2 unipi units together.

The first issue of the dependency on the home assistant instance could be mitigated by moving the automation logic out of home assistant. This would mean that some other component could listen to the input events and do the required actions on the outputs. The unipi units could easily run such extra software themselves.

The second issue related to the network layer could be solved by finding a way to connect to the involved unipi units directly together. This post on unipi connections provided some inspiration in that respect: the third option details using the RS-485 bus which is available on most unipi units to connect devices together.

Control from home assistant will still be possible: either the update should come from the toggle of an input button (via the direct connection) or from an MQTT event. There should be no issue in having both options at the same time.¹

The new architecture now changes into the following:

inputs / outputs: the edges of the diagram just again show the inputs and output
unipi units: 1 unipi unit is shown for inputs, 1 for outputs
home assistant: state updates on inputs and outputs still come in over MQTT. State changes on the outputs can still be pushed using MQTT.
MQTT broker: sits between the unipi units and home assistant.

The key difference over the previous architecture is that the link between the input push buttons and the output relays is no longer established using an automation in home assistant. Instead, there is a direct link over RS-485 using the Modbus protocol.

Hardware

The following image (taken from the post on unipi connections) shows a conceptual wiring diagram to connect one or more unipi units together. Essentially, it is a 2-wire connection, daisy-chaining all units.

As the photo shows, the wiring I did use was a simple piece of leftover CAT6 UTP (the grey piece of cable behind the black ethernet connector in the front). In fact, I only did use 2 of the 8 wires.

For my purposes I only did connect 2 units together, but you could in principle extend this over to multiple clients on the same bus. This would practically mean extending the 2-wire connection from the last connection on.

I did not need to introduce any additional resistors nor change any of the DIP switches, this all worked fine out of the box.

As the unipi units readily expose the RS-485 interface, this step turned out to be really simple.

Software

The protocol you would typically run on this wiring setup is Modbus. Within Modbus, there is one server and all the other clients are daisy-chained on the same bus. All clients provide their own set of inputs and outputs (“coils”, “discrete inputs”, “input registers” and “holding registers”) in a predefined range. Read and write control is fully managed by the server. To read specific values, the server will just poll those values from the client. To write updates, the server can similarly just put the values where they need to be on the clients. The speed at which updates from a client are propagated thus also depend on the polling speed the server uses. In case any communications needs to occur between clients, this is only possible from the server. The server would then first read those values from one client (by polling) and later on write them to the other client.

The unipi OS exposes the RS-485 connection via a specific linux device; see the unipi serial port map. For my neuron L403 and L303 I would need to use the /dev/ttyNS0 devices.

Running a Modbus protocol stack on both the client and server devices was however not provided by default by unipi. Here I did write my own small script modbusbackup.py, which can run both in client and server mode.

The idea is to run a Modbus client (server mode) on the device that will be expecting incoming Modbus requests. A Modbus server is ran (client mode) on the device that sends out Modbus messages.

The implementation itself is based on PyModbus, which makes it possible to easily write both client and server code.

For now, I only implemented this connection between my unipi unit that reads my push buttons and the unipi unit that controls relays connected to lights. To illustrate, let’s again go through toggling a light:

push button is pushed
evok detects the push button change and exposes the event through a websocket event (among others).
the modbusbackup.py client (on the inputs unit) sees the websocket event, maps it our own configured Modbus address and sends it out as a Modbus message
the modbusbackup.py server (on the relays unit) sees the incoming Modbus message, and translates it to the correct relay update; the current relay state is first retrieved (evok REST API) and then toggled.
evok handles the relay update
the relay is toggled; the light is turned on

Client code

The key function is ws_process:

async def _ws_process(payload) -> None:
 """Process incoming websocket payload, push to modbus RTU"""
 obj = json.loads(payload)[0]
 # Ignore analog I?O
 if obj["dev"] in ("ai", "ao"):
 return
 logger.info(f"Incoming message for websocket {obj}")
 # Don't send any events related to trailing edges
 if obj["value"] == 0:
 return

 try:
 address = _circuit_map()[obj["circuit"]]
 except KeyError:
 logger.debug(f"Could not find mapping address for {obj['circuit']}")
 return

 # Blocking sync call
 logger.info(
 f"Writing {obj['value']} to address {address} for circuit {obj['circuit']}"
 )
 _modbus_client().write_coil(address, obj["value"], unit=_UNIT)

The function takes a payload from a websocket event, made available through evok. The reason we use a websocket event here is that it is easier to check any updates from the inputs as opposed to polling ourselves. Additionally, evok already exposes the events using websockets. This is part is quite close to the earlier solution presented in the software post.

The circuit_map function is singleton wrapper around a dictionary mapping the unipi digital inputs to Modbus addresses. Similarly, the _modbus_client function is also a singleton wrapper to avoid having to constantly recreate the Modbus connection to the other unit.

See this link for the full client code.

Server code

The key to get this to work for me was having a Modbus instance being able to accept incoming messages and handling those directly, i.e. in an event-based way. After digging through some PyModbus examples, I did find this callback server example. I did end up with the following set of functions, definitions:

class CallbackDataBlock(pymodbus.datastore.ModbusSparseDataBlock):
 """callbacks on operation"""

 def __init__(self) -> None:
 super().__init__({index: 0x0 for index in _relay_map().keys()})

 def setValues(self, address: int, values: typing.List) -> None:
 logger.debug(f"Got {values} for {address}")
 _trigger(address, values[0])
 super().setValues(address, values)


def _run_server(port: str, timeout: float, baudrate: int) -> None:
 block = CallbackDataBlock()
 store = pymodbus.datastore.ModbusSlaveContext(
 di=block,
 co=block,
 hr=block,
 ir=block,
 zero_mode=True,
 )

 context = pymodbus.datastore.ModbusServerContext(slaves=store, single=True)
 pymodbus.server.sync.StartSerialServer(
 context,
 framer=pymodbus.transaction.ModbusRtuFramer,
 port=port,
 timeout=timeout,
 baudrate=baudrate,
 )

The _run_server code wraps starting up a serial, RTU, Modbus server. The port to be provided to the server is simple the Linux device mentioned above, /dev/ttyNS0. As part of a Modbus server configuration, you need to provide the outline of the various Modbus addresses it exposes for the main Modbus instance to poll. In this, case I did pass in a custom block, the CallbackDataBlock.

The CallbackDataBlock constructor sets up all required addresses based on the configuration (through the _relay_map function). The _relay_map function keeps a (singleton) mapping of the relays for the given Modbus address. In order to have callback-like handlers, you can override the setValues method. This method is called for any Modbus message for the address that is part of the function arguments. At this point, I do have the message handled by another _trigger function.

def _trigger(address: int, value: bool, host="http://localhost") -> None:
 """Process incoming event"""
 # Only check for rising edges since we're dealing with lights
 if not value:
 return

 # Convert address to zero-based address
 try:
 relay = _relay_map()[address]
 except KeyError:
 logger.debug(f"Could not find relay for address {address}")
 return

 try:
 response = _session().get(f"{host}/json/relay/{relay}")
 response.raise_for_status()
 except requests.exceptions.HTTPError as error:
 logger.debug(f"Issue with API call: {error}")
 return
 try:
 current = response.json()["data"]["value"]
 except (KeyError, ValueError):
 logger.debug("Error reading current state")
 return

 # Flip the bit from current by XOR
 toggled = current ^ 0x1
 try:
 _session().post(f"{host}/json/relay/{relay}", json={"value": str(toggled)})
 response.raise_for_status()
 except requests.exceptions.HTTPError as error:
 logger.debug(f"Issue with API call: {error}")
 return
 else:
 logger.info(f"Toggled relay {relay} to {toggled}")

The _trigger function simply takes in the Modbus message and maps it again to a relay on the unipi output device. For the purposes of light control, a simple toggle is enough, so the value is just used to see if the incoming event is actually for a rising edge, i.e. a True value. The current value is read first using evok; the new value is the toggled value (by XOR).

Refer to this section for the full server code.

Configuration format

As configuration format, I did use YAML, similar to what I had before on home assistant. The source code repository also holds a sample configuration file – actually, my configuration file. The format looks like this:

- index: 0
 input: "2_03"
 name: kelder inkom inbouw
 output: "2_01"

The index can be an arbitrary number – yet it needs to be unique. The input relates to the digital input on the unipi unit connected to the inputs. Similarly, the output relates to the relay output on the other unipi unit. The name field is just there for documentation purposes.

The automation part previously handled by home assistant now effectively becomes part of this configuration file. The link between input and output is done by the unique index field. The input-side will map its digital input change to the index; this index is used as Modbus address. On the other end, the output-side will map the incoming Modbus address to the its relay it needs to change.

This also means that this configuration file needs to be shared between both the input and output side.

Closing thoughts

While my home automation setup described in the home automation series has been running OK for me for quite some time, I did want to be less dependent on the correct functioning of the home assistant instance and the network.

To get there, I made changes on 2 levels. On the hardware-side, I did make use of the already existing RS-485 connection, removing any network-related issues. On the software-side, I did add a custom script to run on both the input and output side that essentially notifies the output-side of any input-related events. The actual link between the input and output is through a simple configuration file, shared by both sides.

I only did implement this connection for my lighting setup since I did feel this is the most critical part for me to continue to keep on functioning. For my alarm setup and my shades, all input and outputs are actually each on their own, same unipi unit. This means that in order to decouple those from home assistant I could also write some custom code, but they would not need the whole RS-485 connection, nor any Modbus setup.

Note that some push button-to-light connections still go through home assistant: I have e.g. some Philips Hue lights that get triggered via a push button change. For this case, I think it still makes sense to keep this connection in home assistant. However, once I would start to expand my DALI-bus based setup, I’m not yet sure how to tackle this.

The transition between the fully home assistant-based outline and my current “modbusbackup” solution has gone pretty well though, to that extent that it’s not really clear to others in my home what I have actually been working on. The main advantage I now see is being able to more easily make any changes on the home assistant / network side.

the automation that was already in place in home assistant to link input to output would need to be removed though. ↩︎

evok2mqtt ansible

Mon, 09 Aug 2021 00:00:00 +0000

As part of my home automation series, I did write about a small interfacing library that I did write, called evok2mqtt.

The purpose of this service is to translate between two different protocols: websockets, as exposed by evok and mqtt, which is the primary protocol I did use in communicating with home assistant. If you’d like to get more details about the setup itself, check the blog post series first.

The focus of this post is the automated provisioning of a unipi unit using ansible.

Rationale

Earlier, I did discuss all of the software running on each of my unipi units, where each part is responsible for a layer in the overall process:

the linux-based OS: for capturing and sending I/O events
evok: for exposing this I/O information in various formats and protocols (of which I did use websockets)
evok2mqtt: for translating between websockets and MQTT

Arguably, it would be better to have fewer of these layers, but the current situation is one of many different moving parts.

Next to that, I do have multiple of these unipi nodes, each interfacing different types of I/O, yet fundamentally they share a common setup.

Occasionally, there is a need to update or reprovision the unipi nodes, e.g. in the case of regular software or in the unlucky event that and SD-card crashed.

Considering all of these factors, having a way to automatically provision the unipi units in a predefined state is very helpful to have.

Automated provisioning

Automated provisioning refers to tooling that can handle the tedious tasks related to setting up a client or server automatically. Many different kinds of tooling can be used, e.g.:

scripting like bash:
declarative-style recipes like ansible and puppet
immutable builds like packer or docker
infrastructure-as-code like terraform

Scripting is one of the oldest and most widespread approaches to automation. In scripting, you define a list of commands that each run in sequence after each other, generally from the top of the script to the bottom. Scripting is procedural, meaning you define how a specific action is to be carried out, like a recipe. The advantage of scripting is that it is simple and ubiquitous. On the other side of the equation are all issues related to state manipulation. As a script runs through its sequence of steps, it changes the state to end up in a desired target state. In theory, you can write your scripts in such a way that they can adapt for any kind of initial or in-between state. Also, you could attempt to write some parts of your scripts in an idempotent way, meaning that they perform the same action regardless of their input. At larger scale however, scripting quickly becomes unwieldy of this state manipulation it needs to take into account and idempotency is simply not always possible.

Declarative tools take a different approach, not by defining how the provisioning should take place, but rather what the end-result needs to be. Particularly in the case of provisioning, this makes for an interesting programming paradigm since ideally after running the provisioning tool, it will always result in the same end state, regardless of any initial state the node was in.

Immutable build tools also operate in this fashion, the difference being that the provisioning process itself consists of picking a pre-built image (virtual machine image or container image) and deploying that entirely. Doing updates in this fashion just means killing the entire image and replacing it with another one.

Infrastructure as code augments this approach by extending it to any kind of resource. Cloud providers make all of their virtualised infrastructure components available via APIs. Infrastructure as code provides a common code-based framework to declaratively define your desired cloud architecture.

All of these have their place, but for now, I did focus on using ansible as a declarative provisioning tool:

scripting is too much of a pain, considering the state manipulation issues mentioned before
immutable build tools seem really interesting, but might not fit my use case where I would manually need to swap out built images¹
infrastructure as code: this does not apply here, since I want to run the tooling on the unipi unit itself²
I recently did learn about ansible and also a good use case to apply it

Some basic ansible-lingo before we dive into the details:

ansible is the tool you run on your local machine to perform actions on a remote system
an action on a remote machine is a task
a set of tasks can be combined into a role
a set of roles can be further combined into a playbook
… further higher levels of grouping also still exist (collections?), but I didn’t give that much extra thought
ansible-playbook is the tool to orchestrate the execution of a playbook

For my purposes, I did write a playbook.

Initial setup

Ansible has some basic initial conditions for it to be run on the target host:

the target host needs to be reachable via SSH (ideally key-based)
the target host should run some version of python (as ansible itself is based on python)
depending on the permissions you need to execute, the connecting user needs to have root access

Fortunately, unipi does provides such an image that already satisfies these constraints (most modern linux-based OS images would do, actually): the Neuron OpenSource OS.

Practical next steps include:

flash the SD card for the unipi with this image
ensure an internet connection, preferably wired, although a headless raspberry pi wifi install would also work

From here on, you can start running the playbook!

Running `evok2mqtt-ansible`

The playbook can be found on github evok2mqtt-ansible.

From the README file:

The first step is to clone the repository:

git clone https://github.com/mhemeryck/evok2mqtt-ansible.git

Next, we need some extra requirements which can be installed using ansible-galaxy (a tool to pull in other people’s playbooks).

ansible-galaxy install -r requirements.yaml

Adapt the hosts.yaml file for your own purposes.

Run the playbook with

ansible-playbook site.yaml -i hosts.yaml

A closer look

Taking a step back; what does this playbook actually do? Let’s check the folder structure:

.
├── hosts.yaml
├── LICENSE.txt
├── README.md
├── requirements.yaml
├── roles
│   ├── account
│   │   └── tasks
│   │   └── main.yaml
│   ├── evok2mqtt
│   │   ├── handlers
│   │   │   └── main.yaml
│   │   ├── tasks
│   │   │   └── main.yaml
│   │   └── templates
│   │   └── evok2mqtt.service.j2
│   ├── hostname
│   │   ├── tasks
│   │   │   └── main.yaml
│   │   └── vars
│   │   └── hostnames.yaml
│   ├── security
│   │   ├── handlers
│   │   │   └── main.yaml
│   │   └── tasks
│   │   └── main.yaml
│   └── wifi
│   ├── handlers
│   │   └── main.yaml
│   ├── tasks
│   │   └── main.yaml
│   └── templates
│   └── wpa_supplicant.conf.j2
└── site.yaml

The playbook consists of:

a site.yaml file which is the main entry point which invokes the different roles
a hosts.yaml file which lists the groups of target hosts to run the commands against (you would definitely need to adapt this to your needs)
a requirements.yaml file which defines external roles and plugins to be used
5 different folders, each related to a specific role; a set of tasks that are available for execution

Discussing each of the roles themselves might be a bit too much (you can just was well read the source for that), but I want to discuss the main site.yaml file which brings all of the roles together:

---
- hosts: testpi
 vars:
 ansible_python_interpreter: /usr/bin/python3 # Force py3
 ansible_ssh_password: unipi.technology # not really a secret
 ansible_ssh_user: unipi
 mqtt_host: shuttle.lan
 username: mhemeryck
 wifi_ssd: luctor_IoT
 wifi_passphrase: "{{ lookup('community.general.passwordstore', 'home/luctor') }}"
 roles:
 - account
 - hostname
 - wifi
 - evok2mqtt
- hosts: testpi
 vars:
 ansible_python_interpreter: /usr/bin/python3 # Force py3
 username: mhemeryck
 roles:
 - security

The main file defines two sets of roles, referring to the different roles that are made available. The first set is the biggest one, the second one can only run after the first has completed. The reason for this split is that when the initial image is made available, it does include a default unipi user with a fixed password. The first set of roles will run as this user (as this is the only user available at that time). Afterwards, we can run a second security-related role as a newly available hardened user and then remove that unipi user – you obviously can’t remove the unipi user while you are executing tasks with it. You can see that the first set of roles does include the ansible_ssh_user and ansible_ssh_password: these refer to those standard credentials.

In terms of credentials, I did include those ssh user / password as variables in plain text, as this publicly available knowledge. For the wifi_passphrase however, I did use my password manager pass and the related ansible pass plugin!

A quick look at the different roles:

account: this provisions my own user with proper root access and copies in my public SSH keys from github
hostname: this role changes the host name to a sensible one. The mapping for the host name is determined from a variables file³.
wifi: this sets up wpa_supplicant with proper WIFI credentials for my home network.
evok2mqtt: the main job; it will do a full system upgrade, install evok, install evok2mqt, create a system unit file and make sure it runs as a daemon
security: some cleanup tasks, like disabling password-based SSH logins (key-based only) and removing the default unipi user.

After running these tasks, the unipi unit should in theory be in my ideal end state!

Closing thoughts

Given the central place my unipi units play in my home automations setup, having any failures on them is, however rare, not fun. With this in mind and also a general interest in learning infrastructure automation, I did set out to automate those tasks with ansible.

The current downside of this approach is that the full system upgrade will mean that the unipi is occupied for a large portion of time. Consequently, I haven’t yet set out to use it on my live environment. However, any breaking failures on my unipi units could now be quickly mitigated by flashing an SD-card and running this playbook against, which is a huge improvement over the previous situation.

Alternative future solutions could consist of changing the software required to run on the unipi units (would simplify the install process) and / or fully immutable (packer) builds, which completely prepare the image upfront.

in the cloud server world, this is of course an entirely different situation. ↩︎
a platform that could mimic infrastructure-as-code for local development would be really cool, though! ↩︎
I did not include my own file, since this could contain sensitive information … ↩︎

Service

Tue, 03 Aug 2021 00:00:00 +0000

Up to this point, I did already lay out all of the tidbits concerning the electricity, the I/O hardware units, even some custom software to provide an event-based API to address all of the various components in the overall system. The final fundamental piece to bring everything together is what I call the service layer, and specifically home assistant.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Architecture

To recap, have a look at this drawing from the earlier home automation architecture post.

In summary:

a central MQTT broker: this component takes in events on a topic from the various publishers and forwards them to any of the subscribers for that topic
all the unipi units: can take in any MQTT event to handle as commands and pushes out events as state updates.
home assistant: has logic to trigger specific events on other events incoming

Note that for this publish / subscribe system, all of the clients can (and will) function as both publisher and subscriber.

home assistant

I already did mention home assistant, but what is it really?

Home assistant is an open source software platform that was created with the vision of being able to integrate all sorts of IoT solutions.

It is bundled with a huge amount of integrations for various different vendors; check out the home assistant integrations page for an overview. The way the platform is structured is via a number of entities, e.g. a generic switch, light, etc … Each integration can then implement such entities. The home assistant community would typically encourage new contributors to put the interfacing logic (e.g. an API client) in its own, open source library, to be called from the custom integration. The added value of this approach is twofold:

the community rapidly gets more of these open source libraries
the home assistant installation only needs to pull in the source libraries it needs, instead of having of a big blob of code.

The advantage of running your own platform is that you are more in control of your own hardware, less reliant on cloud solutions.

Though not required, I did find it interesting to learn about more about the way home assistant itself is structured, check the home assistant architecture dev docs.

The integration I have been mostly using is home assistant MQTT integration. At this point, this means that all of the I/O that is provided by the unipi units via an MQTT interface can now be readily represented as entities directly in home assistant!

Scenario

Let’s make it more tangible by presenting the scenario from the home automation architecture post again, where a push button triggers a light.

In terms of the home assistant MQTT integration, the notion of command and state topics are quite relevant:

command topic: this is a specific MQTT topic, e.g. /{device}/{io}/set which accepts and incoming command to update the I/O. This scenario is typically coming from home assistant to the IO module.
state topic: this topic is specific for reporting state updates back to any listeners, e.g. /{device}/{io}/state. This is for the opposite scenario where the IO module itself updates its state back again to home assistant.

With this pair of topics for each entity, a typical flow combining command and state topics means:

home assistant requesting a state update on the command topic of the IO module
the IO module updates its value (e.g. toggles a relay for a light)
the IO module sends out an update on the corresponding state topic
home assistant then updates its matching entity according to the state topic

In relation to the flow diagram above:

on the push of the button, an MQTT event is pushed out to the state topic
the MQTT broker pushes the state update to home assistant
home assistant updates the matching home assistant MQTT switch entity
home assistant triggers an automation which connects a state update for the push button entity to a matching light entity
to update the light entity, an event is published on that home assistant MQTT light entity
the MQTT broker pushes the command to the subscribed I/O module
the I/O module takes in the command to update its state and toggles the light accordingly
after that, the I/O module triggers the relay, it pushes back an update on its state topic
the MQTT broker forwards the state update again to home assistant
home assistant sees the state update for that light entity and updates its internal state

Containers and services

Home assistant has multiple ways of installing and running the platform, see the home assistant installation page.

First of all, you should be aware that the overall platform consists of multiple layers and the meaning of each of those has changed over time. I would refer to the following diagram from the home assistant architecture dev docs.

core: at the center, there’s the main home assistant software.
supervisor: this component manages the core software, but it also has the capability to run other services besides home assistant, yet integrating them in a single UI
operating system: the way I understand this, this is a minimal linux-based OS just to be able to run the supervisor for you. The idea is that at this level, you can just download a disk image, flash an SD-card and fire it up in a raspberry pi.

The core itself is actually a containerized version of the main home assistant software, which in turn is a python application. Check out the home assistant core docker container on docker hub. The rationale behind containerization is that you specify an image that contains an isolated environment for your application to run in. Containers are different from virtual machines in that multiple containers would share the same underlying OS kernel, which is better in terms of performance.

Containerization has become dominant in current-day software development and infrastructure management; I use it on a daily basis. A lot can be said on the topic of containerization and IT infrastructure, but I would like to keep this out of the discussion here. If you would be considering gaining more knowledge on this, before anything, understand that containers are there to support a particular model of software development, i.e. service-based development. This typically means stateless, web-based, services.

Apart from the container itself, you typically need something like a supervisor or container orchestrator to run the containers. The orchestrator is responsible for e.g. starting the containers, making sure the correct amount of containers are running at the same time (redundancy, scaling), moving containers between hosts (in a multi-node setup), resource allocation, external networking, …

As mentioned, home assistant has its own orchestrator; the supervisor. An orchestrator I often use for quick-and-dirty local development is docker-compose. The current de-facto industry standard is the google-backed kubernetes, abbreviated to k8s. The lightweight version of this is k3s. This probably not the end of the list, others like hashicorp nomad also exist.

All of these have their own merits and disadvantages:

orchestrator	pros	cons
home assistant supervisor	built into the eco-system of home assistant	container orchestration is hard to do right, why roll your own?
	allows extra functionality in the same UI	to me, this indicates a bad design, where a supervisor is responsible for more than what it’s intended
	supposedly more suited for embedded platforms (raspi 3)	low-memory footprint k8s alternatives exist, like the k3s mentioned before.
docker-compose	simple syntax	only supports a subset of features
	easy to get up an running
k8s	the de-facto standard	standard for web-based services and IT infrastructure management, might be overkill for IoT
	a lot of features, e.g. zero-downtime deployments!	lots of concepts to grasp

Over the course of time, I think I have actually ran home assistant in all of these configurations (even including directly from source). Like everything in IT, there is no real right or wrong solution, it depends on your specific situation. My current approach is based on k3s – but this is better left as the topic of a future post.

Sample setup

All of the theoretical babble about containers and orchestration is just to come to this point where I can actually show something to run. For this sample setup, I will build part of the service layer using only docker-compose and a set of related home assistant configuration files.

The final folder structure will look like this:

├── automations.yaml
├── blueprints
│   ├── automation
│   │   └── homeassistant
│   │   ├── motion_light.yaml
│   │   └── notify_leaving_zone.yaml
│   └── script
│   └── homeassistant
│   └── confirmable_notification.yaml
├── configuration.yaml
├── deps
├── docker-compose.yaml
├── groups.yaml
├── home-assistant.log
├── scenes.yaml
├── scripts.yaml
├── secrets.yaml
├── tts
└── ui-lovelace.yaml

For those interested, have a look at hass.zip for all of the source files.

Most of these files were actually generated automatically when home assistant started the first time. The format for all of the most relevant files is YAML, which stands for “Yet Another Markup Language”. YAML is fairly widespread nowadays as a format for configuration files. To me, one of the most obvious advantages of the format is its conciseness and readability. Indentation matters though, so be sure you use a proper (plain text) editor when editing in this format.

I will go over the most important files in here:

docker-compose.yaml
configuration.yaml
ui-lovelace.yaml

`docker-compose.yaml`

This the main file that will start all of the required containers.

version: "3.8"
services:
 homeassistant:
 image: homeassistant/home-assistant:2021.6.3
 ports:
 - "8123:8123"
 volumes:
 - .:/config
 links:
 - mosquitto
 - db
 mosquitto:
 image: eclipse-mosquitto:1.6.14
 ports:
 - "1883:1883"
 db:
 image: postgres:13.3-alpine
 environment:
 POSTGRES_DB: hass
 POSTGRES_USER: hass
 POSTGRES_PASSWORD: hass
 volumes:
 - hass-db:/var/lib/postgresql/data

volumes:
 hass-db:

This file defines 3 services, homeassistant (the main home assistant service), mosquitto (an MQTT broker) and db (a database – although this isn’t strictly required). Some notes on the keys used in the configuration:

ports: this is a mapping of the ports within the container to the outside world, e.g. "8123:8123" means “maps the port 8123 inside the container to the host on port 812 “. Port 8123 is the port home assistant runs on by default.
volumes: this is mapping of the folder structure inside the container to the outside host. For the homeassistant service, this means to map the configuration folder to the location where docker-compose is executed. This also means it’s easy to put your own configuration inside the container this way. Volumes needn’t be mapped to an explicit host folder though, see the example for the db service where there’s just a named volume called hass-db, managed by docker.
links: links take care of the networking between containers. A link from homeassistant to mosquitto means that the home assistant instance will be able to reach that other service with the name mosquitto. docker-compose takes care of creating the proper docker network for that. It also means that if the homeassistant service is started, the other 2 services are also automatically started.

At this point, you can actually already just fire up home assistant using docker-compose up!

$ docker-compose up
Starting hass_mosquitto_1 ... done
Starting hass_db_1 ... done
Starting hass_homeassistant_1 ... done
Attaching to hass_mosquitto_1, hass_db_1, hass_homeassistant_1
db_1 |
db_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1 |
db_1 | 2021-06-10 19:19:56.893 UTC [1] LOG: starting PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
db_1 | 2021-06-10 19:19:56.893 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
db_1 | 2021-06-10 19:19:56.893 UTC [1] LOG: listening on IPv6 address "::", port 5432
db_1 | 2021-06-10 19:19:56.902 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1 | 2021-06-10 19:19:56.912 UTC [21] LOG: database system was interrupted; last known up at 2021-06-08 20:37:13 UTC
mosquitto_1 | 1623352796: mosquitto version 1.6.14 starting
mosquitto_1 | 1623352796: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto_1 | 1623352796: Opening ipv4 listen socket on port 1883.
mosquitto_1 | 1623352796: Opening ipv6 listen socket on port 1883.
mosquitto_1 | 1623352796: mosquitto version 1.6.14 running
db_1 | 2021-06-10 19:19:57.219 UTC [21] LOG: database system was not properly shut down; automatic recovery in progress
db_1 | 2021-06-10 19:19:57.227 UTC [21] LOG: redo starts at 0/1920BF0
db_1 | 2021-06-10 19:19:57.231 UTC [21] LOG: invalid record length at 0/1931478: wanted 24, got 0
db_1 | 2021-06-10 19:19:57.231 UTC [21] LOG: redo done at 0/1931450
homeassistant_1 | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
db_1 | 2021-06-10 19:19:57.298 UTC [1] LOG: database system is ready to accept connections
homeassistant_1 | [s6-init] ensuring user provided files have correct perms...exited 0.
homeassistant_1 | [fix-attrs.d] applying ownership & permissions fixes...
homeassistant_1 | [fix-attrs.d] done.
homeassistant_1 | [cont-init.d] executing container initialization scripts...
homeassistant_1 | [cont-init.d] udev.sh: executing...
homeassistant_1 | [cont-init.d] udev.sh: exited 0.
homeassistant_1 | [cont-init.d] done.
homeassistant_1 | [services.d] starting services
homeassistant_1 | [services.d] done.

This should actually have created the other configuration files for you.

`configuration.yaml`

The next file to discuss is the main home assistant configuration file.

Here is my final adapted version:

# Configure a default setup of Home Assistant (frontend, api, etc)
default_config:

# Text to speech
tts:
 - platform: google_translate

group: !include groups.yaml
script: !include scripts.yaml
scene: !include scenes.yaml

lovelace:
 mode: yaml

recorder:
 db_url: postgresql://hass:hass@db/hass

mqtt:
 broker: mosquitto

# Entities representing push buttons
switch:
 - platform: mqtt
 name: study room
 command_topic: tesla/input/2_01/set
 state_topic: tesla/input/2_01/state

# Entities representing lights
light:
 - platform: mqtt
 name: study room
 command_topic: edison/relay/2_05/set
 state_topic: edison/relay/2_05/state
 optimistic: true # only needed because of demo now

# Automation to link the two together
automation:
 - alias: Toggle study room
 trigger:
 platform: state
 entity_id: switch.study_room
 from: "off"
 to: "on"
 action:
 service: light.toggle
 entity_id: light.study_room

The top part of the file contains mostly boilerplate code that was added by homeassistant itself, e.g. the default_config and tts. The group, script and scene blocks were also added automatically and use the !include keyword, which home assistant would interpret as including those other yaml files within this main configuration file. When you have a large number of entities to add, this would be the typical way to structure them. For the discussion here, I did not do that though.

The recorder part is about the home assistant database integration. This contains the connection URL (including the credentials). If you compare this again with the docker-compose.yaml file, you will see those credentials which were set from the environment key for the db postgres service. In a proper setup, you’d use the secrets system, but as the database integration isn’t that important here, I did not do that.

The mqtt integration is quite relevant here. This should point to your MQTT broker. Checking docker-compose.yaml, this service was called mosquitto and should be reachable with just thus name from inside the docker container network¹.

The following entry is a configuration for a push button using the home assistant MQTT switch integration. It minimally needs the integration platform, the name (which is relevant for later referring to the entity) and a command and state topic, the meaning of which was described earlier.

Next would be the matching light which we would like to have triggered whenever the push button is toggled, based on the home assistant MQTT light integration. It also has the MQTT platform, name (which matches now, but needn’t really), command and state topic. I also configured the light to be in optimistic mode. This means the that the state of the light internally in home assistant is updated regardless of changes on the state topic. I did this because this is a dummy setup and this light is not linked to an actual device that will acknowledge commands with a matching update on the state topic. In my real setup, this is obviously omitted.

At this point, both push button and light are known to home assistant, but they are not linked. The automation takes care of that. The general outline of an home assistant automation is:

| trigger | when the switch state changes |
| condition | and it's nighttime |
| action | then toggle the light |

The condition part is optional for the automation, and for the example given here quite nonsensical indeed. Important here is that we only allow toggles when the push buttons gets toggled from off to on, not the other way around – otherwise if would follow the push buttons state, which means the light would only be on for the duration the button is pressed. The light.toggle is an action that is built-in to home assistant and does just take: if the light was off, it will turn it on, and vice versa.

Check the home assistant configuration for more extensive documentation and further links.

`ui-lovelace.yaml`

This file contains the visual outline for home assistant.

views:
 - title: Home
 icon: mdi:home
 cards:
 - type: entities
 title: Entities
 entities:
 - switch.study_room
 - light.study_room
 - automation.toggle_study_room

This just makes the demo a bit easier to look at.

Demo

Let’s see it in action!

The top left terminal just shows the docker-compose output, containing the aggregated log output of the different services. The right hand part shows the simplified home assistant UI, containing (top to bottom) the switch state, the light state and the automation. The middle left terminal uses the mosquitto_sub tool to read all events from the broker; the left one subscribes to tesla/# meaning any topic below tesla/ (for the push button) and the right for edison/ (the light).

The bottom left terminal is used to simulate the push of a button by sending out events directly to the push button’s state topic, using another tool called mosquitto_pub

The commands themselves are:

mosquitto_pub -h localhost -t tesla/input/2_01/state -m ON
mosquitto_pub -h localhost -t tesla/input/2_01/state -m OFF

Each of the lines represent a push to host “h” localhost, on topic “t” tesla/input/2_01/state with message “m” ON or OFF. By rapidly firing them off shortly after each other, this simulates the push of a button (first on, then off again).

When the push button state updates are pushed:

home assistant updates the internal state of the push button entity
the automation kicks in, and updates the matching light entity is toggled
the light entity sends out the update on the command topic

Apart from controlling the light with the push button, it can also be controlled directly from home assistant; when toggling the light entity directly in home assistant, it just sends out the command on the MQTT topic directly.

Closing thoughts

This concludes my approach to using (abusing?) home assistant and the MQTT integration as the central part of my home automation setup.

I did use MQTT as the standard to centralize all my components on, have a look at the other posts presented in the home automation overview post of how to get there. Examples of other components that use MQTT in the same fashion are window contacts, PIR, smoke and water detectors, motorized blinds, etc.

The advantage of home assistant though is that it does not need to all be the same protocol, so it is quite easy to add other integrations in the mix as well. A notable example of this is an automation where I link one of these push buttons to a Philips Hue light; as far as home assistant is concerned, both of these are light entities.

A disadvantage of using home assistant this way and by extent the overall approach is the centralized nature of it. Home assistant needs to be online all the time for everything to work. Initially, I did run home assistant using docker-compose, which was quite stable. However, doing updates was never fun, since it meant that one container needs to go down before the next comes up, leaving in a short (but stressful) period of downtime. Using k3s as the supervisor certainly fixed that. Given that the majority of my interfacing hardware is actually all on MQTT, I might at some point add another service in the mix that mimics the automation engine as a backup.

A similar argument can be made about the mosquitto broker. Currently, I actually run that as a dedicated service on the host itself. Alternative solutions such as emqx exist though, which would run MQTT in a high-availability mode (multiple, load balanced instances).

Currently though, the system works quite fine and everyone at home has gotten so accustomed to it, they only really notice when not being at home. The backlog of improvements and extra bells and whistles is still quite long, so I’ll hope to write more on this later!

the mosquitto version used in the docker-compose.yaml version was explicitly pinned to a version < 2.x, which did not enforce authentication, hence why the broker configuration just needs a host name to connect to. ↩︎

Software

Tue, 27 Jul 2021 00:00:00 +0000

After connecting everything on the different “physical” layers, it is time to have a look at how to connect all on the software side of things. This post shall describe my current setup, specifically the different software components and the protocols they rely on for the unipi hardware.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Particularly relevant is the hardware post, since I my software builds on the unipi neuron platform.

Software

The following diagram shows the different software layers, going from the kernel which polls the IO-boards to eventually pushing out / pulling in MQTT events.

Kernel: polling SPI

Unipi provides their own neuron open source OS, which is a modified version of raspbian including custom kernel drivers to poll the I/O boards over SPI. The kernel driver makes the data available via the modbus protocol. The modbus protocol is a widely accepted, open industry standard for interfacing with PLCs. I believe you would typically use it in a setup where you have one main controller and a series of follower controllers - which is pretty much the situation you would have using SPI. Note that the unipi units also have an RS-485 connection, commonly used for daisy chaining multiple controllers.

Apart from the modbus interface, unipi also provides a sysfs interface, which essentially maps the I/O states and controls to a number of files in a fixed file system structure. I have used this sysfs interface in another project, unipitt, but since I am currently not using this one, I will not discuss this further.

Evok: modbus to web APIs

Unipi also provides an open source library called evok that will periodically poll the underlying modbus interface and make it available via all sorts of common web API formats and protocols: JSON / REST, JSONRPC, SOAP, … See the evok API docs for an extensive overview of all possible interfaces.

The most interesting interface provided by evok is the evok websockets interface, since this is the only interface that can also trigger actions for particular events. This is a definite requirement here, since you’d need to be able toggle the relays fast in case of an incoming request.

For the receiving of messages (I/O board → outside) you would need to register a number of callback functions to the websocket. The arguments to the callback function then hold information about the specific I/O action that triggered them. For the sending of messages (outside → I/O board), you can just send a structured message directly.

Here’s a python snippet directly taken from the evok websockets interface docs:

import json

import websocket

url = "ws://your.unipi.ip.address/ws"


def on_message(ws, message):
 obj = json.loads(message)
 dev = obj["dev"]
 circuit = obj["circuit"]
 value = obj["value"]
 print(message)


def on_error(ws, error):
 print(error)


def on_close(ws):
 print("Connection closed")


# receiving messages
ws = websocket.WebSocketApp(
 url,
 on_message=on_message,
 on_error=on_error,
 on_close=on_close,
)
ws.run_forever()

# sending messages
ws = websocket.WebSocket()
ws.connect(url)
ws.send('{"cmd":"set","dev":"relay","circuit":"3","value":"1"}')
ws.close()

`evok2mqtt`: websocket to MQTT

The final layer is to translate the websocket “events” from and to MQTT events. This translation is required since this is the standard message-based system interface that was chosen to interface with home assistant. In order to implement this, I did create a small application called evok2mqtt. The application is written in python3 and uses the websockets library for interfacing with the evok websocket and paho-mqtt to do the same thing for MQTT. This application runs alongside evok on the unipi neuron itself. Note that the application could in theory also run elsewhere on the network as the websocket and the MQTT broker can just be addressed over the network. Given that the program itself was actually quite simple and fun to write, I will outline how it works with some example code snippets.

To express actions that need to occur on the receiving of messages, the MQTT library works pretty similar to the websockets library using callback functions, see e.g. this evok2mqtt on_message snippet

def on_message(client, userdata, message):
 """Callback for MQTT events"""
 logger.info(
 f"Incoming MQTT message for topic {message.topic} with payload {message.payload}",
 extra={"kind": LOG_KIND.MQTT},
 )
 match = MQTT_COMMAND_TOPIC_REGEX.match(message.topic)
 if match is None:
 return

 device_name = match.group("device_name")
 dev = match.group("dev")
 circuit = match.group("circuit")
 if device_name != _settings().DEVICE_NAME:
 logger.warning(
 "Handling incoming message for device %s, expected %s",
 device_name,
 _settings().DEVICE_NAME,
 extra={"kind": LOG_KIND.MQTT},
 )

 # Update state topic
 client.publish(
 MQTT_TOPIC_FORMAT.format(
 device_name=device_name,
 dev=dev,
 circuit=circuit,
 hass_action=HASS_ACTION.STATE,
 ),
 message.payload,
 )

 # Send to websocket
 value = 1 if message.payload == _settings().MQTT_PAYLOAD_ON else 0
 logger.info(
 f"Push to output {dev}, {circuit}, {value}", extra={"kind": LOG_KIND.WEBSOCKET}
 )
 asyncio.run(_ws_trigger(dev, circuit, value))

This snippet essentially details all actions that occur for incoming events from the MQTT broker (originating from home assistant):

the message is logged
the message topic is checked to ensure whether we should process this message further¹
the state of the I/O is updated to acknowledge to home assistant that the update went OK
a call is issued to the websocket to do the update on the I/O

For the opposite action, a websocket event that needs to be translated to MQTT, a similar callback function is implemented.

async def _ws_loop():
 """Main loop polling incoming events from websockets"""
 logger.info(
 "Connecting to %s",
 _settings().WEBSOCKET_URI,
 extra={"kind": LOG_KIND.WEBSOCKET},
 )
 async with websockets.connect(_settings().WEBSOCKET_URI) as websocket:
 while True:
 payload = await websocket.recv()
 await _ws_process(payload)

This snippet sets up the websocket loop that will continually check the websocket for newly incoming messages. Note that this uses and async-await on the websocket, a feature provided in python>3.5’s asyncio library. The key thing to understand about asyncio here is that even though this is an infinite loop, during the waiting times for new data to come in from the websocket (or more generally, during the waiting time for any I/O), the thread scheduler can yield control to another thread to execute concurrently.

The function _ws_process that processes the incoming payload is next:

async def _ws_process(payload):
 """Process incoming websocket payload, push to MQTT"""
 obj = json.loads(payload)[0]
 logger.debug(
 "Incoming message for websocket %s", obj, extra={"kind": LOG_KIND.WEBSOCKET}
 )

 topic = MQTT_TOPIC_FORMAT.format(
 device_name=_settings().DEVICE_NAME,
 dev=obj["dev"],
 circuit=obj["circuit"],
 hass_action=HASS_ACTION.STATE,
 )
 payload = (
 _settings().MQTT_PAYLOAD_ON
 if obj["value"] == 1
 else _settings().MQTT_PAYLOAD_OFF
 )
 _mqtt_client().publish(topic, payload=payload)
 logger.info(
 "MQTT publish %s to topic %s",
 payload,
 topic,
 extra={"kind": LOG_KIND.WEBSOCKET},
 )

The operations it does, are quite straightforward:

load the object representation from the incoming payload (as JSON).
construct the MQTT topic based on the object name and circuit.
construct the MQTT payload from the object payload
publish the MQTT payload to the topic

There you have it: translation from-and-to websockets / MQTT. Note that this translation covers both inputs (push buttons) as well as outputs (relays).

Closing thoughts

At this point, with all of these layers of software in between, the unipi neuron unit provides a standard MQTT-based interface to home assistant.

The main downside I’ve felt at this point is that the reaction speed from the evok interface is sometimes a bit too slow to react quickly enough to push button toggles. On rare occasions, this can result in push buttons events being missed – which most of the times can be easily fixed by pushing the buttons a bit longer.

Also, having multiple of these translation layers makes the overall solution a bit more complex and will surely also add to the latency.

I do intend at some point to have another look at this setup. One option is to look at the (open source) implementation of evok and replace the async parts there with the more modern asyncio library. However, if performance is key, I will more likely have a look again at the sysfs interface and my unipitt project, which uses golang under the hood.

Nonetheless, this current setup has been working quite OK for me thus far.

Update from unipi

A while after this post went live, I did get some friendly feedback on my unipi forum post.

Some selected parts of the notes I think I should share²:

evok might get an update, including MQTT support!

The Evok is planned (well, maybe “wished” is a better word:)) to be rewritten to Python 3 and to a multi-threaded application. Along with that, major changes to the API will happen. We will abandon some of the less used APIs (RPC and SOAP probably) and definitely add native MQTT support since it is a driving force in the industry.

Missed state changes should be possible to detect with the digital input counter:

The missed state changes of the DI can be detected by checking the state of the DI counter. The counter is implemented in the HW and can count as fast as 10kHz signals.

The low-level communication I did outline using SPI and modbus is slightly different from what I did present above:

And a bit of explanation of the low level communication:

The HW boards are connected over SPI

The protocol on the SPI level is modified ModbusRTU

The Unipi kernel module (part of the unipi-kernel-modules package) exposes this communication channel as /dev/unipispi device

The Unipi ModbusTCP server (part of the unipi-modbus-tools) exposes this as a ModbusTCP server running on TCP/502

The rest is correct. Evok polls the ModbusTCP server as fast as it can. And creates a “system image” of the HW state.

The IO boards might also get a firmware-level upgrade!

And in the firmware of the IOs we plan to introduce a “latching” state of the DIs. Those will be registers and coils which will hold the leading edge of the DIs until they are read from the software. That will prevent the losing of the edge altogether.

I really appreciate this kind of feedback from the unipi team. It is nice to see how they are continuously improving their product (e.g. the idea of including native MQTT support). More importantly, I think this company really understands well the value of their product as an open platform and is actively working on community-building. Like I did mention in some of my earlier posts in this series, this openness is one of the reasons I am really glad about their products!

this could actually also be solved using configuration on the MQTT client setup, by only registering callbacks to certain topics. ↩︎
mostly literal, some formatting changes mine. ↩︎

Hardware

Tue, 20 Jul 2021 00:00:00 +0000

In this post, I will give some more details about the rationale for the hardware that I had chosen for interpreting the input signals and controlling the outputs, specifically the unipi neuron series. Note that I am in no way affiliated with unipi, nor do I intend to do an in-depth description of their hardware. For more details from unipi themselves, have a look at the unipi website.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Inside

The following picture shows what is actually inside such a unit: a raspberry pi 3B+ main controller board and then (depending on the neuron series you have), 1 to 3 I/O boards, connected via SPI – Serial Peripheral Interface.

The SPI interface works by connecting a number of nodes in a series, daisy-chained, on the same SPI-line(s). The raspberry pi functions as the main server node, the I/O boards are the clients. The server nodes periodically selects one of the clients to become active. Consequently, the selected client becomes active and puts its data on the SPI-lines that were made available. Other clients remain inactive for that duration of time they were not selected. Afterwards, the server moves on to the next client. This process continues until all clients have been consulted, at which point the cycle concludes and a new one starts.

The main controller board has all the other functionalities the raspberry pi has, including bluetooth, wifi, wired ethernet, USB ports, and even a HDMI interface¹. The I/O boards either feature digital inputs, relay outputs, analog inputs, analog outputs or a combination of these. Additionally, there are connections like a RS-485 serial lines (e.g. for interfacing with modbus) as well as a connector for a 1-wire bus (e.g. for temperature sensors).

Rationale

Considering the hardware platform, I think the unipi platform provides a number of advantages I couldn’t find with other vendors:

open platform: the main controller board is a raspberry pi 3B+, meaning you can run any software on it you would want.
software support: next to the open platform, they also provide open source OS images as well as software libraries to extend yourself.
local: obviously, it runs from my local network inside of my home.
wired: the I/O interfaces use standard voltages, for instance the power supply to the unit uses 24V, the digital inputs use 24V, the relay outputs can switch 240V², …
form factor: the modules come in form factors of 4, 8 and 12 DIN rail modules and fit nicely on a household DIN rail in the electric cabinet.
wiring connectors: related to the form factor, it is quite easy to connect a large amount of I/O in a relatively small space.
low cost: the units themselves do not come that cheap, but calculated as a cost per I/O, they are quite OK compared to other solutions like industrial PLCs
community: while working on the units and my own custom software, I would often reach out to the unipi community forum. They often reply quite quickly and have been a great help!
connectivity: apart from the I/O, the unipi units come with a lot of extra connectivity options such as wifi, wired ethernet, bluetooth, RS-485, one-wire, …

Specific units

I have number of different unipi units in my setup, since I have a large amount of functionalities to support, for example:

push button read-out
light control (relays)
alarm system (window / door contacts, PIR detector, … but also on the output side an indoor-and outdoor siren).
shades (relays)
…

While you can have both inputs and outputs on the same unipi unit, I did decide to dedicate a bigger unipi unit for the push button control and another one for the lights:

lights, relay control: unipi neuron l403, which has up to 56 relay outputs
push button read-out: the unipi neuron l303. This was an upgraded version of the unipi neuron m303 with up to 64 digital inputs, but at the time of writing, it seems this model is no longer supported.

Unipi axon

As a side note, I also have a unipi axon S605 for DALI light control. The unipi axon series is another series of PLCs from the same brand. It differs mainly in the controller board it features: instead of being based on raspberry pi, it is based on an Allwinner H5 ARM processor. Also, instead of having the main hard drive on a removable flash memory, it has internal eMMC memory.

I did not look into the rest of the axon series as it wasn’t available at the time when I acquired them. Additionally, I also favored the neuron series because I have more familiarity with the raspberry platform in general.

DALI light control is something I haven’t yet fully deployed, but the advantage is that you can have more all-digital control of lights – wherever the light fixtures support it. The main feature I do like about DALI is the ability to dim lights directly from the LED driver that already needs to perform the AC / DC conversion, making it more effective.

Note however that this specific unit also isn’t able anymore, I suspect due to low demand.

Closing thoughts

This was just a short description mainly about why I did choose unipi neuron as the main hardware platform for my home automation.

Note that this sort of hardware is particularly suited for a wired, star topology setup. If I were to name a disadvantage, it would be that working with this kind of hardware requires a lot of tedious manual wiring which also takes up a lot of cabinet space. This is however more of an issue of the overall setup, not the hardware itself. The alternative is a bus-system, where all of the signaling is done on a shared bus. The disadvantage here is either all of the hardware on the bus is more complex and costly (e.g. KNX, DALI) and / or that you are tied to a single hardware vendor. Nonetheless, DALI (and maybe even KNX) is something I would still like to examine a bit more in detail at some point in the future.

Another issue I have faced is with the removable flash drives as these tend to go corrupt after many read / write cycles. Again, this is more of an issue of the raspberry pi and flash drives in general, but it could still pose an issue for the reliability of the overall system. After changing the flash drives to a more durable pSLC SD card, I can confirm that I haven’t faced any issues with corrupt flash drives lately (fingers crossed!) Check the unipi SD card reference for more details.

Overall, I have been very happy with the unipi hardware platform though!

For more commercial details check the unipi main website. On the technical side, check out the unipi kb.

the HDMI interface isn’t readily exposed ↩︎
for highly inductive loads, like motors, it is recommended to switch the loads not directly, but instead via an intermediary set of relays ↩︎

Electrical cabinet

Tue, 13 Jul 2021 00:00:00 +0000

In earlier posts, I did provide an overview of the architecture of my custom-made home automation system. I also did describe some specificities related to a wired home automation setup. For my system, I decide on wiring everything in a star-configuration, meaning all wiring around the house for lights, push buttons, … all converges in a centralized electrical cabinet. The focus of this post is to show you some details of what this cabinet looks like and some pointers for practical issues I had run into.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Inside the cabinet

Given the star-configuration for all of the lights, the amount of wiring in the central electricity cabinet quickly adds up. The following picture shows a work-in-progress view while laying out the cables:

Be sure you have a consistent labelling scheme while pulling all of the cable. Looking back at these pictures, I think it is a bit of miracle I did not miss anything …

Terminal clamps

Besides from labelling the cables during the pulling of the cables, I also did use terminal clamps, like the WAGO rail-mount terminal blocks. On one end of such a block, you have one or more terminals where you can connect your wires. The other end on this block can then later be used to connect whatever you need.

The next figure details the terminal clamps I did use, the WAGO TOPJOB S 2003-7642.

The details include the nominal voltage and current ratings, wire gauges, the physical width of the clamp itself, etc. The lines next to the clamp indicate the different clamp variants available. The first 2 variants are 2 “stories” high (2 electrical connections on a single clamp) whereas the bottom 2 feature 3 “stories”.

Each story represents a single electrical connection, isolated from the other stories. The number of stories all have to do with saving space inside the cabinet: 2 stories really mean you need only half the amount of clamps. Additionally, since most mains voltage wiring is for single phase electrical connection, you can group the phase and neutral wire in one such clamp.

Aside from the first row indicating the details about the terminal clamp, the figure also details many different variants of jumpers available. These jumpers can be placed in the middle ends of the clamps and provide a connection between the different clamps, which facilitates making electrical parallel connections.

Functionally, these terminal clamps does not do all that much. In terms of costs, a single block is not that expensive but since the amount of blocks you need quickly adds up, so do the costs. Nonetheless, I would still highly recommend it to keep a clear overview while inside of your cabinet.

Relays

The next picture shows 2 DIN-rails with all of the terminal blocks for my lights.

top row: termination of phase (brown) and neutral (blue) wires.
bottom row: termination of 2 more wires which I intend to use as a [DALI] bus¹; DALI+ (grey) and DALI- (black).
jumpers: middle of terminal blocks
orange spacers: each set of blocks between these spacers represent a single electric circuit. This makes it easier to map the one-wire diagram to the actual wiring inside the cabinet.
green / yellow wires: these are terminated separately on the copper bar (visible between the two DIN rails) and directly connected to the main earth wire.
two-level terminal blocks: the terminal blocks used here have 2 levels, i.e. the outer 2 terminals are connected as well as the inner 2. A multitude of such terminal blocks exists, ranging for single to multiple levels as well as different functionalities (built-in diodes, resistors, …). Here, two-level terminal blocks were used mostly because of space constraints.

Note that at this point that the wires from the light fixtures in the house were just connected to these terminal blocks: having these blocks there means you can later still have all flexibility for connecting them elsewhere.

The next part is connecting them to the unit that contains the relays themselves. The unipi relay outputs KB lists all of the details for connecting a single relay:

This connection in itself is not all that complex for a single relay. Since multiple lights on the same circuit are connected to the same circuit breaker, there is the practical concern of how to ensure these are all properly connected in parallel while still maintaining again a good overview. The following picture shows another set of terminal blocks that I did use for this (orange blocks on the left of the unipi unit):

phase wire (brown): the terminal block holders (orange, left from the unipi module) hold a set of terminal blocks which have one end connected to the circuit breaker, the other ends serve to distribute the voltage to multiple relays.
neutral wire (blue): since all neutral wires of a single circuit are placed next to each other, these can just be connected in parallel using a jumper.
relay: each of the lights are connected to “RO” connections coming from the unipi module.

Push buttons

Next to the series of clamps for terminating the cables for the lights, I also did use the same type of clamps for terminating the SVV analog signaling cable:

At the top, the incoming signaling cable is terminated. Spacers are placed series of clamps which belong to the same signaling cable run. The bottom part then connects each of the signaling wires to a digital input somewhere lower in the cabinet. In the middle of the clamps, you can see a number of overlapping jumpers. These jumpers have been set up such that they only make an electrical contact in a fixed number of slots, in this case for connecting one of the wires for each cable run to 24V.

As before, the following image shows the electrical connection from the signaling cable to the unipi digital input unit. The DIGND corresponds to the ground level of your 24VDC power source. The DI level is the signaling cable to one of the push buttons. Pushing a button means the 24V level gets connected to the signal cable and thus giving a DI level of 24V. For more technical details, consult the unipi digital input KB.

As can be seen from the following picture, making the physical connection between the incoming signal wires and the unipi units is easier than for the relay unit, shown earlier.

There is no need to have dedicated circuit breakers for the different signal wire cable runs; all of them can run on the same 24VDC power source. Also, since the wire diameter is much lower, the screw connectors can just be used directly to terminate multiple wires and effectively provide electrical parallel connections. Since each of the green screw connectors could have up to 4 signal wire connections and 1 ground level, I did use 4-wire SVV cable to connect them to terminal clamps higher up in the cabinet. Note that I also did use my color coding again here (red / yellow / blue / white).

Closing thoughts

I started this post with some scary pictures of electrical wiring spaghetti.

However, it turns out that with proper planning, schematics and some determination, it was actually quite doable to wire the full cabinet together. Terminal blocks also help a lot in this process: it makes it easier to visually separate everything, removes the need to do all of the wiring in the beginning and enables rewiring at some later point in time.

As before in the series, I did try to explain how to make the connection for push buttons (inputs) and relays (outputs). The connections for the inputs were a bit simpler in the sense that the wiring runs on a safer lower voltage of 24VDC. For the relays, as it runs on mains voltage, there are the additional constraints of the need to add in a circuit breaker for each electrical circuit as well as a higher wire gauge. Consequently, I did some extra clamps to ensure the proper connections.

I did opt for this approach at the time, since it seemed the most simple to me (comes closest to a “classical” setup), the most flexible in terms of rewiring (all wires are within reach) and does not depend on some proprietary, more expensive closed-off bus system. In hindsight, the main disadvantage of such a central star-configuration is the excessive amount of wiring you need to deal with. For costs, you have the additional costs of the wiring material itself. In terms of work, I did do all of the connections myself, so even though I did not lose any money on that, it surely was a very long, laborious process I might rather avoid in the future. With that in mind, at some point in the future I will surely have a closer look at bus-based systems.

DALI is a bus system for digital light control. In theory, you could do full light control with DALI only (removing the need for relay-based control), but this means all of your light fixtures need to be DALI-aware. ↩︎

Schematics

Tue, 06 Jul 2021 00:00:00 +0000

In previous posts, I did discuss the importance of planning and design as well as details on the physical wires to be used.

Here, I’d like to focus on two types of schematics which are not only very useful, but also mandatory for doing the electrical wiring for a Belgian house: the one-wire diagram and the floor plan.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

One-wire diagram

The one-wire diagram is an easy representation of all of the circuits and its related consumers and actors in an electrical installation. As its name implies, it does not show each of the individual wires of the installation but rather how the cables are laid out logically. Initially, this diagram seemed a bit strange to me in the sense that it seems overly simple and does not take the different conductors into account, i.e. the neutral, phases, earth wire … From the point of view of an installer, this diagram actually does make a lot of sense and includes all the details you need.

Note that I no longer make the distinction between push buttons and relays, as was the case in the previous posts.

According to my local regulations, it is not required and even discouraged to include low voltage wiring on the one-wire diagram since it adds to the overall complexity of the diagram. This does have some implications on the representation of light-related circuits though.

In a “traditional” light circuit, a light toggle switch forms a direct mechanical connection to the mains voltage, either through terminating one (one-pole connection) or two (two-pole connection) of the conductors in the wire. Two-pole connections are considered more safe and are even mandatory for humid environments. In terms of the one-wire diagram, the diagram would reflect which toggle switch would connect to which light.

For my setup however, none of the lights are directly connected to the push buttons as these run on their own circuit of 24VDC, which is not on the diagram. Given the push button assignment is controlled by software, it might not even make sense to include them in this diagram. As for safety, since the push button is nowhere near the physical mains voltage cabling, this obviously much safer.

To illustrate, here’s a small annotated (red) part of my installation’s one-wire diagram:

The bottom left corner holds the entry-point from the street level where the main cable enters the house. In this case, it is a 400V three-phase connection: 3 phase wires plus 1 neutral¹.
Straight after entering the house, the main cable passes through the utility company’s power meter.
Main residual current device, ΔI=300 mA. This means that if the circuit breaker detects a leakage current ≥ 300 mA, it will automatically shut down the entire installation. Note that for Belgium, the 300 mA is mandatory for the overall installation, for circuits that pass through more humid environments (e.g. bathrooms, toilets), an additional 30 mA residual-current device is required.
At this level, all of the various circuits are placed in parallel, each with their own circuit breaker.
Wire specs: note that the circuit breakers need to match the requirements of the wires and the related consumers, as indicated in the physical wires post.
This level just illustrates two kinds of consumers; first a power socket, then a generic consumer.
Another example with emergency lighting.
An example circuit with a 240VAC / 24VDC transformer.
An example circuit for my solar panel installation.
This is a circuit exclusively containing lights: each of the branches indicates which lights are wired against the same control point. This means that each of the branches are switched using the same switch, or in this case relay. Note that this does not relate to how they are wired electrically: typically everything is wired in parallel².
This is another example of a circuit with lights.

Floor plan

Next to the one-wire diagram, there’s an additional diagram that lays out the circuits and their consumers on a floor plan. Low voltage wiring is also not required for the official floor plan, but it does make sense to have it mapped out somewhere, even if that would be on a separate diagram. The push buttons are obviously also on this plan.

Essentially, it just re-uses the numbering from the one-wire diagram presented earlier, so I won’t discuss these further.

Closing thoughts

Even though these diagrams only represent part of the information, they proved to be really valuable to me along the way:

planning: since there’s a finite number of consumers you can add to a circuit, it forces you to think ahead of the number of consumers on a single wire run.
installation: on the construction site, you really don’t want to spend too much time (re)thinking your setup.
posterity: in case anything breaks down (e.g. a circuit breaker switching off), you just want an easy to digest overview of the whole thing.

the voltage between each phase wire and the neutral wire is 240V ↩︎
although it could be wired in series, e.g. for a series of current-controlled LED fixtures ↩︎

Wiring

Tue, 29 Jun 2021 00:00:00 +0000

In this post, I will take a deeper dive into the bottom layer, the wiring. Different kinds of inputs and outputs can be discussed, but for the sake of simplicity, I will focus on a push button operating a lamp. These components will also come back later on in the series.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

I will cover:

caution: safety first
principle: operating principle behind a light push button and relay
conductors: the physical wires and cables themselves

A mandatory word of caution (!)

This post will touch a number of points where working with mains voltage is required. If you do not have any background in working with electricity, please be sure you get help from a qualified electrician. Also, some of the details really only apply to my local regulations (Belgium), so make sure you are well aware of your local regulations. Related to this, some of the terminology used might be a poor translation of some term which would have been straightforward in my native language (Dutch). Even though I had a background as an electrical engineer, I still got a local professional electrician involved to ensure the overall safety …

Principle

Relays

A relay is an electrically operated switch. The voltage over two of its terminals can then control (switch) the voltage over the two other terminals. The control voltage will typically be on a safe, low voltage, whereas the other terminals can carry higher (mains) voltage. You can use it to control any other electrically switched device, e.g. lights or motors.

Push buttons

Where the relay serves as the output, the push button, together with a digital input sensor, serves as the input. The push button itself is actually a simple mechanical contact. Pushing the button results in the internal mechanical contact making an electrical contact.

There are two types of contacts: a normally open (NO) contact or a normally closed (NC) contact. Normally open means that the mechanical contact maintains an open electrical contact for the majority of the time, unless the button is pressed. Normally closed is then the opposite situation, meaning a closed electrical contact whenever the button is not pressed. For push buttons, such as those to operate lamps, an NO contact would typically be used. NC contacts make sense for situations where you want to be sure that the electrical contact is always guaranteed, e.g. a sensor for an alarm system. Loss of signal for an NC contact might indicate the sensor has been tampered with.

The voltage used for a push button is typically not mains voltage (240VAC), but rather a lower, safer voltage like 24VDC. Typical digital input sensors such as those found on DIY devices like the arduino or raspberry pi would operate on 5V or 3.3V, respectively. For wiring a house, this is not an option because of the longer line runs and the resulting voltage drop related to the resistance of the wiring itself.

An important thing to consider in the push button - lamp constellation is that the way the push button is used is stateless. Considering the lamp is off, one push would switch the light on. Another push would switch it back off again. Based on the position of the push button itself, you would not be able to tell the state of the light. It differs from a rocker switch in that it only maintains the contact for the duration of the push of the button. While these observations might be rather trivial, for the purposes of modeling them higher up on the stack, they are relevant.

Conductors (Belgium only)

Relays

The following image shows a typical tube containing 3 wires with diameters of 1.5mm², fit for lights.

For the cables carrying mains voltage, the following rules are relevant (non-exhaustive see e.g. ¹, ², project huisinstallatie naslagwerk):

for circuits up to 16A (typically lights), a min 1.5 mm² cross-section wire is to be used
for circuits up to 20A (lights and / or others like power sockets), a min 2.5 mm² cross-section wire is to be used
each of the circuits need to have proper matching circuit breakers
all cables consist of neutral wire, live wire and an earth wire
color coding:
- neutral: blue
- live: brown
- earth wire: green / yellow
max 8 consumers (lights, power sockets) per circuit

Keep in mind:

circuit breakers protect your wires from burning up
residual-current devices protect living things!

Note that this is only a subset of the rules and pointers to keep into account while doing the physical wiring. Other things to consider are the type of cable (in a plastic tube or as a cable), solid vs stranded wire, stiff vs flexible wire, etc. …

Push buttons

Contrary to the cabling for the lights, the push buttons run on the lower, safer voltage of 24VDC. Consequently, the wire diameter can be a lot smaller and there is no strict requirement on the use of circuit breakers. An additional advantage in terms of safety compared to a more “traditional” installation is that the push button itself is never connected to the consumer on the higher mains voltage.

The following image shows an SVV signal cable, which is typically used in Belgium for analog signaling applications. I did some online research into the international naming for this type of cable, but there does not seem to be any. For the sake of the discussion here however, the characteristics are actually more important:

small diameters, on the order of 0.8mm²
solid core, as this makes connecting to the push buttons easier as they mostly have screw connectors
multiple wires in 1 cable (as shown in the picture). These cables mostly come with either 4, 8, 12 or 16 wires.
visually clearly differing wires, i.e. highly contrasting colors for the wires

The idea of having this multitude of wires in one signal cable is that you would ideally need one cable run for e.g. the push buttons for one floor. Each push button contact then uses one of the wires of the signal cable. Apart from the push button contacts, you also need to reserve one of the wires for the positive level. Essentially, the push buttons thus internally connect 1 signal wire (on ground level) to the common positive level – which can be detected by your digital input readout.

As a side note, some manufacturers also have push buttons with a built-in LED light. If you want to provide power to that LED, you would also need to reserve an extra wire for that feature.

Concerning the various colors available in the signal cable, I did find it useful to have a fixed order in which to assign them to the different inputs. I would always use the red wire for the positive (24V) level. The remainder of the wires would then be assigned according to the colors on the visible electromagnetic spectrum. For example, you could go from black, grey, brown, orange, green, blue, … all the way to white.

The following image shows the wiring diagram for a simple NO push button from Niko, a Belgian manufacturer of electrical switches and sockets.

The diagram indicates the push button can be used to switch voltages up the 230VAC, but since we’re only interested in for the signaling application, it might just as well be a lower voltage. Also visible is the optional LED light.

Closing thoughts

The focus of this post was first to describe the basic principle of some key components, the relay and the push button. I did want to highlight what exactly they are and also how the operating principle relates to the modelling higher up in the stack, e.g. the fact that I mostly consider the push button to be stateless.

The second part of the post dealt with the details of the physical electrical conductors connecting to and from these components. Relays, for lights, use 3 x 1.5mm², whereas higher loads require a higher wire gauge of 3 x 2.5mm². The push buttons can use a lower wire gauge of e.g. 0.8mm² since you would run these on a safer, lower voltage of 24VDC. While the regulations pertaining to the wiring are specific to my country’s regulations, I still hope someone else might learn from it!

Finally, always keep safety into account! Even you feel you are really knowledgeable, I still think it was an added value for me to involve a local professional electrician to get proper hands-on advice.

ISBN/EAN 9789030142942 ↩︎
the official Belgian regulation is the AREI ↩︎

Architecture

Tue, 22 Jun 2021 00:00:00 +0000

Prior to actually building anything, it is important to think on a higher level what the various components are and how they will interact with each other, i.e. the architecture. Over the process of doing my own setup, I came across a large number of different possible configurations, each with their own benefits and downsides. For the first part of this post, I will focus on the final layout.

While discussing the architecture, it is important to realize there are a number of different layers of abstraction to reason about these. This will be the topic of the second part of this post.

To make everything more tangible, I did add a third section detailing toggling a light with push button.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts.

Architecture

The following diagram gives an high-level overview of the overall architecture.

Its main characteristics are:

distributed communication:
- communication is (mostly) done via an event-based system, MQTT
centralized wiring:
- all I/O (push buttons, relays for lights, …) is connected to modules which fit in a centralized electric cabinet
- there is a central service with automations connecting the different modules (home assistant)

Let’s elaborate a bit more on the constituent components.

MQTT broker

In the center of the drawing is a mosquitto MQTT broker. MQTT is a publish-subscribe protocol:

there is one central instance, called the broker, which manages different topics
different clients can publish updates, called events, to the topics
in addition, all clients can also subscribe to topics to receive the events

MQTT is a popular choice for IoT since its event-based model fits well for a lot of problems (e.g. a push button toggle). Another advantage is that the each of the different clients only needs to maintain a connection to the broker and the topics they want to be subscribed to. This makes for a very loosely coupled and scalable architecture: it is quite easy to add new components. A disadvantage however is that all connections rely on the availability of the central broker.¹

MQTT clients

The principal other components of the setup are all MQTT clients which both subscribe and publish to specified topics, specifically:

subscribe: they listen to topics which instruct them to do some behavior (“command topics”)
publish: they publish events to topics that indicates their state (“state topics”)

The clients in this setup can provide (a combination of):

sensors: push buttons, motion sensors, smoke detectors, …
actors: lights, covers, sirens, …
logic: send out events on a certain condition, e.g. on another event (an automation)

unipi

For the sensors and actors I did use hardware from unipi. In essence the unipi neuron units are DIN-rail mountable units housing a raspberry pi and a couple of I/O boards depending on the model you have. The role of the unipi units is thus to provide an MQTT event-based interface for all home automation I/O: reading push buttons, toggling lights, controlling shades, reading smoke and movement detectors, etc.

home assistant

Whereas the I/O modules themselves are distributed both physically and logically, I did want to centralize the controlling logic in a dedicated service. Cue in home assistant.

home assistant is an open source software project which aims to integrate all sorts of smart home devices. Given my choice of MQTT as an interfacing protocol throughout my setup, I heavily rely on the home assistant MQTT integration.

From a high level, all sensors and actors which are available as MQTT topics can be represented as entities within the home assistant configuration. The home assistant instance itself connects to the central MQTT broker and listens to those MQTT topics for which it has entities. Automations, which are also part of the configuration, can then link actions from one entity to another. An obvious example is the push of a button that switches a light on. The rationale behind using automations to link inputs with outputs, is that nothing is really hard-wired in this setup!

Note that home assistant supports a wide range of integrations beyond MQTT. The thermostat in the schematic for instance has an integration based on REST / HTTP.

Layers of abstraction

Planning for a DIY wired open home automation setup requires some other specific thinking in terms of the different layers of abstraction. The following diagram focuses on these different layers.

From low to high level:

wiring: the physical wiring which connects to electrical components such as push buttons and light fixtures
hardware: the physical hardware component that will interface with the electrical components (I/O).
software: a client library interfacing with the hardware and which can translate low-level hardware events to protocol events
network: layer which connects the different components together using the protocols
service: the services are those parts which actually talk to each other over the network

Notice that not all components need to have all of the layers. Services like home assistant and the mosquitto broker act completely on the network itself.

Why does this matter? For the majority of commercially available IoT devices, you would get something that takes care of most of the lower levels. Wiring mostly isn’t an issue (it is typically wireless) and the hardware and software are part of the package that you buy. In this wired layout with custom components however, all of these layers need to be addressed.

As an example, have a look at the wiring I did for my cover control:

Most of my wiring was done in a star topology. This means for example that you do need a single cable run for each of the individual (group of) lights you want to control. The upside is that this is very flexible towards the future, since physical rewiring is still possible from the electrical cabinet. The (visible) downside is that the amount of wires piles up rapidly and you will need to think of some kind of system to keep a sane overview of this.

Bus-based systems

Alternatively, the wiring could have been done with a bus-based system like KNX or DALI. The advantage of systems like these is that the wiring would have been a lot easier and they can also provide some of the functionalities on other parts of the stack. For the case of KNX for instance, each of the components (push buttons, light control, …) are provisioned individually, resulting in a fully distributed, more fault-tolerant system. Note that there is a home assistant KNX integration, so even in this case, it could fit the proposed architecture (but obviously less reliant on MQTT).

At this point, I did not choose for a bus-based system though. The first reason was one of cost, notably in the case of KNX. Also, I did want the possibility to easily (re)program my setup using general purpose programming languages, whereas KNX relies on (costly, vendor-restricted) visual programming tools. All of my light sockets are wired with 5-wire cable, so at some point I might have another look at using DALI for controlling my lights.

Toggling a light switch …

To illustrate the theory, let’s just walk through all the steps of toggling a push button:

a push button is pushed down, making contact (off-on) and then released again (on-off)
the unipi digital input unit sees this change and translates this to an MQTT state change event which is pushed to the central mosquitto broker
the home assistant instance has a subscription on the broker for the input button state change and updates its internal entity state
the home assistant instance has an automation configured for the push button entity and triggers the matching light entity to an action
the light entity update pushes out another event to the light action topic to turn on
another unipi unit which is connected to the physical light and has a subscription to the light action topic pulls in the update
the unipi unit toggles a relay, switching on the light
finally, after the light has turned on, a state update is also pushed out from the unipi unit, which updates the internal state on home assistant

Closing thoughts

The focus of this post was to outline the main components and links in my home automation setup, principally based on MQTT and home assistant as the brains behind it all.

The main advantage of this setup is the flexibility if offers since everything is configurable from within home assistant. On the other side of the equation, once home assistant breaks down, the entire system can break down, which poses high reliability constraints on such central components.

I did not yet dive down into the details of the different components i.e. the sensors and actuators, most notably the unipi platform. Also, the different applications themselves, such as push buttons, lighting fixtures, DALI, shades, an alarm system, … all merit their own discussion. Finally, I did also point the different layers of abstraction to consider when laying out a wired home automation system.

This should conclude the “theoretical” part of the home automation series. Check the home automation overview post for the more “practical” upcoming posts.

although even this could be mitigated using a highly available MQTT broker. EMQX seems to be an example of this, but I haven’t tried it myself yet. ↩︎

Why

Tue, 15 Jun 2021 00:00:00 +0000

This post shall be the first in a series of blog posts where I intend to outline my home automation setup.

For this first post, I want to start by first explaining why I worked on this. Secondly, I want to give some more attention to the vision and guiding principles I will use further on. At the end, there is an overview of all the future posts that I have currently planned for this series.

Why

I am an electrical engineer by background, specialized in signal processing and programming for embedded devices (high complexity on small systems). Within my career, I did make the switch to software engineering, doing the opposite (low complexity on large systems).

One of the things I really did enjoy while building this system (and by extension: my house) was being able to build something from the ground up. Along the way, I was able to learn about a myriad of things: electricity and lighting, IoT hardware, various concepts, paradigms, languages … related to software development, networking and network-related infrastructure … Non-technical aspects, such as project management, were also quite important.

Even though I feel there’s still a lot of room left for improvement, I did consider that now was a good time do a write-up of everything accomplished thus far. Firstly, I believe that writing everything down would help me in documenting everything. Additionally, this provides the opportunity to reflect back on those things that went well as well as other parts that might still benefit from some improvements. The process of writing itself is also a good exercise to improve on my technical writing (in a non-native language). Possibly these posts might help or even inspire my readers to get into this topic. Lastly, I do want to reach out to a larger community and get feedback.

Vision

There are many different approaches to build a home automation system. Since the open source home assistant platform plays a central role in my system, I did borrow some inspiration from home assistant’s vision.

Considering my own ideal system, the following are the key ideas.

Wired

A wired solution has the advantage of not being affected by unreliable wireless links. Also, there’s generally no issue of ensuring a power supply, so there is no need to keep an eye on batteries. The disadvantage is however that the overall setup is less flexible and needs to be thought out really well in advance. The process of wiring itself can also be quite time consuming, costly or simply impossible, depending on your housing situation.

Open

All hardware and software should be “open” in the sense that it can readily understood and changes can be made to it. It should be possible to take a given component and swap it out for another one, unrelated to the original vendor. On the software-side, it should be possible to program against it with standard protocols using general-purpose programming languages. Combining different sorts of hardware and software should be straightforward.

Local

The home automation should always work, even in the absence of an Internet connection. The data from the home automation system itself should be owned by yourself. This does not mean that you cannot use common network protocols, you can still have everything connected via a local network (LAN). Nor does this mean that you cannot have non-critical integrations which are web-based, e.g. an integration that pulls in a weather forecast. Ideally though, the majority of communications the home automations needs to do to function properly should be in the home itself.

Low cost

Home automation systems have been around for quite some time, but what has changed the latest years is the availability of both hardware and software that allows to mix and match. Also open hardware platforms like the raspberry pi have made it much easier to develop own platforms. The disadvantage of such DIY platforms is the lower intrinsic robustness, compared to an off-the-shelf commercial solution.

Reproducible

This is a key idea coming from my background as a software developer. Essentially, this means that the provisioning and deployment of the overall setup should be automated to the extent that in case of any failure it can readily be reconstructed.

Posts

In the following series of blog posts, I will outline more of the details of the actual setup that I did design and build.

why introduction to the series, this post
architecture topology of the home automation components and different levels of abstraction
wiring details about basic components (relays and push buttons) and the related wiring
schematics how to represent the electrical wiring in a one-wire diagram and floor plan
cabinet a peek inside my electrical cabinet
hardware details about the central hardware units I did use
software different layers of interfacing software
service bringing it all together with home assistant

Home Automation on mhemeryck

modbusbackup

Recap

Problem: centralized points of failure

Solution: direct link

Hardware

Software

Client code

Server code

Configuration format

Closing thoughts

evok2mqtt ansible

Rationale

Automated provisioning

Initial setup

Running evok2mqtt-ansible

A closer look

Closing thoughts

Service

Architecture

home assistant

Scenario

Containers and services

Sample setup

docker-compose.yaml

configuration.yaml

ui-lovelace.yaml

Demo

Closing thoughts

Software

Software

Kernel: polling SPI

Evok: modbus to web APIs

evok2mqtt: websocket to MQTT

Closing thoughts

Update from unipi

Hardware

Inside

Rationale

Specific units

Unipi axon

Closing thoughts

Electrical cabinet

Inside the cabinet

Terminal clamps

Relays

Push buttons

Closing thoughts

Schematics

One-wire diagram

Floor plan

Closing thoughts

Wiring

A mandatory word of caution (!)

Principle

Relays

Push buttons

Conductors (Belgium only)

Relays

Push buttons

Closing thoughts

Architecture

Architecture

MQTT broker

MQTT clients

unipi

home assistant

Layers of abstraction

Bus-based systems

Toggling a light switch …

Closing thoughts

Why

Why

Vision

Wired

Open

Local

Low cost

Reproducible

Posts

Running `evok2mqtt-ansible`

`docker-compose.yaml`

`configuration.yaml`

`ui-lovelace.yaml`

`evok2mqtt`: websocket to MQTT