Unipi on mhemeryck

Software

Tue, 27 Jul 2021 00:00:00 +0000

After connecting everything on the different “physical” layers, it is time to have a look at how to connect all on the software side of things. This post shall describe my current setup, specifically the different software components and the protocols they rely on for the unipi hardware.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Particularly relevant is the hardware post, since I my software builds on the unipi neuron platform.

Software

The following diagram shows the different software layers, going from the kernel which polls the IO-boards to eventually pushing out / pulling in MQTT events.

Kernel: polling SPI

Unipi provides their own neuron open source OS, which is a modified version of raspbian including custom kernel drivers to poll the I/O boards over SPI. The kernel driver makes the data available via the modbus protocol. The modbus protocol is a widely accepted, open industry standard for interfacing with PLCs. I believe you would typically use it in a setup where you have one main controller and a series of follower controllers - which is pretty much the situation you would have using SPI. Note that the unipi units also have an RS-485 connection, commonly used for daisy chaining multiple controllers.

Apart from the modbus interface, unipi also provides a sysfs interface, which essentially maps the I/O states and controls to a number of files in a fixed file system structure. I have used this sysfs interface in another project, unipitt, but since I am currently not using this one, I will not discuss this further.

Evok: modbus to web APIs

Unipi also provides an open source library called evok that will periodically poll the underlying modbus interface and make it available via all sorts of common web API formats and protocols: JSON / REST, JSONRPC, SOAP, … See the evok API docs for an extensive overview of all possible interfaces.

The most interesting interface provided by evok is the evok websockets interface, since this is the only interface that can also trigger actions for particular events. This is a definite requirement here, since you’d need to be able toggle the relays fast in case of an incoming request.

For the receiving of messages (I/O board → outside) you would need to register a number of callback functions to the websocket. The arguments to the callback function then hold information about the specific I/O action that triggered them. For the sending of messages (outside → I/O board), you can just send a structured message directly.

Here’s a python snippet directly taken from the evok websockets interface docs:

import json

import websocket

url = "ws://your.unipi.ip.address/ws"


def on_message(ws, message):
 obj = json.loads(message)
 dev = obj["dev"]
 circuit = obj["circuit"]
 value = obj["value"]
 print(message)


def on_error(ws, error):
 print(error)


def on_close(ws):
 print("Connection closed")


# receiving messages
ws = websocket.WebSocketApp(
 url,
 on_message=on_message,
 on_error=on_error,
 on_close=on_close,
)
ws.run_forever()

# sending messages
ws = websocket.WebSocket()
ws.connect(url)
ws.send('{"cmd":"set","dev":"relay","circuit":"3","value":"1"}')
ws.close()

`evok2mqtt`: websocket to MQTT

The final layer is to translate the websocket “events” from and to MQTT events. This translation is required since this is the standard message-based system interface that was chosen to interface with home assistant. In order to implement this, I did create a small application called evok2mqtt. The application is written in python3 and uses the websockets library for interfacing with the evok websocket and paho-mqtt to do the same thing for MQTT. This application runs alongside evok on the unipi neuron itself. Note that the application could in theory also run elsewhere on the network as the websocket and the MQTT broker can just be addressed over the network. Given that the program itself was actually quite simple and fun to write, I will outline how it works with some example code snippets.

To express actions that need to occur on the receiving of messages, the MQTT library works pretty similar to the websockets library using callback functions, see e.g. this evok2mqtt on_message snippet

def on_message(client, userdata, message):
 """Callback for MQTT events"""
 logger.info(
 f"Incoming MQTT message for topic {message.topic} with payload {message.payload}",
 extra={"kind": LOG_KIND.MQTT},
 )
 match = MQTT_COMMAND_TOPIC_REGEX.match(message.topic)
 if match is None:
 return

 device_name = match.group("device_name")
 dev = match.group("dev")
 circuit = match.group("circuit")
 if device_name != _settings().DEVICE_NAME:
 logger.warning(
 "Handling incoming message for device %s, expected %s",
 device_name,
 _settings().DEVICE_NAME,
 extra={"kind": LOG_KIND.MQTT},
 )

 # Update state topic
 client.publish(
 MQTT_TOPIC_FORMAT.format(
 device_name=device_name,
 dev=dev,
 circuit=circuit,
 hass_action=HASS_ACTION.STATE,
 ),
 message.payload,
 )

 # Send to websocket
 value = 1 if message.payload == _settings().MQTT_PAYLOAD_ON else 0
 logger.info(
 f"Push to output {dev}, {circuit}, {value}", extra={"kind": LOG_KIND.WEBSOCKET}
 )
 asyncio.run(_ws_trigger(dev, circuit, value))

This snippet essentially details all actions that occur for incoming events from the MQTT broker (originating from home assistant):

the message is logged
the message topic is checked to ensure whether we should process this message further¹
the state of the I/O is updated to acknowledge to home assistant that the update went OK
a call is issued to the websocket to do the update on the I/O

For the opposite action, a websocket event that needs to be translated to MQTT, a similar callback function is implemented.

async def _ws_loop():
 """Main loop polling incoming events from websockets"""
 logger.info(
 "Connecting to %s",
 _settings().WEBSOCKET_URI,
 extra={"kind": LOG_KIND.WEBSOCKET},
 )
 async with websockets.connect(_settings().WEBSOCKET_URI) as websocket:
 while True:
 payload = await websocket.recv()
 await _ws_process(payload)

This snippet sets up the websocket loop that will continually check the websocket for newly incoming messages. Note that this uses and async-await on the websocket, a feature provided in python>3.5’s asyncio library. The key thing to understand about asyncio here is that even though this is an infinite loop, during the waiting times for new data to come in from the websocket (or more generally, during the waiting time for any I/O), the thread scheduler can yield control to another thread to execute concurrently.

The function _ws_process that processes the incoming payload is next:

async def _ws_process(payload):
 """Process incoming websocket payload, push to MQTT"""
 obj = json.loads(payload)[0]
 logger.debug(
 "Incoming message for websocket %s", obj, extra={"kind": LOG_KIND.WEBSOCKET}
 )

 topic = MQTT_TOPIC_FORMAT.format(
 device_name=_settings().DEVICE_NAME,
 dev=obj["dev"],
 circuit=obj["circuit"],
 hass_action=HASS_ACTION.STATE,
 )
 payload = (
 _settings().MQTT_PAYLOAD_ON
 if obj["value"] == 1
 else _settings().MQTT_PAYLOAD_OFF
 )
 _mqtt_client().publish(topic, payload=payload)
 logger.info(
 "MQTT publish %s to topic %s",
 payload,
 topic,
 extra={"kind": LOG_KIND.WEBSOCKET},
 )

The operations it does, are quite straightforward:

load the object representation from the incoming payload (as JSON).
construct the MQTT topic based on the object name and circuit.
construct the MQTT payload from the object payload
publish the MQTT payload to the topic

There you have it: translation from-and-to websockets / MQTT. Note that this translation covers both inputs (push buttons) as well as outputs (relays).

Closing thoughts

At this point, with all of these layers of software in between, the unipi neuron unit provides a standard MQTT-based interface to home assistant.

The main downside I’ve felt at this point is that the reaction speed from the evok interface is sometimes a bit too slow to react quickly enough to push button toggles. On rare occasions, this can result in push buttons events being missed – which most of the times can be easily fixed by pushing the buttons a bit longer.

Also, having multiple of these translation layers makes the overall solution a bit more complex and will surely also add to the latency.

I do intend at some point to have another look at this setup. One option is to look at the (open source) implementation of evok and replace the async parts there with the more modern asyncio library. However, if performance is key, I will more likely have a look again at the sysfs interface and my unipitt project, which uses golang under the hood.

Nonetheless, this current setup has been working quite OK for me thus far.

Update from unipi

A while after this post went live, I did get some friendly feedback on my unipi forum post.

Some selected parts of the notes I think I should share²:

evok might get an update, including MQTT support!

The Evok is planned (well, maybe “wished” is a better word:)) to be rewritten to Python 3 and to a multi-threaded application. Along with that, major changes to the API will happen. We will abandon some of the less used APIs (RPC and SOAP probably) and definitely add native MQTT support since it is a driving force in the industry.

Missed state changes should be possible to detect with the digital input counter:

The missed state changes of the DI can be detected by checking the state of the DI counter. The counter is implemented in the HW and can count as fast as 10kHz signals.

The low-level communication I did outline using SPI and modbus is slightly different from what I did present above:

And a bit of explanation of the low level communication:

The HW boards are connected over SPI

The protocol on the SPI level is modified ModbusRTU

The Unipi kernel module (part of the unipi-kernel-modules package) exposes this communication channel as /dev/unipispi device

The Unipi ModbusTCP server (part of the unipi-modbus-tools) exposes this as a ModbusTCP server running on TCP/502

The rest is correct. Evok polls the ModbusTCP server as fast as it can. And creates a “system image” of the HW state.

The IO boards might also get a firmware-level upgrade!

And in the firmware of the IOs we plan to introduce a “latching” state of the DIs. Those will be registers and coils which will hold the leading edge of the DIs until they are read from the software. That will prevent the losing of the edge altogether.

I really appreciate this kind of feedback from the unipi team. It is nice to see how they are continuously improving their product (e.g. the idea of including native MQTT support). More importantly, I think this company really understands well the value of their product as an open platform and is actively working on community-building. Like I did mention in some of my earlier posts in this series, this openness is one of the reasons I am really glad about their products!

this could actually also be solved using configuration on the MQTT client setup, by only registering callbacks to certain topics. ↩︎
mostly literal, some formatting changes mine. ↩︎

Hardware

Tue, 20 Jul 2021 00:00:00 +0000

In this post, I will give some more details about the rationale for the hardware that I had chosen for interpreting the input signals and controlling the outputs, specifically the unipi neuron series. Note that I am in no way affiliated with unipi, nor do I intend to do an in-depth description of their hardware. For more details from unipi themselves, have a look at the unipi website.

This post is a part of a larger series of posts on my home automation setup. See the home automation overview post, to learn about the rationale and a description of the other posts!

Inside

The following picture shows what is actually inside such a unit: a raspberry pi 3B+ main controller board and then (depending on the neuron series you have), 1 to 3 I/O boards, connected via SPI – Serial Peripheral Interface.

The SPI interface works by connecting a number of nodes in a series, daisy-chained, on the same SPI-line(s). The raspberry pi functions as the main server node, the I/O boards are the clients. The server nodes periodically selects one of the clients to become active. Consequently, the selected client becomes active and puts its data on the SPI-lines that were made available. Other clients remain inactive for that duration of time they were not selected. Afterwards, the server moves on to the next client. This process continues until all clients have been consulted, at which point the cycle concludes and a new one starts.

The main controller board has all the other functionalities the raspberry pi has, including bluetooth, wifi, wired ethernet, USB ports, and even a HDMI interface¹. The I/O boards either feature digital inputs, relay outputs, analog inputs, analog outputs or a combination of these. Additionally, there are connections like a RS-485 serial lines (e.g. for interfacing with modbus) as well as a connector for a 1-wire bus (e.g. for temperature sensors).

Rationale

Considering the hardware platform, I think the unipi platform provides a number of advantages I couldn’t find with other vendors:

open platform: the main controller board is a raspberry pi 3B+, meaning you can run any software on it you would want.
software support: next to the open platform, they also provide open source OS images as well as software libraries to extend yourself.
local: obviously, it runs from my local network inside of my home.
wired: the I/O interfaces use standard voltages, for instance the power supply to the unit uses 24V, the digital inputs use 24V, the relay outputs can switch 240V², …
form factor: the modules come in form factors of 4, 8 and 12 DIN rail modules and fit nicely on a household DIN rail in the electric cabinet.
wiring connectors: related to the form factor, it is quite easy to connect a large amount of I/O in a relatively small space.
low cost: the units themselves do not come that cheap, but calculated as a cost per I/O, they are quite OK compared to other solutions like industrial PLCs
community: while working on the units and my own custom software, I would often reach out to the unipi community forum. They often reply quite quickly and have been a great help!
connectivity: apart from the I/O, the unipi units come with a lot of extra connectivity options such as wifi, wired ethernet, bluetooth, RS-485, one-wire, …

Specific units

I have number of different unipi units in my setup, since I have a large amount of functionalities to support, for example:

push button read-out
light control (relays)
alarm system (window / door contacts, PIR detector, … but also on the output side an indoor-and outdoor siren).
shades (relays)
…

While you can have both inputs and outputs on the same unipi unit, I did decide to dedicate a bigger unipi unit for the push button control and another one for the lights:

lights, relay control: unipi neuron l403, which has up to 56 relay outputs
push button read-out: the unipi neuron l303. This was an upgraded version of the unipi neuron m303 with up to 64 digital inputs, but at the time of writing, it seems this model is no longer supported.

Unipi axon

As a side note, I also have a unipi axon S605 for DALI light control. The unipi axon series is another series of PLCs from the same brand. It differs mainly in the controller board it features: instead of being based on raspberry pi, it is based on an Allwinner H5 ARM processor. Also, instead of having the main hard drive on a removable flash memory, it has internal eMMC memory.

I did not look into the rest of the axon series as it wasn’t available at the time when I acquired them. Additionally, I also favored the neuron series because I have more familiarity with the raspberry platform in general.

DALI light control is something I haven’t yet fully deployed, but the advantage is that you can have more all-digital control of lights – wherever the light fixtures support it. The main feature I do like about DALI is the ability to dim lights directly from the LED driver that already needs to perform the AC / DC conversion, making it more effective.

Note however that this specific unit also isn’t able anymore, I suspect due to low demand.

Closing thoughts

This was just a short description mainly about why I did choose unipi neuron as the main hardware platform for my home automation.

Note that this sort of hardware is particularly suited for a wired, star topology setup. If I were to name a disadvantage, it would be that working with this kind of hardware requires a lot of tedious manual wiring which also takes up a lot of cabinet space. This is however more of an issue of the overall setup, not the hardware itself. The alternative is a bus-system, where all of the signaling is done on a shared bus. The disadvantage here is either all of the hardware on the bus is more complex and costly (e.g. KNX, DALI) and / or that you are tied to a single hardware vendor. Nonetheless, DALI (and maybe even KNX) is something I would still like to examine a bit more in detail at some point in the future.

Another issue I have faced is with the removable flash drives as these tend to go corrupt after many read / write cycles. Again, this is more of an issue of the raspberry pi and flash drives in general, but it could still pose an issue for the reliability of the overall system. After changing the flash drives to a more durable pSLC SD card, I can confirm that I haven’t faced any issues with corrupt flash drives lately (fingers crossed!) Check the unipi SD card reference for more details.

Overall, I have been very happy with the unipi hardware platform though!

For more commercial details check the unipi main website. On the technical side, check out the unipi kb.

the HDMI interface isn’t readily exposed ↩︎
for highly inductive loads, like motors, it is recommended to switch the loads not directly, but instead via an intermediary set of relays ↩︎

Unipi on mhemeryck

Software

Software

Kernel: polling SPI

Evok: modbus to web APIs

evok2mqtt: websocket to MQTT

Closing thoughts

Update from unipi

Hardware

Inside

Rationale

Specific units

Unipi axon

Closing thoughts

`evok2mqtt`: websocket to MQTT