Homelab on mhemeryck

unifi terraform

Thu, 16 Dec 2021 00:00:00 +0000

Last week, I did write about resetting my unifi cloud key.

The main reason I did gain a renewed interest in the cloud key was because I wanted to add some more fixed IPs to my home network. One of the nice things about the unifi controller software is that it provides a single dashboard interface to manage everything network-related.

On the downside, the interface is overall quite limited. While it does offer DHCP (the fixed IPs) in this fashion, it does not offer built-in DNS (e.g. for naming dedicated hosts on the home network).

Additionally, I also don’t really like these point-and-click interfaces which are hard to put under version control except for a full snapshot. Coming from a software development background and with the current infrastructure-as-code movement, I wanted to look into an alternative.

infrastructure-as-code and HashiCorp terraform

Nowadays, if you want to run some kind of software service, all the major cloud providers do provide web APIs to manage the (virtual) infrastructure for you. Managing infrastructure at this point becomes another software problem for interacting with these APIs, thus the term infrastructure-as-code.

One platform that has become increasingly popular in that regard has been HashiCorp’s terraform. It provides an abstraction layer for all of these different cloud provider and offers an ecosystem for easily adding more modules.

Additionally, the format is declarative, meaning that the tool will manage the life cycle of the components you describe. As an example, adding a resource means just declaring it in the configuration file and running it through the terraform CLI tool. At that point, the resource becomes managed by terraform. Destroying the resource is done by removing the entry from the configuration file and once again applying. Essentially, the configuration file will always reflect the state of the resources being managed and vice versa.

unifi-terraform

What does any of that have to do with my tiny homelab setup? Enter unifi-terraform, which is a community provided terraform module that interacts with the API of the unifi cloud key. This way, I could manage all the required fixed IPs.

As I mentioned in my previous post, I did have some trouble getting started though, so let’s walk a bit through the steps I had taken.

terraform configuration file

The first step is creating a main.tf file containing the actual configuration. I’m not going to elaborate too much on this here, see the terraform docs for that.

Generally, my configuration file looked like this (redacted).

terraform {
required_providers {
unifi = {
source = "paultyng/unifi"
version = "0.34.0"
}
}
}
provider "unifi" {
username = var.username # These are in a variables file
password = var.password
api_url = var.api_url
}
resource "unifi_site" "sitename" {
description = "Default"
}
resource "unifi_user_group" "default" {
name = "Default"
}
# the network / DHCP config
resource "unifi_network" "lan" {
name = "LAN"
purpose = "corporate"
dhcp_dns = [
"192.168.1.21",
"1.1.1.1",
"8.8.8.8",
"9.9.9.9",
]
dhcp_enabled = true
dhcp_start = "192.168.1.20"
dhcp_stop = "192.168.1.254"
domain_name = "lan"
igmp_snooping = true
subnet = "192.168.1.0/24"
}
# WLAN
resource "unifi_wlan" "wlan" {
name = "wlan"
security = "wpapsk"
passphrase = var.passphrase_wifi_wlan
network_id = unifi_network.lan.id
user_group_id = unifi_user_group.default.id
wlan_band = "5g"
multicast_enhance = true # chromecast
# default group ID
ap_group_ids = [
var.ap_group_id
]
# enable WPA2/WPA3 support
wpa3_support = true
wpa3_transition = true
pmf_mode = "optional"
}
...
# Devices
resource "unifi_device" "my-router" {
name = "my-router"
}
...
# Users: network clients
resource "unifi_user" "my_client" {
name = "my-client"
mac = "d9:6e:3d:12:8f:53"
fixed_ip = "192.168.1.10"
network_id = unifi_network.lan.id
}
...
resource "unifi_user" "floating_client" {
name = "floating"
mac = "bc:e4:26:db:df:e4"
}

At the top of this file the required_provider is mentioned, i.e. the module that contains the client code to talk to the unifi cloud key API. The provider block contains the data used for the provider to do the actual connection (essentially username / password and a URL to connect to).

There’s a couple of interesting resources in there:

unifi_site: just the physical location – I guess this makes more sense if you have multiple locations to manage
unifi_user_group: this is relevant if you want to separate out specific access for e.g. WiFi clients.
unifi_network: main (wired) network parameters, including e.g. the DHCP settings
unifi_wlan: wireless network configuration. Helpful if you want push in passwords
unifi_device: the unifi hardware such as switches, routers and wireless access points. I actually don’t have anything in particular to manage here, but I guess if you want to have specific port profiles for a switch, this would be the way to link to it
unifi_user: the clients on the network. It’s at this point that I can actually fix the IPs.

Have a look at unifi-terraform for more resources that can be managed.

terraform import

At this point, you could in theory start creating all of the resources by applying it (with terraform apply). In reality, these resources are likely already recognized and managed by the cloud key itself. The way around that is by importing the resources via terraform import {resource_name} {resource_id}

The {resource_name} is just the name as you did provide it in the main configuration file, e.g. unifi_user.my_client The {resource_id} is an ID that’s used in the API of the cloud key. This was for me the point for which it all broke down since I couldn’t find any good documentation of this API.

cloud key API

Fortunately, I did find some pointer into the cloud key API in a blog post from the original module author. Another real help to me has been this unifi PHP client code which also lists most of the endpoints.

I did distill all my findings into this cloud key postman collection.

Let’s walk through some of the API calls, built using httpie.

login: `POST {host}/api/login`

printf '{
 "username": "foo",
 "password": "bar"
}'| http --follow --timeout 3600 POST 'https://cloudkey.lan:8443/api/login' \
 Content-Type:'application/json' \
 Cookie:'csrf_token=abc; unifises=dfcwZvHcpBwc5LTlOzgxYSlsutpC6F2b'
HTTP/1.1 200
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Connection: keep-alive
Content-Length: 30
Content-Type: application/json;charset=UTF-8
Date: Thu, 16 Dec 2021 20:55:24 GMT
Keep-Alive: timeout=60
Set-Cookie: unifises=19uFvBLta4optBuzwV4QyVip5TbdR0t6; Path=/; Secure; HttpOnly
Set-Cookie: csrf_token=Z1dK47sp8NsjfpdbgHNyVNgYewNYIodd; Path=/; Secure
X-Frame-Options: DENY
vary: Origin

{
 "data": [],
 "meta": {
 "rc": "ok"
 }
}

This call will perform the login on the login endpoint and set a cookie for subsequent calls. Note that all other calls will need this cookie and that it will expire.

self: `GET {host}/api/s/{site}/self`

http --follow --timeout 3600 GET 'https://cloudkey.lan:8443/api/s/default/self' \
 Cookie:'csrf_token=cNHfNcA1zQoMW9wzFvCmGwEEo22irJ6D; unifises=dfcwZvHcpBwc5LTlOzgxYSlsutpC6F2b'
HTTP/1.1 200
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: Access-Control-Allow-Origin,Access-Control-Allow-Credentials
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json;charset=UTF-8
Date: Thu, 16 Dec 2021 20:58:28 GMT
Keep-Alive: timeout=60
Transfer-Encoding: chunked
X-Frame-Options: DENY
vary: accept-encoding,origin,accept-encoding

{
 "data": [
 {
 ...
 "site_id": "{site_id}",
 "site_name": "default",
 ...
 }
 ],
 "meta": {
 "rc": "ok"
 }
}

This just lists some generic information about the site (location), e.g. the site_id to be imported.

devices: `GET {host}/api/s/{site}/stat/device`

http --follow --timeout 3600 GET 'https://cloudkey.lan:8443/api/s/default/stat/device' \
 Cookie:'csrf_token=cNHfNcA1zQoMW9wzFvCmGwEEo22irJ6D; unifises=dfcwZvHcpBwc5LTlOzgxYSlsutpC6F2b'
...
{
 "meta": {
 "rc": "ok"
 },
 "data": [
 {
 "_id": "5ea4395234309a00041ffa6e",
 "adopted": true,
 "anon_id": "d5416fde-c2ad-4558-9aa0-4b4024c441de",
 "antenna_table": [
 {
 "default": true,
 "id": 4,
 "name": "Combined",
 "wifi0_gain": 3,
 "wifi1_gain": 3
 }
 ],
 ...
 }
 ]
}

This endpoint is interesting to get the IDs of the devices to import.

users: `GET {host}/api/s/{site}/rest/user`

http --follow --timeout 3600 GET 'https://cloudkey.lan:8443/api/s/default/rest/user' \
 Cookie:'csrf_token=cNHfNcA1zQoMW9wzFvCmGwEEo22irJ6D; unifises=dfcwZvHcpBwc5LTlOzgxYSlsutpC6F2b'
...

{
 "meta": {
 "rc": "ok"
 },
 "data": [
 {
 "_id": "9dfeee213c227000f47aade2",
 "mac": "48:98:e3:f6:21:a5",
 "site_id": "5dffdda64c2370010eb91fa0",
 ...
 },
 ]
}

This was the most important endpoint for my purposes: it lists the resource ID under _id and makes it possible to import it.

Practically, this means

terraform import unifi_user.my_client 9dfeee213c227000f47aade2

At this point, the client should be managed by terraform.

Refer to the cloud key postman collection for more resources. Note that I only really looked into the read-only endpoints (the GET operations) since I would have terraform do the actual operations (POST, PATCH).

Closing thoughts

I was really pleased to see that after some crawling of the cloud key API I was able to get the unifi resources managed with terraform. At this point, I can manage my unifi configuration under source control, most notably the DHCP fixed IPs and some firewall rules. This only scratches the surface of what is now possible at this point though.

Since terraform also has modules for various DNS providers such as powerdns, coredns and even pihole I could start thinking of managing the fixed IPs and the host names from a single spot. The same local network DNS could then be used by my local kubernetes to register service names into.

Since it’s supposedly infrastructure-as-code, I could also start thinking of integrating everything in CI/CD flow, where the terraform code is first validated and then automatically applied. This in turn would mean I’d need to find a way to either self-host a source control / CI/CD platform (thinking of gitlab) or find some way to safely tunnel out to a cloud-based platform with (thinking github actions + wireguard).

Another change I’d like to make, is to use a remote state back-end instead of the local file storage.

… but that’s for another time!

cloud key reset

Wed, 08 Dec 2021 00:00:00 +0000

After my last password issues post, I still had some places for which I didn’t have an easy approach to reset the login. For my local network, I still use a unifi cloud key gen 1, to keep an overview of the network.

Cloud key

The ubiquiti unifi cloud key is a small client device on the local network, specifically running the ubiquiti unifi controller software. The controller software is there to e.g. provision new (ubiquiti) hardware, to configure the network (switch ports, VLANs, firewalls, …) but also to allow remote management from a web interface or phone app.

Note that you actually don’t need dedicated hardware to run the controller software. You can just as well run the software from your own local computer or instead use the dockerized controller software. Having it on a dedicated device though means that it is always available, doing logging and monitoring in the background and providing an entry point for remote access. Additionally, the cloud key does indeed mainly run the controller software, but it also has onboard firmware to set manage the controller software for you.

On the upside, I think the unifi experience is overall quite nice and graphically slick. Despite being considered a prosumer kind of brand, some features I would consider as basic are missing though (built-in DHCP / DNS, VPN termination, …)

In the end, I still like the unifi products since they do have somewhat more enhanced features yet mostly “just work”. There was a time when I was running my own openWrt-enabled router, but the reality just is that after the initial setup, this is just something I barely feel the need to touch (and maintain).

Factory reset

As mentioned above, the cloud key hardware runs both the main unifi controller software as well as its own firmware to manage said software. The login for the controller software is done via my ui.com account. Retrieving that one was as simple as resetting this account.

The controller firmware management however is also password-protected and being a good citizen, I obviously changed it from its default of ubnt / ubnt. Even though I still had access to the controller software, I figured it’d be better to have full access to the firmware management as well. The rationale is that the times when you often need this is in case of some breaking issue – in which event you want to have immediate access.

The cloud key has an SD-card, so I figured that it would run from there (pretty much how a raspberry pi does). After checking the SD card contents though, they only seemed to contain the regularly scheduled backups.

Checking the cloud key gen1 pdf docs showed me that there is simple factory reset button. After grabbing a backup snapshot of the controller software, I did decide on just triggering the factory reset.

Sure enough, the firmware management did now work with the default credentials (prompting me to change it straight after that).

Let’s Encrypt certificates setup

I want to be able to also access the cloud key on a regular URL even if it’s only from my local network. Also, by default the cloud key controller software will use an HTTPS version (which is good), but it obviously has no built-in way to provide a valid certificate (not so good).

A side-effect of this factory reset was that I had now lost this setup in where the cloud key itself regularly pulls in an updates Let’s Encrypt SSL certificate. In the past, I had followed this nicely detailed blog post on cloud key SSL certificates with Let’s Encrypt.

To summarize the steps for myself for future reference:

Fixed IP for the cloud key: this actually easy to do from the unifi controller software (running on the cloud key)
DNS entry for this fixed IP: I use pihole’s built-in dnsmasq for this. This means that all devices on my local network will be able to resolve the host for the cloud key
Set up a job on the cloud key to regularly pull in updated SSL certificates

`acme.sh` Let’s Encrypt cloud key bot setup

The way Let’s Encrypt works is that you can request a certificate for your own domain name. Afterwards, the Let’s Encrypt server will verify with you whether or not you actually own the domain for which you did request the SSL certificate. This can be done in a multitude of ways, but the most common ones are:

HTTP-based: on the service you expose, you host some kind of secret that was given to you before by Let’s Encrypt
DNS-based: on the DNS provider on which you host the domain, you provide an txt kind of record which Let’s Encrypt, again containing the secret Let’s Encrypt had provided before

For my home network case, I am obliged to use the DNS-based approach, since my own service isn’t exposed on the wide internet for Let’s Encrypt to call back to. It’s a nice approach to be able to get valid SSL certificates for a machine on a local network. Additionally, a lot of tooling is readily available to automate all details for you, such as certbot or cert-manager.

Essentially, these steps are completely taken from the aforementioned post. Since I wouldn’t like to lose these and since I have some minor differences due to using digital ocean as DNS provider, I detail them here again for myself.

Installing the bot:

curl https://get.acme.sh | sh

Installing a hook script to run on certificate renewal in /root/.acme.sh/cloudkey-renew-hook.sh

#!/bin/bash
# Renew-hook for ACME / Let's encrypt
echo "** Configuring new Let's Encrypt certs"
cd /etc/ssl/private
rm -f /etc/ssl/private/cert.tar /etc/ssl/private/unifi.keystore.jks /etc/ssl/private/ssl-cert-snakeoil.key /etc/ssl/private/fullchain.pem

openssl pkcs12 -export -in /etc/ssl/private/cloudkey.crt -inkey /etc/ssl/private/cloudkey.key -out /etc/ssl/private/cloudkey.p12 -name unifi -password pass:aircontrolenterprise

keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /usr/lib/unifi/data/keystore -srckeystore /etc/ssl/private/cloudkey.p12 -srcstoretype PKCS12 -srcstorepass aircontrolenterprise -alias unifi

rm -f /etc/ssl/private/cloudkey.p12
tar -cvf cert.tar *
chown root:ssl-cert /etc/ssl/private/*
chmod 640 /etc/ssl/private/*

echo "** Testing Nginx and restarting"
/usr/sbin/nginx -t
/etc/init.d/nginx restart ; /etc/init.d/unifi restart

This will modify the retrieved certificates in such a way that they can be used by the unifi controller software.

Export the proper API keys to the environment. In my case, I am using the digitalocean DNS provider. This changes the line in the .bashrc file to adapt to something like

export DO_API_KEY=<key>

The first certificate retrieval line changes to

acme.sh --force --issue --dns dns_dgon -d <YOUR_DOMAIN> --pre-hook "touch /etc/ssl/private/cert.tar; tar -zcvf /root/.acme.sh/CloudKeySSL_`date +%Y-%m-%d_%H.%M.%S`.tgz /etc/ssl/private/*" --fullchainpath /etc/ssl/private/cloudkey.crt --keypath /etc/ssl/private/cloudkey.key --reloadcmd "sh /root/.acme.sh/cloudkey-renew-hook.sh"

The crontab entry becomes:

0 0 * * * /root/.acme.sh/acme.sh --renew --apache --renew-hook /root/.acme.sh/cloudkey-renew-hook.sh -d <YOUR_DOMAIN>

At this point, everything should be back up and running and the cloud key should be accessible from the custom domain with a valid certificate!

Closing thoughts

This post serves mainly as a note-to-self after needing to go through some steps to manually recover my cloud key setup.

At this point, the cloud key is just back running as before. Going through this process again made me realize some points of improvement.

Firstly, in the past I did run the controller software as part of my home k3s cluster, but I ended up not doing that since it cost me a lot of resources and I basically did not want to manage that. However, I still have some plans to upgrade the cluster setup itself at which point it might come interesting again to run it there. The added value would be that I could manage the certificate renewal using cert-manager instead of via a cron job. The downside of the cron job on the cloud key is that it gets wiped for firmware upgrades.

Secondly, I am also not completely happy with the manual host name management. I first need to configure a static IP manually in the controller software and then also manually configure pihole / dnsmasq. It seems there is a unifi terraform provider that would be able to handle the IP management – but I haven’t been able yet to get that working with the cloud key (perhaps it does not work for the gen 1?). DNS configuration can be done via API using e.g. powerdns or coredns, which is another route to explore. Services running on the kubernetes cluster could then also register their host names using external-dns

Thirdly, it would also be nice that any of the host names I set for my local network are actually accessible in a similar manner over a wireguard VPN.

A final small change would be for me to change DNS provider from digital ocean to hetzner cloud since I have everything else there.

Still a lot of home lab ideas, so little time, let’s see what’s next!

Homelab on mhemeryck

unifi terraform

infrastructure-as-code and HashiCorp terraform

unifi-terraform

terraform configuration file

terraform import

cloud key API

login: POST {host}/api/login

self: GET {host}/api/s/{site}/self

devices: GET {host}/api/s/{site}/stat/device

users: GET {host}/api/s/{site}/rest/user

Closing thoughts

cloud key reset

Cloud key

Factory reset

Let’s Encrypt certificates setup

acme.sh Let’s Encrypt cloud key bot setup

Closing thoughts

login: `POST {host}/api/login`

self: `GET {host}/api/s/{site}/self`

devices: `GET {host}/api/s/{site}/stat/device`

users: `GET {host}/api/s/{site}/rest/user`

`acme.sh` Let’s Encrypt cloud key bot setup