My Raspberry PI based IOT stack, introduced in an earlier article, is productive and online for more than one year. It’s time to make a full upgrade of the operating system, the Docker containers, and all associated configurations.
This article completely covers my approach, including regrettable mistakes that I disclose to help other preventing them or finding a quick solution. Learn about compatibility problems of Alpine Linux with Raspberry Pi OS based on Debian Buster, and feature changes in the IOT Stack itself.
The technical context of this article is to start an upgrade from Home Assistant 2022.11 and ESPHome 2022.11.3, but it should work with older and newer versions as well.
Recommended Update Procedure
The official IOT Stack documentation explains why updates can be difficult and things can break: The Docker container use ephemeral storage for runtime, and they are versioned projects themselves, with specific configuration options passed as environment variables, config files, or flags to the binary. This versioned configuration complexity is very well handled in each commit version of the master branch. But with each passing commit, config changes accumulate, and it some point in time, changes will not be fully backward compatible. In addition, newer Docker containers might require newer libraries of the base operating system. And the final catch is that a new Raspberry Pi OS based on Debian Bullseye was released in November 2021.
This is a complex situation. The documentation therefore recommends:
- Make a complete disc image of your Raspberry Pi as a backup
- Upgrade the Raspberry Pi OS libraries
- Upgrade only the Docker container
If you want to push even further, you can also:
- Upgrade to the next Raspberry Pi release
- Update the IoT Stack project files
Knowing this, let’s get started.
Context
My IOT stack installation was initially completed in June 2021, on these components:
The Raspberry Pi version is based on the older Debian Buster release.
Linux nexus 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux
RETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs
The IoT Stack version that I used since the inception is a commit in 2021-06 on the master branch. Essentially, I have a docker-compose file that references versioned Docker images and list the options required to run them (env vars, ports, container names for inter-container communication). In this file, only a few images were versions pinned, several just used the latest tag. My assumption was that this means every time a container is restarted it will also pull the latest image. But this was wrong! Running a docker image ls, it showed that I still used images that were over 15 months old!
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/home-assistant/raspberrypi4-homeassistant stable 6573960f8063 14 months ago 1.05GB
esphome/esphome 2021.9.1 76da910d1982 14 months ago 942MB
iotstack_mosquitto latest df0853b69834 15 months ago 11.7MB
influxdb 1.8 69a9438650d8 15 months ago 246MB
telegraf latest a721ac170fad 15 months ago 273MB
iotstack_nodered latest 74a0b2f9d520 15 months ago 428MB
grafana/grafana latest 7914b5231e0b 15 months ago 178MB
portainer/portainer-ce latest bd3c978cdaec 15 months ago 157MB
pihole/pihole 2021.12.1 d463557f7f81 11 months ago 236MB
Now let’s see how to upgrade the stack systematically.
Update Raspberry Pi OS Libraries
The first step is to perform al full upgrade of the Raspberry Pi OS itself. Two commands are sufficient:
sudo apt-get update
sudo apt-get full-upgrade
The following packages will be upgraded:
base-files bind9-host bluez containerd.io distro-info-data dnsutils docker-ce docker-ce-cli docker-ce-rootless-extras
ffmpeg folder2ram git git-man isc-dhcp-client isc-dhcp-common libarchive13 libavcodec58 libavdevice58 libavfilter7
...
This update went smoothly. After a reboot, all things worked as before.
Update Docker Images
ESPHome
ESP Home is an essential part of my IOT stack: A highly usable, sophisticated framework to quickly add esp8266/esp32 boards and hundreds of sensors into a managed fleet complete with over-the-air update functionality. I'm still utterly fascinated by this project.
To update it, I researched the official Docker hub for tags, and then slowly upgraded the version information in the docker-compose.yml file:
2021.9.1
2021.12.3
2022.3.2
2022.6.3
2022.11.3
Every time I changed the version information in the docker-compose file, I also used the following command to enforce the creation of a new container:
docker-compose up --force-recreate --build -d esphome
In between, I always checked that the UI was working. With the final update complete, let’s try updating the software that runs on a sensor:
INFO Reading configuration /config/esp8266-01.yaml...
INFO Generating C++ source...
INFO Core config or version changed, cleaning build files...
INFO Compiling app...
Processing esp8266-01 (board: d1_mini; framework: arduino; platform: platformio/espressif8266 @ 3.2.0)
--------------------------------------------------------------------------------
Platform Manager: Installing platformio/espressif8266 @ 3.2.0
INFO Installing platformio/espressif8266 @ 3.2.0
...
INFO framework-arduinoespressif8266@3.30002.0 has been installed!
Then, connect to the sensor and see its streaming log files:
[19:31:17][C][mqtt_subscribe.sensor:031]: MQTT Subscribe 'Configure Sleep Time'
[19:31:17][C][mqtt_subscribe.sensor:031]: State Class: ''
[19:31:17][C][mqtt_subscribe.sensor:031]: Unit of Measurement: 'm'
[19:31:17][C][mqtt_subscribe.sensor:031]: Accuracy Decimals: 0
[19:31:17][C][mqtt_subscribe.sensor:032]: Topic: esp/sleep_time
[19:31:17][C][mqtt.sensor:027]: MQTT Sensor 'esp8266-01.voltage':
[19:31:17][C][mqtt.sensor:031]: State Topic: 'nodes/esp8266-01/sensor/esp8266-01voltage/state'
[19:31:17][C][mqtt.sensor:027]: MQTT Sensor 'WiFi Signal esp8266-01':
[19:31:17][C][mqtt.sensor:031]: State Topic: 'nodes/esp8266-01/sensor/wifi_signal_esp8266-01/state'
[19:31:17][C][mqtt.binary_sensor:022]: MQTT Binary Sensor 'ESP01 Motion':
[19:31:17][C][mqtt.binary_sensor:023]: State Topic: 'nodes/esp8266-01/binary_sensor/esp01_motion/state'
[19:31:42][D][sensor:126]: 'VCC': Sending state 2.96582 V with 2 decimals of accuracy
[19:31:56][D][sensor:126]: 'WiFi Signal esp8266-01': Sending state -70.00000 dBm with 0 decimals of accuracy
That’s good. The first update process worked.
PiHole
The next image to be updated is PiHole, a handy utility for removing DNS queries to undesired external services. I used the very same strategy as before: upgrade tag by tag. The intended upgrade order was as follows:
2022.02.1
2022.04.3
2022.05
But very soon into the upgrade process, I noticed that the dashboard did not work anymore:
So, I went back the versions, and was eventually stuck with 2022.02.1.
The hunt for the root error was surprising: Two different threads came to the same conclusion.
First, I searched the Internet about PiHole, Docker and Raspberry Pi and stumbled over this thread that explained that the Debian version of the Raspberry Pi is missing an more up-to-date version of the library libseccomp.
Second, I looked into logs from another Docker container that I wanted to update, and saw these messages:
s6-linux-init-hpr: fatal: unable to reboot(): Operation not permitted
s6-svscan: warning: unable to iopause: Operation not permitted
s6-svscan: warning: executing into .s6-svscan/crash
s6-svscan crashed. Killing everything and exiting.
Searching the internet again for these error messages, I stumbled upon a github issue in which one solution was also to upgrade to a newer version of libseccomp.
And then those two threads came together in this blog post about libseccomp and Alpine3.13, the very Docker images that several IoT stack images are built upon. Alpine based images after version 3.13 cannot run on a Raspberry Pi with Buster Debian, because the libseccomp version is just too old.
In a nutshell, the solution is to get an up-to-date libseccomp from the next upstream Raspberry Pi OS release (which is now based on bullsye). The concrete commands to get this done are these:
echo 'deb http://deb.debian.org/debian buster-backports main contrib non-free' | sudo tee -a /etc/apt/sources.list.d/debian-backports.list
sudo apt update
sudo apt install libseccomp2 -t buster-backports
After this, I could update the PiHole container to the latest version successfully, and the logfiles indicate a successful start:
Pi-hole version is v5.14.2 (Latest: v5.14.2)
AdminLTE version is v5.17 (Latest: v5.17)
FTL version is v5.19.2 (Latest: v5.19.2)
Container tag is: 2022.11.2
Update all other Docker Containers
The libseccomp problem was the root cause that prevented all the other updates as well!
Once solved, I could upgrade all remaining images to the following versions:
✅ esphome/esphome:2022.11.3
✅ homeassistant/home-assistant:2022.11
✅ pihole/pihole:2022.11.2
✅ mosquitto:2022.12 (IOT Stack Template)
✅ influxdb:1.8.10
✅ telegraf:1.24.4
✅ nodered:2022.12 (IOT Stack Template)
✅ portainer:linux-arm-2.16.2
✅ grafana/grafana:9.3.1
The final proof is the Home Assistant dashboard, showing the sensor data just as before:
Version Updates & Migrations
Although all applications could be updated successfully, some changes needed to be applied nevertheless. Here is a short summary:
ESPHome: FastLED Driver is not supported anymore
During updating the ESPHome code on the sensors, I saw this error:
Failed config
light.fastled_clockless: [source /config/esp8266-dht.yaml:60]
This feature requires framework version 2.7.4 or lower. Please see note on documentation for FastLED.
The fastled module is simply not supported anymore. Changing the config to this one worked:
esp8266:
framework:
version: 2.7.4
board: d1_mini
esphome:
name: esp8266-dht
Home Assistent: Config File Update
The Home Assistant update included a heavy database migration as seen in the first log lines in the newly started container:
2022-12-07 11:16:50.466 WARNING (Recorder) [homeassistant.components.recorder.migration] Database is about to upgrade from schema version: 22 to: 30
2022-12-07 11:16:50.495 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding columns name to table statistics_meta. Note: this can take several minutes on large databases and slow computers. Please be patient!
2022-12-07 11:16:50.825 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding index `ix_statistics_statistic_id_start` to database. Note: this can take several minutes on large databases and slow computers. Please be patient!
2022-12-07 11:16:50.893 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding index `ix_statistics_short_term_statistic_id_start` to database. Note: this can take several minutes on large databases and slow computers. Please be patient!
2022-12-07 11:16:51.554 WARNING (Recorder) [homeassistant.components.recorder.migration] Adding columns attributes_id to table states. Note: this can take several minutes on large databases and slow computers. Please be patient!
Additional container logs showed several errors related to new features of the most recent versions. Particularly following:
2022-12-07 11:16:52.554 WARNING (MainThread) [homeassistant.components.mqtt] The 'broker' option near /config/configuration.yaml:34 is deprecated, please remove it from your configuration
2022-12-07 11:16:53.001 WARNING (MainThread) [homeassistant.components.mqtt] The 'password' option near /config/configuration.yaml:34 is deprecated, please remove it from your configuration
2022-12-07 11:16:53.401 WARNING (MainThread) [homeassistant.components.mqtt] The 'username' option near /config/configuration.yaml:34 is deprecated, please remove it from your configuration
2022-12-07 11:16:53.998 WARNING (MainThread) [homeassistant.components.mqtt.mixins] Manually configured MQTT sensor(s) found under platform key 'sensor', please move to the mqtt integration key, see <https://www.home-assistant.io/integrations/sensor.mqtt/#new_format>
2022-12-07 11:16:54.456 ERROR (MainThread) [homeassistant.components.camera] The mjpeg platform for the camera integration does not support platform setup. Please remove it from your config.
I updated the config file as suggested by these logfiles. And for the camera, I switched to the new MJPG IP camera config.
Conclusion
IOT stack is an amazing project that bundles essential applications for the setup and deployment of sensors, combined with data extraction, visualization, and dashboards to build automation. This blog post showed how to update a 15 month old installation to the newest version without breaking things. The essential steps are these: a) Update all Raspberry Pi packages to their latest version, b) Install the most recent libseccomp library from the next upstream Raspberry Pi OS, c) force recreate the Docker images, climbing from version to version, d) follow recommended config updates for the Docker containers. These steps resulted in an up-to-date versioned stack, and only some minor configuration changes in the config files were required.