Skip to content

OpenMQTT Gateway: Messages and Commands

By Sebastian Günther

—

—

Posted in Iot, Openmqtt

With the introduction of OpenMQTT to my IOT@Home stack, the availability of long-range sensors increased dramatically. Over the course of the last articles I explored RF433, Bluetooth BLE, LoRa and even IR signals. With these gateways, messages from surrounding devices are captured, for example an outdoor temperature sensor, your smart watch, or a long-range GPS sensor. Reading messages is only one feature of OpenMQTT. Sensing messages to the devices is the opposite direction.

This article investigates how MQTT messages are send from a gateway to the receiving device, and to what amount both the gateway and its detected devices can be configured or controlled. Its separated into the following four sections: Gateway Status & Configuration, RF Gateway, and BLE Gateway

The technical context of this article is OpenMQTT Gateway v1.7.0, but the instructions and examples should work with newer versions as well.

OpenMQTT Gateway Messages

MQTT Topic Structure

Each OpenMQTT Gateway, independent of its types, exposes generic status information. Essentially, these are topics with the following naming structure and content.

  • nodename/version: The installed OpenMQTT version
  • deviceName/LWT: An acronym for "last will message", a retained topic that expresses the last time the devices status changed from online to offline
  • deviceName/SYStoMQT: The complete configuration of this particular gateway, containing both generic information as well as transport protocol specific instructions.
  • deviceName/WebUitoMQTT: Exposes configuration option of the WebUI for this device.
  • deviceName/${SENSOR}: OpenMQTT allows to connect additional sensors and actuators, such as an DHT11 temperature sensor, an SSD1306 OLED display, or an HCSR501 PIR sensor. These connected sensor expose configuration and measurements over a topic
  • devicName/${TRANSPORT_PROTOCOL}toMQTT: This topic lists all detected consumer products and sensor, either with a name or ID, and it contains the sensors payload. The placeholder is a short string for concrete transport protocol, for example be RTL_433toMQTT for an RF or BTtoMQTT for Bluetooth BLE gateway.

Let’s take a concrete look at concrete examples for these topics.

Topic "SYStoMQT"

The SYStoMQTT topic contains the following generic information:

{
  "uptime": 60405,
  "version": "v1.6.0",
  "discovery": false,
  "ohdiscovery": false,
  "env": "lilygo-rtl_433",
  "freemem": 115572,
  "mqttport": "1883",
  "mqttsecure": true,
  "minfreemem": 76492,
  "tempc": 57.77778,
  "freestack": 6064,
  "rssi": -39,
  "SSID": "SECRET",
  "BSSID": "SECRET",
  "ip": "SECRET",
  "mac": "SECRET",
  "modules": [
    "LilyGo_SSD1306",
    "WebUI",
    "rtl_433"
  ]
}

The modules section contains the transport protocol, the WebUI, and all additionally connected sensors. For example, for an RF Gateway based on the LilyGo Lora T3 v1.61:

{
  "modules": [
    "LilyGo_SSD1306",
    "WebUI",
    "rtl_433"
  ]
}

When the Gateway is of type RF, the additional topics are:

{
  "actRec": 3,
  "mhz": 433.92,
  "RTLRssiThresh": -87,
  "RTLRssi": -101,
  "RTLAVGRssi": -96,
  "RTLCnt": 19901,
  "RTLOOKThresh": 15
}

And for a Bluetooth Gateway:

{
  "lowpowermode": -1,
  "interval": 100,
  "intervalcnct": 3600000,
  "scnct": 20501
}

Topic WebUItoMQTT

A typical message in this topic contains the following information:

{
  "displayMetric": true,
  "webUISecure": true,
  "displayQueue": 1
}

The values meaning whether additional metrics should be shown and if the access is protect with a HTTP basic-auth challenge.

Topic BTtoMQTT

This topic provides a detailed payload with all Bluetooth specific configuration options, including its scan interval and duration, whether to detect only sensors, and other details.

{
  "bleconnect": true,
  "interval": 100,
  "adaptivescan": true,
  "intervalacts": 100,
  "intervalcnct": 3600000,
  "scanduration": 1000,
  "onlysensors": false,
  "randommacs": false,
  "hasspresence": false,
  "presenceTopic": "presence/",
  "presenceUseBeaconUuid": false,
  "minrssi": -100,
  "extDecoderEnable": false,
  "extDecoderTopic": "undecoded",
  "filterConnectable": false,
  "pubadvdata": false,
  "pubBeaconUuidForTopic": false,
  "ignoreWBlist": false,
  "presenceawaytimer": 120000,
  "movingtimer": 60000,
  "btqblck": 0,
  "btqsum": 1020,
  "btqsnd": 991,
  "btqavg": 1.029263,
  "bletaskstack": 2132,
  "blecoretaskstack": 3016,
  "blestarts": 1
}

MQTT Messages from Detected RF Devices

With an understanding of how the gateway itself exposes information, let’s take a look at the messages of detected devices.

In all RF device topics, the following information is contained:

  • model: Plaintext name
  • id: When operating several devices of the same type, a random (?) id is assigned to them.
  • protocol: Plaintext information about the device and the technical context how it transmits data
  • channel When operating several devices of the same type, the consumer products typically have a mechanism to select a unique channel or even find a channel on which noise is lesser
  • rssi: The signal strength
  • duration: The sending duration of the received signal

The other payload is devices specific. Let’s take a look at two examples.

First, the payload for an outside weather sensor:

{
  "model": "Ambientweather-F007TH",
  "id": 104,
  "channel": 1,
  "battery_ok": 0,
  "temperature_C": 12.6667,
  "humidity": 8,
  "mic": "CRC",
  "protocol": "Ambient Weather F007TH, TFA 30.3208.02, SwitchDocLabs F016TH temperature sensor",
  "rssi": -84,
  "duration": 56996
}

Second, a freezer temperature sensor:

{
  "model": "Acurite-986",
  "id": 1032,
  "channel": "1R",
  "battery_ok": 1,
  "temperature_C": -17.7778,
  "status": 24,
  "mic": "CRC",
  "protocol": "Acurite 986 Refrigerator / Freezer Thermometer",
  "rssi": -84,
  "duration": 592996
}

Third, a rain gauge sensor:

{
  "model": "Acurite-Rain",
  "id": 64,
  "rain_mm": 834,
  "protocol": "Acurite 896 Rain Gauge",
  "rssi": -85,
  "duration": 1080000
}

In these examples, several device-specific fields occur, such as battery_ok, temperature_C and more. Which type of information is exposed can be determined by looking into the source code of the Github project that implements these sensor readings. In the rtl433 library, we can find the acurite.c source file. Here is an excerpt from this file that details how Acurite freezer temperature sensor protocol works and which data is detected:

// Source: https://github.com/merbanan/rtl_433/blob/master/src/devices/acurite.c
/**
Acurite 00986 Refrigerator / Freezer Thermometer.

Includes two sensors and a display, labeled 1 and 2,
by default 1 - Refrigerator, 2 - Freezer.

PPM, 5 bytes, sent twice, no gap between repeaters
start/sync pulses two short, with short gaps, followed by
4 long pulse/gaps.


Data Format - 5 bytes, sent LSB first, reversed:

    TT II II SS CC
- T - Temperature in Fahrenheit, integer, MSB = sign.
      Encoding is "Sign and magnitude"
- I - 16 bit sensor ID
      changes at each power up
- S - status/sensor type
      0x01 = Sensor 2
      0x02 = low battery
- C = CRC (CRC-8 poly 0x07, little-endian)

2018-04 A user with a dedicated receiver indicated the
  possibility that the transmitter actually drops the
  last bit instead of the demod.

leaving some of the debugging code until the missing
bit issue gets resolved.
*/
static int acurite_986_decode(r_device *decoder, bitbuffer_t *bitbuffer)
{
    int const browlen = 5;
    uint8_t *bb, sensor_num, status, crc, crcc;
    uint8_t br[
    8
  ];
    int8_t tempf; // Raw Temp is 8 bit signed Fahrenheit
    uint16_t sensor_id, valid_cnt = 0;
    char sensor_type;
    char const *channel_str;
    int battery_low;
    data_t *data;
  // ...
}

MQTT Messages from Detected Bluetooth Devices

We find a similar distinction into generic information and device-specific payload data.

The general information is, at least during testing the devices in my vicinity, very limmitted:

  • id: The Bluetooth sender ID, a 6-digit hexadecimal number
  • name: The device name
  • rssi: The measures signal strength of the device

All other data appears to be device specific - see the following two examples.

First, an iBeacon sensor reading:

{
  "id": "50:C1:F0:48:56:F0",
  "rssi": -98,
  "brand": "GENERIC",
  "name": "iBeacon",
  "model_id": "IBEACON",
  "type": "BCON",
  "mfid": "4c00",
  "uuid": "000250ed3ef1cbf8dc4895b252a21005",
  "major": 31516,
  "minor": 21415,
  "txpower": -84
}

Second, an an iPencil:

{
  "id": "67:1C:0F:35:03:33",
  "name": "Apple Pencil",
  "rssi": -98
}

The official documentations points to this list of compatible Bluetooth devices. However, above devices are not included in this list, and yet I see their information. Therefore, I assume the Gateway has a generic BLE protocol interpreter and can read the information exposed by many more devices.

Sending MQTT Messages to the Gateway

An OpenMQTT Gateway can also read incoming messages. Let’s take a concrete look at the message format and what can be achieved.

Having covered the reading part, lets now take a look at how to send messages. The general requirements to send commands are as follows:

  1. Determine the base path to your device: In the official documentation, all example commands are prefixed with home/OpenMQTTGateway/. Replace this path with the device path that shows up in your MQTT explorer. In my case, that is lilygo for the RF gateway, and esp32-bleOMG_ESP32_BLE for the Bluetooth
  2. The payload needs to be formatted as JSON
  3. The command option you want to use needs to be compiled into the gateway binary. For example, when changing the configuration of the OLED device, I could not activate printing serial console messages because this feature is not included into the precompiled binaries offered in the web upload section.

If a command is accepted successfully, a new "response topic" will be created on the device. For example, when changing the radio frequency of a RF gateway by sending commands/MQTTtoRTL_433, the new topic MQTTtoRTL_433 is created and a message with a payload specifying the new configuration is generated.

General Gateway Configuration

When the board is based on an esp32, the following messages can be send as well:

  • Restart: This commands restarts the device
commands/MQTTtoSYS/config {"cmd":"restart"}
  • Erase: Completely erase the flash ram of the device
commands/MQTTtoSYS/config {"cmd":"erase"}
  • Status: The device status will be printed via serial or shown on the display
commands/MQTTtoSYS/config {"cmd":"status"}
  • Low Power Mode: Reduce the energy consumption off the device (but also its transmitting/receiving range)
// normal mode
commands/MQTTtoBT/config {"lowpowermode":0}
// low power mode
commands/MQTTtoBT/config {"lowpowermode":2}
  • Auto Discovery Mode: Can be enabled or disabled.
// enable
commands/MQTTtoSYS/config {"discovery":true}
//disable
commands/MQTTtoSYS/config {"discovery":false}
  • Configure WiFi credentials: Change the wifi credentials with an MQTT message
commands/MQTTtoSYS/config {"wifi_ssid":"ssid", "wifi_pass":"password"}
  • Change the MQTT broker credentials: You can even change which MQTT broker to use. After submitting this message, the device will no longer listen to the currently used broker - be sure to get all parameters right.
commands/MQTTtoSYS/config

{
  "mqtt_user": "user_name",
  "mqtt_pass": "password",
  "mqtt_server": "host",
  "mqtt_port": "port",
  "mqtt_secure": "false"
}
  • Change the main topic and device name:
commands/MQTTtoSYS/config

{
  "mqtt_topic": "openmqtt/",
  "gateway_name": "rf433"
}
  • Trigger the firmware update: A very useful feature to trigger OTA updates. I could not try this yet because at the time of writing this article, no new firmware was available
commands/MQTTtoSYS/firmware_update {"version": "latest"}

Configuring RF Gateways

The Gateway can be controlled as follows:

  • Change the listening Radio Frequency: The default frequency is exactly 433.92Mhz. This can be changed to any value in the ranges of 300-348 Mhz, 387-464Mhz, and 779-928Mhz. Set the value via sending a command to
commands/MQTTtoRTL_433 {"mhz":315.026}
  • Change the RSSI threshold value: This value determines the noise level for detecting the start and end of a signal transmission from a supported device.
commands/MQTTtoRTL_433 {"rssi": 9}
  • Receive the current status: This message instructs the device to broadcast its detailed RF configuration to the topic RTL_433toMQTT/status
commands/MQTTtoRTL_433 {"status":1}

Here is an example:

{
  "model": "status",
  "protocol": "rtl_433_ESP status message",
  "modulation": "OOK",
  "RTLRssi": -102,
  "RTLAVGRssi": -106,
  "RTLRssiThresh": -97,
  "signalRssi": -96,
  "RTLOOKThresh": 15,
  "train": 0,
  "RTLCnt": 286,
  "totalSignals": 1,
  "signalRatio": 0,
  "ignoredSignals": 1,
  "unparsedSignals": 0,
  "StackHWM": 5328,
  "RTL_HWM": 480,
  "DCD_HWM": 2176,
  "freeMem": 116800,
  "_enabledReceiver": 1,
  "receiveMode": 0
}

Configuring Bluetooth Gateways

For a Bluetooth gateway, only some properties can be changed.

  • Device Filtering: devices can be filtered with an allows or deny list. For this, the device ID needs to be determined and send to the appropriate topic.
// put on deny list
commands/MQTTtoBT/config {"black-list":["01:23:14:55:16:15"]}
// put on allow list
commands/MQTTtoBT/config {"white-list":["01:23:14:55:16:15"]}

For this filter to work, meaning it needs to detect a device to decide if it should be filtered, these lists need to be deactivated, then activated again.

// disable filtering
commands/MQTTtoBT/config {"ignoreWBlist":true}

// enable filtering
commands/MQTTtoBT/config {"ignoreWBlist":false}
  • Change the scan interval: You can set the time between scans to any value in milliseconds, or force an immediate scan
// Scan every 100 seconds
commands/MQTTtoBT/config {"interval":1000000}

// Scan immediateley
commands/MQTTtoBT/config {"interval":0}
  • Change the scan duration: A scan lasts 10s, but you can change this value as needed
commands/MQTTtoBT/config -m '{"scanduration":5000}'
  • Change the Bluetooth connection time interval: The gateway connects to detected devices periodically, this interval can be changed too. The value is in milliseconds.
// Try connect every 100 seconds
commands/MQTTtoBT/config -m '{"intervalcnct":300000}'
  • Publish all BLE devices, not only sensors: In its default setting, only BLE sensors are detected and reported. This can be changed with the following command:
commands/MQTTtoBT/config -m '{"onlysensors":true}'
  • Adaptive Scanning: A useful default configuration that uses a device database to decide if a device should be contacted actively and continuously, such as BLE trackers and motion detection, or if a regular broadcast signal, followed by a listening period, is sufficient. You can disable adaptive scanning and use the scan periods as-is
  • Setting the minimum RSSI for device connection: You can lower the RSSI value to -200 to discover very weak signals too
commands/MQTTtoBT/config -m '{"minrssi":-200}'
  • Sett detailed device date: Published device data is filtered and decoded to show essential identification only. Debug-like messages can be enabled with this option:
commands/MQTTtoBT/config -m '{"pubadvdata":true}
  • Publish raw data: You can capture and publish raw packet data too, opening up the option to implement a custom decoder. Enable this option as follows:
commands/MQTTtoBT/config -m '{"extDecoderEnable":true}

Configuring Modules

Connected sensors, actuators and displays can be controlled as well. Since this is a very device-specific configuration, I will only mention some examples, please refer to the original documentation about sensors, actuators and displays to learn about the specific commands that need to be used.

  • Measure the value of an ADC pin
  • Measure if a digital button is pressed
  • Trigger digital pins as on/off state
  • Control PWMs
  • Temperature and humidity measurements (DHT11, DHT22, HTU21, AHT10, AHT20, DS1820)
  • Motion sensor (HCSR501)
  • Control Neopixels (WS2812B)
  • LCD Display: on/off, brightness, orientation, display logo show serial log messages (SSD1306)

Configuring Detected Devices

The final question is: Can an OpenMQTT Gateway transmit configuration data, or on general any payload, to its devices? The answer depends on the Gateway type.

An RF gateway can be operated as a switch, which means it can receive as well as transmit data. The RC Switch Documntation shows several examples, and in one of them, the gateway itself is instructured to emit an RF message. The message home/OpenMQTTGateway/commands/MQTTto433 '{"value":1315156,"protocol":2,"length":24,"delay":315}' instructs which protocol, pulse length and delay to use for sending a specified value. However, in order to invoke a configuration change in the receiving device, this device also needs to be manufactured and programmed to do so. This is not the case with most RF devices - I assume as it would open the door to firmware updates in general, which is not common for simple devices.

For a Bluetooth gateway, the option to send data is given. For detected Bluetooth devices, the documentation lists very specific use cases dependent on the type of the device, including:

  • presence detection with BLE trackers
  • motion detection with BLE trackers and accelerometers
  • Change the presence detection topic that is used inside a home automation application, e.g. home assistant
  • Use an iBeacon UUID instead of its MAC address, which is randomized, as the published MQTT topic

Finally, there is also an option to read from/write to arbitrary data. The documentation example shows that the devices MAC address and MAC type need to be determined, and then a specific payload needs to be crafted.

// write data to BLE device
commands/MQTTtoBT/config -m '{
  "ble_write_address":"AA:BB:CC:DD:EE:FF",
  "ble_write_service":"cba20d00-224d-11e6-9fb8-0002a5d5c51b",
  "ble_write_char":"cba20002-224d-11e6-9fb8-0002a5d5c51b",
  "ble_write_value":"TEST",
  "value_type":"STRING",
  "ttl":4,
  "immediate":true }'

// read data from BLE device
ommands/MQTTtoBT/config -m '{
  "ble_read_address":"AA:BB:CC:DD:EE:FF",
  "ble_read_service":"cba20d00-224d-11e6-9fb8-0002a5d5c51b",
  "ble_read_char":"cba20002-224d-11e6-9fb8-0002a5d5c51b",
  "value_type":"STRING",
  "ttl": 2 }'

Conclusion

OpenMQTT Gateway is a strong community-driven product that comfortably integrates consumer products into your home automation software. You just need to setup a gateway for the signal type that you are interested in, such as RF 433 or Bluetooth, and then signals from detected devices will be captured, parsed, and published as MQTT messages. This article completely explored the topic structure and message content of the MQTT messages, both for messages send from the gateway, and for those messages that will be read by the gateway. The essential facts for send messages are these: a) All gateways expose their generic configuration options and signal specific information, b) sensor data only share a limited amount of general information, c) sensor data content is very product-specific, and which information is exposed can be checked looking into the project source code. And for messages that are read by the gateway: a) essential configuration, like Wi-Fi credentials, MQTT base topic, can be configured, b) gateway connected sensors, especially supported displays, can be configured, c) to a limited amount, raw signal packet data can be send from the gateway to its detected devices. Overall, one can only be impressed with the amount of versatile information that is provided, and the comfort with which to keep configured gateways up-to-date.


Previous: OpenMQTT Gateway for Infrared Signals