MQTT Client Protocol (Message Queue Telemetry Transport)

Supported device types and versions 
About the MQTT Sparkplug protocol
Communication line configuration
Communication line parameters
Communication station configuration
I/O tag configuration
Literature
Document revisions

Supported device types and versions

The protocol is an implementation of the MQTT 3.1.1 standard (October 2014). MQTT protocol is a client/server protocol of a subscribe/publish type. It is simple, has little overhead, and is easy to implement. It is used for M2M communication (Machine to Machine) and in the IoT context (Internet of Things). The MQTT server is also called the MQTT broker.
D2000 KOM implements the client part of the protocol. The protocol is implemented on a TCP/IP line. MQTTS (Secure MQTT, MQTT over TLS) is also supported - either directly in the D2000 KOM process (using TLS settings on TCP/IP-TCP line) or via the tunnel utility.
For the transfer of LoRaWAN data encapsulated within the MQTT protocol, see LoRaWan protocol description.

The Sparkplug 3.0.0 standard (Sparkplug B MQTT Payload) is also partially supported. Support includes:

• reading simple value types
• reading arrays (with support for the Destination column)
• writing simple value types
• reading of template items (UDT)
• reading of dataset items

The communication was tested/deployed with:

• www.TheThings.network cloud 
• Loriot.io cloud 
• Slovanet cloud (loralink.slovanet.sk)
• Pixii PowerShaper (energy storage solution)
• liveobjects.orange-business.com cloud
• test.mosquitto.org (Sparkplug B)

Note: communication with the cloud liveobjects.orange-business.com via WebSockets (wss://liveobjects.orange-business.com:443/mqtt) was also tested. The program https://github.com/jimparis/unwebsockify.git was used as a WSS wrapper. This program started with the following parameters:
./unwebsockify.py --port 1883 --listen 172.16.0.1 wss://liveobjects.orange-business.com:443/mqtt
The D2000 KOM process connected to address 172.16.0.1 on port 1883. The WSS wrapper connected to the defined URL and wrapped the MQTT communication data in a WebSocket envelope.

Each PUBLISH message contains a topic (Topic), data (Payload), and level of confirmation (QoS). PUBLISH messages can be sent both by the client and the server. The clients at the beginning of the communication will use the SUBSCRIBE message to indicate what topics (parameter of Topic Filter protocol) they are interested in. 
The protocol defines the following levels of confirmation of PUBLISH messages - QoS (Quality of Service):

• QoS_0 - PUBLISH message is not confirmed, it may be lost
• QoS_1 - PUBLISH message is confirmed by the other side's PUBACK, it may be duplicated
• QoS_2 - PUBLISH message is confirmed by the other side's PUBREC which is then confirmed back by the PUBREL message and that one by a final PUBCOMP message.

The level of confirmation of the messages sent by the D2000 KOM process is defined by the protocol parameter Publish QoS. The D2000 KOM process considers the writing of the output tag to be successfully finished depending on the QoS:

• QoS_0 - after the data is successfully sent via the TCP connection
• QoS_1 - after receiving PUBACK
• QoS_2 - after receiving PUBCOMP

The MQTT communication starts with the CONNECT message sent by the client (D2000 KOM). The message contains User Name, Password, and other parameters, from which only Clean Session Flag and Client ID can be modified (parameter Will Flag is not used, as well as Will QoS and Will Retain, parameter Keep Alive is set to 0). The server replies with a CONNACK message with a return code that contains information about the success of the connect operation.

Then the client sends a SUBSCRIBE message with a filter of topics (Topic Filter parameter), specifying which topics it is interested in, and with the required maximum level of confirmation (parameter Subscribe QoS).

The server responds with a return code that contains information about the success and maximum QoS that was assigned to the requested topics.

Then follows a phase of communication, during which both the client and the server send PUBLISH messages (the client with any topic, the server with topics relating to the filter of topics of the received SUBSCRIBE message) and confirm them according to the value of the QoS parameter of the received PUBLISH messages.

If the server does not send a message for longer than Ping Interval seconds, the client sends the PING request message, to which the server must respond with the PING response message (within the time specified by the Reply Timeout parameter).
If parameters change on the line, the connection is closed and re-created.

About the MQTT Sparkplug protocol

Sparkplug's MQTT protocol defines 3 groups of applications:

• Edge Node - supports the MQTT protocol and connects to the MQTT server. It sends him data obtained from the Device or its own or aggregated data
• Device/Sensor - represents a physical or logical device connected to an Edge Node and providing data, process data, or metrics
• Host Application - represents a data consumer (SCADA/MES system, Historian, analysis tool) that connects to the MQTT server and receives MQTT data from the Edge Node/Device and, if necessary, commands. D2000 is a Host Application.

MQTT Sparkplug defines the following message types:

• NBIRTH – Birth certificate for Sparkplug Edge Nodes (information about the connection of the Edge Node to the MQTT Server)
• NDEATH – Death certificate for Sparkplug Edge Nodes (information about the loss of the connection between the MQTT Server and the Edge Node)
• DBIRTH – Birth certificate for Devices (information about Device connection to Edge Node)
• DDEATH – Death certificate for Devices (information about the loss of Edge Node connection with Device)
• NDATA – Edge Node data message (data from Edge Node)
• DDATA – Device data message (data from Device)
• NCMD – Edge Node command message (Host Application command for Edge Node)
• DCMD – Device command message (Host Application command for Device)
• STATE – Sparkplug Host Application state message (information from the state of the Host Application for the Edge Node/Device [online/offline]).

MQTT Sparkplug defines a Topic syntax for all message types, which has the form namespace/group_id/message_type/edge_node_id/[device_id], where:

• namespace is a spBv1.0 constant (for Sparkplug B)
• group_id is an arbitrary name of a logical group (e.g. by device type, operation, organizational structure, etc.)
• message_type is the message type
• edge_node_id is the identifier of the Edge Node
• device_id is the identifier of the Device (only if the message is from/for a Device)

Examples:

• spBv1.0/Sparkplug Devices/DDATA/Marvin/Sensor1
• pBv1.0/MIMIC/DDATA/20:19:AB:F4:0E:0D/sensor11
• spBv1.0/SparkplugDevices/NBIRTH/JsonScada

Note: STATE messages have a Topic of the form spBv1.0/STATE/sparkplug_host_id, where sparkplug_host_id is the unique name of the host application.

Examples:

• spBv1.0/STATE/CFGHUB2023
• spBv1.0/STATE/D2Kom

All message types except STATE have a binary Sparkplug payload. The STATE messages have a JSON payload (contains only the items "online" (True/False value) and "timestamp" (time stamp in milliseconds).

The Sparkplug payload contains metrics - each required metric can be configured into a single I/O tag. The metric contains a text identifier (specified in the I/O tag configuration as sparkplug_address), value type, value (simple/array/structured), optionally timestamp, and other attributes. D2000 supports working with simple values ​​and array values ​​- the first field item is stored in the I/O tag; it is possible to configure the Destination Column for the I/O tag, in which all array items are stored (or the first N, if the structure does not have enough rows).
Note: if a metric arrives with the is_null flag set, the I/O tag will have the Invalid value.

Communication line configuration

• Communication line category: TCP/IP-TCP.
• Host: IP address of MQTT server (or redundant addresses separated by a comma or semicolon).
• Port: the default port number is 1883 or 8883 for the encrypted SSL/TLS variant.
• Line number: unused, set the value to 0.

Note: The default port for the MQTT protocol is 1883 or 8883 for the SSL/TLS version. D2000 KOM contains an implementation of the SSL/TLS protocol variant, alternately t is possible to configure it by using the stunnel utility http://www.stunnel.org working in a client mode (client = yes). Stunnel running on the same computer as the D2000 KOM should listen to the 1883 local port and after connecting of D2000 KOM process to the port should encrypt the communication using SLL/TLS and send it to the target MQTT server (typically on port 8883).

Note: when using an SSL/TLS connection, you must pay attention to the expiration of the certificates used. The expiration date can be found on Linux with the command

openssl x509 -text -in file.crt | grep "After"

Forced disconnection: If all stations on the line are in the simulation mode or the communication is stopped for them, the line will be disconnected (the communication socket will be closed). If the simulation is disabled for at least one station and the communication is not stopped for it (the Parameters tab of the Station type object), the line will be connected again.

Communication line parameters

Dialog link configuration - Protocol parameters tab.
They affect some optional protocol parameters. The following protocol line parameters can be entered:

Table 1

Communication station configuration

• Communication protocol "MQTT Client Protocol".
• Station address: the station address corresponds to the Topic field in the PUBLISH message received from the MQTT server. The address can be a specific Topic, a regular expression, a # character representing all Topics, or a topic .* representing all Topics that are not suitable for other stations. The processing priority is as follows:
  • If there is a station with address # on the line, all messages are directed to its I/O tags and no further search is performed.
  • Otherwise, all other stations on the line are searched (except the .* address). If the Topic matches the address of a station, the message is directed to that station and no further search is performed.
  • Otherwise, all other stations on the line are searched (except the .* address), and their address is evaluated as a regular expression. If the Topic matches the station address, the message is directed to that station and no further search is performed.
    Stations are searched in descending order (by station address), so more specific terms go first (e.g., status/battery before status/batt.*)
  • Finally, if there is a station with a .* address, the message is addressed to it.
• Polling parameters on the Time parameters tab - recommended value is Delay=0.

Note: for a SparkPlug MQTT server, Topic has the form 'namespace/group_id/message_type/edge_node_id/[device_id]', where message_type indicates the message type (e.g. DDATA, DBIRTH, DDEATH).
A regular expression (e.g. spBv1.0/Sparkplug Devices/.*/MyDevice/Sensor2) can be used instead of message_type to cover all message types.
If Payload Type=Sparkplug, it is possible to omit the namespace and message_type parts and write the Topic in the abbreviated form 'group_id/edge_node_id/[device_id]' (e.g. Sparkplug Devices/MyDevice/Sensor2).

Note: If the station address is in abbreviated form, commands (DCMD, NCMD) are not processed for it. If it is in the form of a regular expression ('namespace/group_id/.*/edge_node_id/[device_id]'), the station also processes commands (including the command sent by the D2000 KOM process if Send Node Control/Rebirth=YES). Therefore, we recommend specifying the station address in abbreviated form. If it is also necessary to process commands (from other Host Applications), then create another station with an address in the form of a regular expression (e.g. spBv1.0/Sparkplug Devices/DCMD/MyDevice).

The Browse button opens a browse dialog for the station address. As long as the communication is functional, a dialogue containing the Topics received so far will be displayed. The list of received topics can be cleared with the Refresh button.
Double-clicking on a particular line will insert the value from the Address column into the configuration of the station from which the browse dialog was opened.
Note: The Station column shows the station to which the Topic has been assigned (based on the abovementioned processing priorities). For Sparkplug addresses, the abbreviated form of the address is displayed.

Station protocol parameters

Communication station - configuration dialog box - "Protocol parameters" tab.
They influence some of the optional parameters of the protocol.

Table 2

I/O tag configuration

Possible value types of I/O tags: Ci, Co, TxtI, TxtO, Qi, Ci, Co, Ai, Ao, Di, Do, TiR, ToR, TiA, ToA.

The MQTT implementation supports three work modes:

• Text mode: The original implementation of the MQTT protocol contained only input text I/O tags with the addresses IN_TOPIC, IN_DATA, and optionally a pair of I/O tags with the addresses IN_ID and ACK_ID. The first two I/O tags were used to publish the received Topic and Payload (which then needed to be parsed in the script), the second two points were used to publish the packet identifier and confirm the processing of the packet. Thus, it was possible to ensure that for data sent with QoS > QoS_0, confirmation was sent only after data processing in the script.
  Output I/O tags with the addresses OUT_TOPIC and OUT_VALUE are used for writing.
• JSON mode: An extension for processing Payload with JSON data (Payload Type=JSON) was implemented with the help of input I/O tags with addresses JA=json_address. The D2000 KOM process directly parses the JSON payload and sets the values of I/O tags​​with JSON addresses. I/O tags with addresses IN_TOPIC, IN_DATA, IN_ID, and ACK_ID may not exist at all.
  Output I/O tags with the addresses OUT_TOPIC and OUT_VALUE are used for writing.
• Sparkplug mode: An extension for Payload processing with Sparkplug data (Payload Type=Sparkplug) has been implemented using input I/O tags with addresses SA=sparkplug_address. The D2000 KOM process directly parses the Sparkplug payload and sets the values of I/O tags ​​with the Sparkplug addresses. I/O tags with addresses IN_TOPIC, IN_DATA, IN_ID, and ACK_ID may not exist at all.
  Writing simple values ​​is possible using output I/O tags with addresses ST=type;SA=sparkplug_address, where type is the Sparkplug definition of a data type (eg Int8, UInt16, DateTime, String, etc). The output I/O tags must be on the station with the Sparkplug address since the Topic is derived from it when writing.
  Note: messages of STATE type with a JSON payload can be parsed using I/O tags with a JSON address (typically an I/O tag of Di type with an address of JA=online).

{
    "metrics": [
        {
            "datatype": 3,
            "int_value": 7338992,
            "name": "Corrected Vol Acc Stn",
            "timestamp": 1729664005479
        }
    ],
    "seq": 32,
    "timestamp": 1729664005479
}

{
    "metrics": [
        {
            "datatype": 12,
            "datatype_txt": "String",
            "name": "Node Properties/Configuration",
            "string_value": "{}",
            "timestamp": 1730305529539,
            "timestamp_txt": "30-10-2024 17:25:29.539"
        },
        {
            "alias": 30064771073,
            "datatype": 5,
            "datatype_txt": "Uint8",
            "int_value": 0,
            "name": "Node Properties/Missing Param",
            "properties": {
                "keys": [
                    "usage"
                ],
                "values": [
                    {
                        "string_value": "technical information",
                        "type": 12,
                        "type_txt": "String"
                    }
                ]
            },
            "timestamp": 1730305529537,
            "timestamp_txt": "30-10-2024 17:25:29.537"
        },
        {
            "alias": 0,
            "dataset_value": {
                "columns": [
                    "topic_name",
                    "offset",
                    "length",
                    "crc"
                ],
                "num_of_columns": 4,
                "rows": [
                    {
                        "row": [
                            "N/A",
                            0,
                            0,
                            0
                        ]
                    }
                ],
                "types": [
                    12,
                    7,
                    7,
                    7
                ],
                "types_txt": [
                    "String",
                    "UInt32",
                    "UInt32",
                    "UInt32"
                ]
            },
            "datatype": 16,
            "datatype_txt": "DataSet",
            "name": "Node Control/FW Update",
            "timestamp": 1730305529537,
            "timestamp_txt": "30-10-2024 17:25:29.537"
        }
    ],
    "seq": 0,
    "timestamp": 1730305529536,
    "timestamp_txt": "30-10-2024 17:25:29.536"
}

Note: it is also possible to monitor the status of other Sparkplug Host Applications connected to the MQTT server. If the Identifier of Host Application is e.g. "ACME", so it is necessary to create a station with the address "spBv1.0/STATE/ACME" (or in abbreviated form "ACME") and on it an I/O tag of type Di with the address "JA=online" (since the Host Application sends a STATE message with a JSON payload).

Browse

In the Sparkplug mode, for the I/O tags, it is possible to discover the list of metrics, as long as the KOM process is running and communication with an MQTT server is established.
Clicking the Browse button opens the MQTT Item Browser window and displays a list of received metrics. The object list is created dynamically as a result of received messages.

The list of objects is dynamic, i.e. when a new value arrives in the KOM process, it is updated. Filtering in individual columns (with the exception of Time column) is also supported, asterisks can be used in the mask (e.g. *Short*).

Double-clicking on a particular line will insert the value from the Address column into the configuration of the I/O tag from which the MQTT Item Browser window was opened. At the same time, it is also inserted into the clipboard.

The Refresh button clears the list of values in both the CNF and the KOM process and optionally sends the Rebirth command (if Send Node Control/Rebirth=YES).

The Value column contains the received value.

Literature

Links
Official website of MQTT protocol http://mqtt.org

Official website of MQTT Sparkplug protocol https://sparkplug.eclipse.org


Specifications and Standards
MQTT 3.1.1 specification http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html
ISO/IEC 20922:2016 http://www.iso.org/standard/69499.html

Descriptions of Data Formats and API
www.loriot.io - Application API Data Format https://www.loriot.io/home/documentation.html#docu/app-data-format
www.thethingsnetwork.org - API Reference https://www.thethingsnetwork.org/docs/applications/mqtt/api.html

Document revisions

• Ver. 1.0 - August 8th, 2017 - document creation.
• Ver. 1.1 - October 15th, 2021 - support LastWill and Retain parameters
• Ver. 1.2 - October 27th, 2021 - support for parsing of JSON messages
• Ver. 1.3 - February 1st, 2022 - support for timestamps in JSON messages
• Ver. 1.4 - October 4th, 2024 - support for MQTT Sparkplug, support for browsing
• Ver. 1.5 - November 8th, 2024 - support for MQTTS (encrypted MQTT, MQTT over TLS)