MQTT Client Protocol (Message Queue Telemetry Transport)

Supported device types and versions 
About the MQTT Sparkplug protocol
Communication line configuration
Communication line parameters
Communication station configuration
I/O tag configuration
Tell commands
Literature
Document revisions

Supported device types and versions

The protocol is an implementation of the MQTT 3.1.1 standard (October 2014) and MQTT 5.0 standard (March 2019). MQTT protocol is a client/server protocol of a subscribe/publish type. It is simple, has little overhead, and is easy to implement. It is used for M2M communication (Machine to Machine) and in the IoT context (Internet of Things). The MQTT server is also called the MQTT broker.
D2000 KOM implements the client part of the protocol. The protocol is implemented on a TCP/IP line. MQTTS (Secure MQTT, MQTT over TLS) is also supported - either directly in the D2000 KOM process (using TLS settings on TCP/IP-TCP line) or via the tunnel utility.
For the transfer of LoRaWAN data encapsulated within the MQTT protocol, see LoRaWan protocol description.

The Sparkplug 3.0.0 standard (Sparkplug B MQTT Payload) is also supported. Support includes:

• Connecting to the MQTT broker as a Host Application (data consumer, in Payload Type = Sparkplug Host mode):
  • reading simple value types
  • reading arrays (with support for the Destination column)
  • writing simple value types
  • reading of template items (UDT)
  • reading of dataset items
  • support for aliases
  • compression support (GZIP, DEFLATE)
  • browsing - finding a list of metrics
• Connecting to an MQTT broker as an Edge Node (data producer, in Payload Type = Sparkplug Edge Node mode):
  • publishing values ​​with simple types
  • processing commands (NCMD, DCMD) with simple value types
  • processing commands (NCMD, DCMD) with template (UDT) items
  • processing commands (NCMD, DCMD) with dataset items
  • processing commands (NCMD, DCMD) with arrays
  • processing of the NCMD command with the Node Control/Next Server metric (connecting to the next MQTT server, if several are configured on the line)
  • processing of the STATE message with the content online=false, which comes from the Primary Host Application (disconnecting from the MQTT server and trying another one, if several are configured on the line - searching for an MQTT server with a connected Primary Host Application)
  • support for aliases

The communication was tested/deployed with:

• www.TheThings.network cloud 
• Loriot.io cloud 
• Slovanet cloud (loralink.slovanet.sk)
• Pixii PowerShaper (energy storage solution)
• liveobjects.orange-business.com cloud
• test.mosquitto.org (in both Payload Type = Sparkplug Host and Payload Type = Sparkplug Edge Node modes)
• broker.hivemq.com (in both Payload Type = Sparkplug Host and Payload Type = Sparkplug Edge Node modes)
• ThingsBoard (in Payload Type = Sparkplug Edge Node mode)
• Eclipse™ Sparkplug™ Technology Compatibility Kit (TCK) using version 3.0.0

Note: communication with the cloud liveobjects.orange-business.com via WebSockets (wss://liveobjects.orange-business.com:443/mqtt) was also tested. The program https://github.com/jimparis/unwebsockify.git was used as a WSS wrapper. This program started with the following parameters:
./unwebsockify.py --port 1883 --listen 172.16.0.1 wss://liveobjects.orange-business.com:443/mqtt
The D2000 KOM process was connected to address 172.16.0.1 on port 1883. The WSS wrapper was connected to the defined URL and wrapped the MQTT communication data in a WebSocket envelope.

Each PUBLISH message contains a topic (Topic), data (Payload), and level of confirmation (QoS). PUBLISH messages can be sent both by the client and the server. The clients at the beginning of the communication will use the SUBSCRIBE message to indicate what topics (parameter of Topic Filter protocol) they are interested in. 
The protocol defines the following levels of confirmation of PUBLISH messages - QoS (Quality of Service):

• QoS_0 - PUBLISH message is not confirmed; it may be lost
• QoS_1 - PUBLISH message is confirmed by the other side's PUBACK, it may be duplicated
• QoS_2 - PUBLISH message is confirmed by the other side's PUBREC, which is then confirmed back by the PUBREL message, and that one by a final PUBCOMP message.

The level of confirmation of the messages sent by the D2000 KOM process is defined by the protocol parameter Publish QoS. The D2000 KOM process considers the writing of the output tag to be successfully finished depending on the QoS:

• QoS_0 - after the data is successfully sent via the TCP connection
• QoS_1 - after receiving PUBACK
• QoS_2 - after receiving PUBCOMP

The MQTT communication starts with the CONNECT message sent by the client (D2000 KOM). The message contains User Name, Password, and other parameters, which can be modified:  Clean Session Flag, Client ID, Will Flag, Will QoS, Will Retain, Keep Alive, and Will Topic. The server replies with a CONNACK message with a return code that contains information about the success of the connect operation.

Then the client sends a SUBSCRIBE message with a filter of topics (Topic Filter parameter), specifying which topics it is interested in, and with the required maximum level of confirmation (parameter Subscribe QoS).

The server responds with a return code that contains information about the success and maximum QoS that was assigned to the requested topics.

Then follows a phase of communication, during which both the client and the server send PUBLISH messages (the client with any topic, the server with topics relating to the filter of topics of the received SUBSCRIBE message) and confirm them according to the value of the QoS parameter of the received PUBLISH messages.

If the server does not send a message for longer than Ping Interval seconds, the client sends the PING request message, to which the server must respond with the PING response message (within the time specified by the Reply Timeout parameter).
If parameters change on the line, the connection is closed and re-created.

About the MQTT Sparkplug protocol

Sparkplug's MQTT protocol defines 3 groups of applications:

• Edge Node - supports the MQTT protocol and connects to the MQTT server. It sends him data obtained from the Device or its own or aggregated data
• Device/Sensor - represents a physical or logical device connected to an Edge Node and providing data, process data, or metrics
• Host Application - represents a data consumer (SCADA/MES system, Historian, analysis tool) that connects to the MQTT server and receives MQTT data from the Edge Node/Device and, if necessary, commands.

The D2000 can operate in Host Application mode or Edge Node mode (plus one or more Device/Sensor devices) according to the Payload Type line parameter. In Edge Node mode, exactly one station with an Edge Node address and optionally additional stations with slave Device/Sensor addresses must be configured on the line.

MQTT Sparkplug defines the following message types:

• NBIRTH – Birth certificate for Sparkplug Edge Nodes (information about the connection of the Edge Node to the MQTT Server)
• NDEATH – Death certificate for Sparkplug Edge Nodes (information about the loss of the connection between the MQTT Server and the Edge Node)
• DBIRTH – Birth certificate for Devices (information about Device connection to Edge Node)
• DDEATH – Death certificate for Devices (information about the loss of Edge Node connection with Device)
• NDATA – Edge Node data message (data from Edge Node)
• DDATA – Device data message (data from Device)
• NCMD – Edge Node command message (Host Application command for Edge Node)
• DCMD – Device command message (Host Application command for Device)
• STATE – Sparkplug Host Application state message (information from the state of the Host Application for the Edge Node/Device [online/offline]).

MQTT Sparkplug defines a Topic syntax for all message types, which has the form namespace/group_id/message_type/edge_node_id/[device_id], where:

• namespace is a spBv1.0 constant (for Sparkplug B)
• group_id is an arbitrary name of a logical group (e.g. by device type, operation, organizational structure, etc.)
• message_type is the message type
• edge_node_id is the identifier of the Edge Node
• device_id is the identifier of the Device (only if the message is from/for a Device)

Examples:

• spBv1.0/Sparkplug Devices/DDATA/Marvin/Sensor1
• pBv1.0/MIMIC/DDATA/20:19:AB:F4:0E:0D/sensor11
• spBv1.0/SparkplugDevices/NBIRTH/JsonScada

Note: STATE messages have a Topic of the form spBv1.0/STATE/sparkplug_host_id, where sparkplug_host_id is the unique name of the host application  (Sparkplug Host ID parameter).

Examples:

• spBv1.0/STATE/CFGHUB2023
• spBv1.0/STATE/D2Kom

All message types except STATE have a binary Sparkplug payload. The STATE messages have a JSON payload - they contain only the items "online" (True/False value) and "timestamp" (time stamp in milliseconds).

The Sparkplug payload contains metrics - each required metric can be configured into a single I/O tag. The metric contains a text identifier (specified in the I/O tag configuration as sparkplug_address), value type, value (simple/array/structured), optionally a timestamp, and other attributes. D2000 supports working with simple values ​​and array values ​​- the first field item is stored in the I/O tag; it is possible to configure the Destination Column for the I/O tag, in which all array items are stored (or the first N, if the structure does not have enough rows). D2000 can read items of template (UDT) and Dataset metrics.
Note: If a metric arrives with the is_null flag set, the I/O tag will have the Invalid value.

Recommended settings for optimizing data flow

• If the MQTT server supports Topic aliases, use MQTT Version = MQTT 5.0, and set Topic Alias Maximum to a value greater than the number of stations on the line (for Payload Type = Sparkplug Edge Node) or greater than the number of topics expected from the MQTT broker (for Payload Type = Text only / JSON). On the MQTT broker side, ensure that its Topic Alias ​​Maximum value is greater than the number of topics that will be sent to the D2000 KOM process, so that numeric aliases can be used instead of text topics.
  Note: the Topic Alias ​​Maximum value that the MQTT broker sends when connecting as a parameter of the CONNACK message is visible in the communication logs:
  09:31:12.086 13-05-2025|D|MQTT> CONNACK Property # 1 Topic Alias Maximum ( 34)
  09:31:12.087 13-05-2025|T|MQTT> Recv:<00><0A>
  09:31:12.088 13-05-2025|D|MQTT> CONNACK Property value= 10
  
  
• Use aliases for metrics (for Payload Type = Sparkplug Edge Node): automatically using the Sparkplug Alias Mode=Automatic (HOBJ) parameter or manually using the SAL=alias item in the I/O tag address. If you use manual aliases, use low numbers (aliases 1-127 are encoded as 1 byte).
  
  
• For Payload Type = Sparkplug Edge Node: if there are many changes to the output I/O tags, we recommend setting the Batch Size parameter to a larger value than the default (10) and possibly increasing the Delay parameter in the station's time parameters. This will cause fewer messages to be generated (and each will contain multiple metrics), which reduces the overhead required to transmit one metric. If specific I/O tags change frequently, the message may contain several values ​​for the same I/O tag (with timestamps). The disadvantage is the delay in the values.

Communication line configuration

• Communication line category: TCP/IP-TCP or TCP/IP-TCP Redundant.
• Host: IP address of MQTT server (or redundant addresses separated by a comma or semicolon).
  Note: In Payload Type = Sparkplug Edge Node mode, the Node Control/Next Server metric is supported, which can be used to command the D2000 KOM process to connect to the next MQTT server (if multiple servers are specified).
• Port: the default port number is 1883 or 8883 for the encrypted SSL/TLS variant.
• Line number: unused, set the value to 0.

Note: The default port for the MQTT protocol is 1883 or 8883 for the SSL/TLS version. D2000 KOM contains an implementation of the SSL/TLS protocol variant; it is also possible to configure it by using the stunnel utility http://www.stunnel.org working in a client mode (client = yes). Stunnel running on the same computer as the D2000 KOM should listen to the 1883 local port, and after connecting of D2000 KOM process to the port, it should encrypt the communication using SSL/TLS and send it to the target MQTT server (typically on port 8883).

Note: When using an SSL/TLS connection, you must pay attention to the expiration of the certificates used. The expiration date can be found on Linux with the command

openssl x509 -text -in file.crt | grep "After"

Notes on MQTT broker redundancy

If the TCP/IP-TCP line is configured, one TCP connection is created, which can be directed to one of multiple configured IP addresses.

If the TCP/IP-TCP Redundant line is configured, two TCP connections are created (to 2 MQTT brokers), each of which can be directed to one of multiple configured IP addresses. The D2000 KOM process receives values from both MQTT brokers and sends all values of the output I/O tags to both MQTT brokers. The write is considered successful if the value is successfully sent to at least one MQTT broker (in the case of confirmation, if "Subscribe QoS" > QoS_0, if it is confirmed by at least one of the MQTT brokers). Such a configuration allows you to configure a redundant system in which the D2000 KOM connects to two redundant MQTT brokers and the other side to at least one MQTT broker, while the MQTT brokers do not have to form a "cluster", i.e. synchronize messages with each other (so the simple and free Eclipse Mosquitto can also be used).

Forced disconnection: If all stations on the line are in the simulation mode or the communication is stopped for them, the line will be disconnected (the communication socket will be closed). If the simulation is disabled for at least one station and the communication is not stopped for it (the Parameters tab of the Station type object), the line will be connected again.

Communication line parameters

Dialog line configuration - Protocol parameters tab.
They affect some optional protocol parameters. The following protocol line parameters can be entered:

Table 1

Communication station configuration

• Communication protocol "MQTT Client Protocol".
• Polling parameters on the Time parameters tab - recommended value is Delay=0.
• Station address: The station address corresponds to the Topic field in the PUBLISH message received from the MQTT server. The address can be a specific Topic, a regular expression, a # character representing all Topics, or a topic .* representing all Topics that are not covered by other stations. The processing priority is as follows:
  • If there is a station with address # on the line, all messages are directed to its I/O tags, and no further search is performed.
  • Otherwise, all other stations on the line are searched (except the .* address). If the Topic matches the address of a station, the message is directed to that station, and no further search is performed.
  • Otherwise, all other stations on the line are searched (except the .* address), and their address is evaluated as a regular expression. If the Topic matches the station address, the message is directed to that station, and no further search is performed.
    Stations are searched in descending order (by station address), so more specific terms go first (e.g., status/battery before status/batt.*)
  • Finally, if there is a station with a .* address, the message is addressed to it.

For a SparkPlug MQTT server, the Topic has the form 'namespace/group_id/message_type/edge_node_id/[device_id]', where message_type indicates the message type (e.g., DDATA, DBIRTH, DDEATH).

If Payload Type = Sparkplug Host: A regular expression (e.g., spBv1.0/Sparkplug Devices/.*/MyDevice/Sensor2) can be used instead of message_type to cover all message types. It is, however, recommended to omit the namespace and message_type parts and write the Topic in the abbreviated form group_id/edge_node_id/[device_id] (e.g., Sparkplug Devices/MyDevice/Sensor2).
Note: Disabling communication on all stations will cause communication to be disconnected from the MQTT broker.

If Payload Type = Sparkplug Edge Node: One station on the line should correspond to the Edge Node; its address must be in the form group/node (e.g., myGroup/myNode). There may be other stations corresponding to the Device/Sensor on the line, their addresses must be in the form group/node/device (e.g., myGroup/myNode/myDevice1), where group/node is the same as the address of the station corresponding to the Edge Node. According to the Sparkplug standard, these Device/Sensor stations belong to the corresponding Edge Node station (are its substations) and represent devices with which the Edge Node station communicates (e.g., via Modbus protocol).
Note: Disabling communication on the Edge Node station will cause communication to be disconnected from the MQTT broker.

Note:
If Payload Type = Sparkplug Host and the station address is in abbreviated form, the station doesn't process commands (DCMD, NCMD). If it is in the form of a regular expression ('namespace/group_id/.*/edge_node_id/[device_id]'), the station also processes commands (including the command sent by the D2000 KOM process if Send Node Control/Rebirth=YES). Therefore, we recommend specifying the station address in abbreviated form. If it is also necessary to process commands (from other Host Applications), then create another station with an address in the form of a regular expression (e.g. spBv1.0/Sparkplug Devices/DCMD/MyDevice).
If Payload Type = Sparkplug Edge Node, the station processes commands (DCMD, NCMD) and ignores data messages (NDATA, DDATA, NBIRTH, DBIRTH).

The Browse button opens a browse dialog for the station address. As long as the communication is functional, a dialogue containing the Topics received so far will be displayed. The list of received topics can be cleared with the Refresh button.
Double-clicking on a particular line will insert the value from the Address column into the configuration of the station from which the browse dialog was opened.
Note: The Station column shows the station to which the Topic has been assigned (based on the abovementioned processing priorities). For Sparkplug addresses, the abbreviated form of the address is displayed.

Station protocol parameters

Communication station - configuration dialog box - "Protocol parameters" tab.
They influence some of the optional parameters of the protocol.

Table 2

I/O tag configuration

Possible value types of I/O tags: Ci, Co, TxtI, TxtO, Qi, Ci, Co, Ai, Ao, Di, Do, TiR, ToR, TiA, ToA.

The MQTT implementation supports three work modes:

• Text mode: The original implementation of the MQTT protocol contained only input text I/O tags with the addresses IN_TOPIC, IN_DATA, and optionally a pair of I/O tags with the addresses IN_ID and ACK_ID. The first two I/O tags were used to publish the received Topic and Payload (which then needed to be parsed in the script), the second two points were used to publish the packet identifier and confirm the processing of the packet. Thus, it was possible to ensure that for data sent with QoS > QoS_0, confirmation was sent only after data processing in the script.
  Output I/O tags with the addresses OUT_TOPIC and OUT_VALUE are used for writing.
• JSON mode: An extension for processing Payload with JSON data (Payload Type=JSON) was implemented with the help of input I/O tags with addresses JA=json_address. The D2000 KOM process directly parses the JSON payload and sets the values of I/O tags​​with JSON addresses. I/O tags with addresses IN_TOPIC, IN_DATA, IN_ID, and ACK_ID may not exist at all.
  Output I/O tags with the addresses OUT_TOPIC and OUT_VALUE are used for writing.
• Sparkplug mode: An extension for Payload processing with Sparkplug data (Payload Type = Sparkplug Host/Edge Node) has been implemented using input I/O tags with addresses SA=sparkplug_address. The D2000 KOM process directly parses the Sparkplug payload and sets the values of I/O tags ​​with the Sparkplug addresses. I/O tags with addresses IN_TOPIC, IN_DATA, IN_ID, and ACK_ID may not exist at all.
  Writing simple values ​​is possible using output I/O tags with addresses ST=type;SA=sparkplug_address, where type is the Sparkplug definition of a data type (eg Int8, UInt16, DateTime, String, etc). The output I/O tags must be on the station with the Sparkplug address since the Topic is derived from it when writing.
  Note: Messages of STATE type (generated by Host Application clients) with a JSON payload can be parsed using I/O tags with a JSON address (typically an I/O tag of Di type with an address of JA=online).

{
    "metrics": [
        {
            "datatype": 3,
            "int_value": 7338992,
            "name": "Corrected Vol Acc Stn",
            "timestamp": 1729664005479
        }
    ],
    "seq": 32,
    "timestamp": 1729664005479
}

{
    "metrics": [
        {
            "datatype": 12,
            "datatype_txt": "String",
            "name": "Node Properties/Configuration",
            "string_value": "{}",
            "timestamp": 1730305529539,
            "timestamp_txt": "30-10-2024 17:25:29.539"
        },
        {
            "alias": 30064771073,
            "datatype": 5,
            "datatype_txt": "Uint8",
            "int_value": 0,
            "name": "Node Properties/Missing Param",
            "properties": {
                "keys": [
                    "usage"
                ],
                "values": [
                    {
                        "string_value": "technical information",
                        "type": 12,
                        "type_txt": "String"
                    }
                ]
            },
            "timestamp": 1730305529537,
            "timestamp_txt": "30-10-2024 17:25:29.537"
        },
        {
            "alias": 0,
            "dataset_value": {
                "columns": [
                    "topic_name",
                    "offset",
                    "length",
                    "crc"
                ],
                "num_of_columns": 4,
                "rows": [
                    {
                        "row": [
                            "N/A",
                            0,
                            0,
                            0
                        ]
                    }
                ],
                "types": [
                    12,
                    7,
                    7,
                    7
                ],
                "types_txt": [
                    "String",
                    "UInt32",
                    "UInt32",
                    "UInt32"
                ]
            },
            "datatype": 16,
            "datatype_txt": "DataSet",
            "name": "Node Control/FW Update",
            "timestamp": 1730305529537,
            "timestamp_txt": "30-10-2024 17:25:29.537"
        }
    ],
    "seq": 0,
    "timestamp": 1730305529536,
    "timestamp_txt": "30-10-2024 17:25:29.536"
}

Note: it is also possible to monitor the status of other Sparkplug Host Applications connected to the MQTT server. If the Identifier of Host Application is e.g. "ACME", so it is necessary to create a station with the address "spBv1.0/STATE/ACME" (or in abbreviated form "ACME") and on it an I/O tag of type Di with the address "JA=online" (since the Host Application sends a STATE message with a JSON payload).

Browse

In the Sparkplug Host mode, for the I/O tags, it is possible to discover the list of metrics, as long as the KOM process is running and communication with an MQTT server is established.
(For I/O tags in Sparkplug Edge Node, it is only possible to find the list of metrics to which writes were made using NCMD/DCMD Topics).
Clicking the Browse button opens the MQTT Item Browser window and displays a list of received metrics. The object list is created dynamically as a result of received messages.

The list of objects is dynamic, i.e. when a new value arrives in the KOM process, it is updated. Filtering in individual columns (with the exception of the Time column) is also supported; asterisks can be used in the mask (e.g., *Short*).

Double-clicking on a particular line will insert the value from the Address column into the configuration of the I/O tag from which the MQTT Item Browser window was opened. At the same time, it is also inserted into the clipboard.

The Refresh button clears the list of values in both the CNF and the KOM process and optionally sends the Rebirth command (In the Sparkplug Host mode, if Send Node Control/Rebirth=YES).

The Value column contains the received value.

Tell commands

Literature

Links
Official website of the MQTT protocol http://mqtt.org

Official website of the MQTT Sparkplug protocol https://sparkplug.eclipse.org


Specifications and Standards
MQTT 3.1.1 specification http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html
ISO/IEC 20922:2016 http://www.iso.org/standard/69499.html

Descriptions of Data Formats and API
www.loriot.io - Application API Data Format https://www.loriot.io/home/documentation.html#docu/app-data-format
www.thethingsnetwork.org - API Reference https://www.thethingsnetwork.org/docs/applications/mqtt/api.html

Document revisions

• Ver. 1.0 - August 8th, 2017 - document creation.
• Ver. 1.1 - October 15th, 2021 - support LastWill and Retain parameters
• Ver. 1.2 - October 27th, 2021 - support for parsing of JSON messages
• Ver. 1.3 - February 1st, 2022 - support for timestamps in JSON messages
• Ver. 1.4 - October 4th, 2024 - support for MQTT Sparkplug (in Sparkplug Host mode), support for browsing
• Ver. 1.5 - November 8th, 2024 - support for MQTTS (encrypted MQTT, MQTT over TLS)
• Ver. 1.6 - April 25, 2025 - support for MQTT Sparkplug (in Sparkplug Edge Node mode)
• Ver. 1.7 - May 5, 2025 - support for aliases in MQTT Sparkplug mode
• Ver. 1.8 - May 12, 2025 - support for MQTT 5.0