MQTT QoS2 Considerations

Konstantin Dotchkoff

Challenges associated with implementing QoS 2 in large scale distributed systems• Replication of QoS 2 messages and delivery state to other nodes

• is very expensive in terms of latency • increases the network traffic• impacts the overall system availability

• Network partitioning [split brain] invalidates QoS 2 behavior across nodes in different partitions• Distributed systems typically designed for availability and resiliency through

eventual consistency• Common alternatives are

• use of idempotent operationsE.g., for systems that exchange state changes “at least once” semantics is sufficient (assuming in order message delivery, unless the messages are commutative)

• de-duplication at the receiver

Specifics of the Azure Implementation• Largely distributed environment with 64 or more nodes (i.e. MQTT Server

side) – running clusters with 128 nodes is common• Nodes may enter or leave the cluster at any time• Clients may migrate between nodes

• Server to Client communication:• PUBREL message state needs to be replicated across all nodes• Received PUBCOMP message state needs to be replicated across all nodes

• Client to Server communication:• State of received messages needs to be replicated across all nodes

(or a messages in-flight list for all nodes needs to be managed centrally with concurrent access by all nodes)

• Received PUBREL messages need to be replicated across all nodes