Load balance messaging protocols like AMQP, MQTT, STOMP over TLS for ActiveMQ Artemis
Messaging protocols like AMQP, MQT, STOMP, OpenWire, HornetQ can all be load balanced trough the F5. Making the setup I encountered some issues. There is some specific configuration that needs to be apllied, which i will try to clearify. The setup: devices sending AMQP messages trough the F5 to a backend server over TLS. The F5 is configured as full proxy. The Virtual Server was pretty basic config. Standard Virtuals server, TCP, HTTP, SSL client & server profiles. Also websocket auto SNAT automap enabled. I selected the virtual server which had the correct certificate in the Clientssl profile. The client initiating the request has our root /& issuing CA trusted in their truststore. Using a tcpdump and wireshark, I saw the corect TLS handshakes, Application data over TLS is being send, but not much data and due an error *** the client sends a TCP reset. The F5 will only apply a profile, when that specific data is being detected (trigger). So yeah, there are TCP profiles to handle TCP and i have an SSL Client & server profile to handle TLS. Strangly it doesn't work. In the capture i took on the F5, wireshark sees the TLS application data as http-over-tls. hmm.. what if the F5 did the same? Then it would apply the http profile on the AMQP data, and that might screw things up. Disable the HTTP processing for that hostname, bingo. The next problem: the hostname, part off [HTTP::host] is not our event scopes. It is only activated when http profile is triggered. So the solution is to get the hostname, in this case the SNI (server name indication) from the TLS Client hello in eventCLIENTSSL_HANDSHAKE. And check the SNI value to disable http processing. caveat: this will only work for TLS 1.2. When TLS 1.3 is used with encrypted SNI, another solution is needed. Solution in short: it should work using a seperate virtual server with only TCP, SSL client & server profiles and have a load balancing default pool. If your setup is more complex and are reusing an existing VS, do the following. I use a combination of an iRule and datagrouplists to extract the SNI, disable HTTP processing and send it to the right pool. For a current setup the SNI is also inserted server-side. I think this step is optional but i'll paste the code too. datagrouplist dgl_vs01_sni_targetpool: messaging.company.local and value "the targetpool" datagrouplist dgl_vs01_disable_http_procesin: messaging.company.local without a value when CLIENTSSL_HANDSHAKE { if { [SSL::extensions exists -type 0] } { set dgl "dgl_vs01_sni_targetpool" set dgl_nohttp "dgl_vs01_disable_http_processing" # read SNI value and place into variable sni_value binary scan [SSL::extensions -type 0] {@9A*} sni_value log local0. "sni value: [expr {[info exists sni_value] ? ${sni_value} : {not found} }]" #disable HTTP processing for AMQP, MQTT, STOMP, etc if { [class match $sni_value equals $dgl_nohttp] }{ HTTP::disable log local0. "HTTP Disabled for $sni_value" } if { [class match $sni_value equals $dgl] }{ set pool_target [class match -value [string tolower $sni_value] equals $dgl] pool $pool_target log local0. "pool chosen for $sni_value, pool $pool_target " } } } when SERVERSSL_CLIENTHELLO_SEND { #Inject SNI serverside if { [class match $sni_value equals $dgl] }{ SSL::extensions insert [binary format SSScSa* 0 [expr { [set sni_length [string length $sni_value]] + 5 }] [expr { $sni_length + 3 }] 0 $sni_length $sni_value] log local0. "SNI inserted $sni_value" } } For those less comfterable with iRule, you can do SNI-based load balancing like thishttps://community.f5.com/t5/technical-articles/sni-routing-with-big-ip/ta-p/282018 but I'm not sure how to disable HTTP processing with a policy in the ssl client hello. If someone knows please post below, thx. Now you are all set! Good luck with your F5 adventures. Documentation used: https://clouddocs.f5.com/training/community/irules/html/class1/module1/iRuleEventsFlowHTTPS.html https://community.f5.com/t5/crowdsrc/serverside-sni-injection-irule/ta-p/286745 https://community.f5.com/t5/crowdsrc/extracting-the-sni-server-name/ta-p/288029 https://activemq.apache.org/components/artemis/documentation/1.5.1/protocols-interoperability.html
MQTT Broker - logging irule
Hi i am trying to setup an MQTT LTM VS and running into a few issues, i was trying to use some of the sample irules to capture some more logging but can't quite figure out this one that should log the messages https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm-iot-administration-13-0-0/1.html Common/irule_mqtt_loggging:14: error: [undefined procedure: MQTT::message_id][MQTT::message_id] /Common/irule_mqtt_loggging:26: error: ["The command does not expect argument."][MQTT::message topics qos $topic] i had thought that these were standard variables that could just be used...if anyone has managed to get this working any help would be apprecated thanks chris
F5 BIG-IP MQTT protocol support and use cases in an IoT environment
The Internet-of-Things is a system of physical devices, vehicles, buildings, and any other items each provided with a unique identifier allowing them to communicate with one another without any human intervention. MQTT is a widely-used machine-to-machine connectivity protocol that helps these devices communicate and exchange messages. The wide adoption and popularity of the MQTT protocol led to its support in the BIG-IP version 13.0. MQTT is a publish/subscribe messaging protocol. A device such as a camera, heat sensor, lP-enabled light bulb, etc. can publish data to an intermediary module using the MQTT protocol. An application can then subscribe to this intermediary module and retrieve the published data. Intermediary modules are also known as message brokers. The BIG-IP with MQTT awareness can sit in front of the MQTT message brokers to provide scalability, security, visibility, and availability. Fig. 1 – Basic architecture of using the BIG-IP platform in an IoT environment. In an MQTT configuration, clients publish messages and the BIG-IP system parses those MQTT messages based on which it invokes MQTT specific iRule events if defined and performs connection based load balancing on the MQTT messages to a pool of message brokers. The message brokers then transport and route the messages to subscribing servers A typical BIG-IP MQTT configuration includes: MQTT pool of message brokers Are grouped together to receive and process traffic. After the pool is created, associate the pool with a virtual server. Some common open source message brokers examples are Mosquitto and VerneMQ iRules for MQTT Use iRules to exercise MQTT functionality, for example to log the messages that the BIG-IP system passes, or to pass the client certificate's common name in the CONNECT message Client SSL profile Use a Client SSL profile when you want the BIG-IP ® system to authenticate and decrypt/encrypt client-side application traffic Virtual server configured to use MQTT functionality For more details please refer to: https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm-iot-administration-13-0-0/1.html BIG-IP natively supports the MQTT protocol. BIG-IP has a full proxy architecture which makes BIG-IP itself an endpoint and an originator of the MQTT protocol. iRules take advantage of the deep understanding of the protocol and can be used at different events (specific moments within the session flow of a network connection, such as MQTT_CLIENT_INGRESS, MQTT_SERVER_INGRESS) to intelligently parse the MQTT protocol and make traffic decisions. Let’s explore some of the use cases using iRules such as load balancing and steering traffic based on a unique identified present within the MQTT protocol, preventing attacks based on TOPICS/LENGTH of a PUBLISH message, client authentication based on CN name present in the client certificate. Use Case: prevent MQTT DDoS attacks based on the below parameters Topic validation Validate the ‘TOPIC’ of a PUBLISH message and if it does not fit the validation criteria then drop the connection. This will prevent messages with bogus TOPICS from reaching the message brokers. And help in preventing MQTT DDoS attacks based on TOPIC. when MQTT_CLIENT_INGRESS { log local0. "CLIENT [MQTT::type]" if {[MQTT::type] eq "PUBLISH" } { # Check against a allowed topic. if { [MQTT::topic] eq "bogus/bogus" } { # Invalid topic log local0. " Topic is Invalid. Potential DDoS Attack -> Dropping Topic: [MQTT::topic]" drop } } } The above iRule can be enhanced to use data groups where the data groups will have the entire list of TOPICS and the [MQTT::TOPIC] can be checked against the data group MQTT PUBLISH message length validation An application will typically have knowledge about the size of the data that is being transmitted. Taking advantage of that knowledge the length of a PUBLISH message can be validated and if it does not satisfy the expected length then drop the connection. This will prevent large messages from reaching the message brokers and not overload the broker. when MQTT_CLIENT_INGRESS { log local0. "CLIENT [MQTT::type]" if {[MQTT::type] eq "PUBLISH" } { # Check length of message. if { [MQTT::length] > 100 } { # Invalid length log local0. " Length is Invalid. Potential DDoS Attack -> Dropping message: [MQTT::topic] : [MQTT::length]" drop } } } Use Case: pool selection/load balancing based on username as a unique identifier Use the username as a unique identifier to steer traffic to a message broker. This enables stickiness between an end user and a message broker by mapping a device/end user to a pool in the load balancer. The mapping helps in conditions where in a device needs to reconnect to the message broker on which the active client session lives and can use the state information present on the message broker to make decisions. when MQTT_CLIENT_INGRESS { log local0. "CLIENT [MQTT::type]" if {[MQTT::type] eq "CONNECT" } { # Select pool based on username, example license ID if { [MQTT::username] eq "D1111111" } { log local0. "Load balancing to pool MQTT-BrokerPool1" pool MQTT-BrokerPool1 } if { [MQTT::username] eq "D1111112" } { log local0. "Load balancing to pool MQTT-BrokerPool2" pool MQTT-BrokerPool2 } } } The above iRule can be enhanced to use data groups where the data groups will have the list of unique usernames. Multiple data groups can be defined, each one representing users for multiple Pools. The [MQTT::username] can be checked against the data group and based on that the pool selection can be made. Use Case: client certificate authentication over TLS BIG-IP can terminate MQTT encrypted SSL/TLS messages and then send the un-encrypted traffic to the backend Broker. This allows the Broker to scale as it can offload all SSL/TLS functions and configuration to the BIG-IP. In addition to terminating SSL/TLS the BIG-IP can use iRules to perform Client Certificate Authentication. The BIG-IP can embed the client CN name from the client certificate into the backend connection along with the username for authentication by the back-end Broker. when CLIENT_ACCEPTED { set cn "" } when CLIENTSSL_CLIENTCERT { log local0. " Parsing X509 Info for SSL Client Cert CN Name" set cn [findstr [X509::subject [SSL::cert 0]] "CN=" 3 ","] log local0. "Found Client Cert Common Name (CN) : $cn" } when MQTT_CLIENT_INGRESS { log local0. "MQTT Client MmessageTtype [MQTT::type]" if {[MQTT::type] == "CONNECT"} { log local0. "Connecting to Broker" if {$cn == ""} { # if we didn't see a client cert, return an authentication error MQTT::drop MQTT::respond type CONNACK return_code 5 MQTT::disconnect } else { log local0. "Passing Username and CN to Backend for Authentication" log local0. "MQTT Username: [MQTT::username]" log local0. "Username: [MQTT::username] CN: $cn" set mqttuser [MQTT::username] MQTT::username "$mqttuser $cn" } } } Conclusion Bookmark this page if you are interested in learning more. We will be update this blog with new compelling IoT features developed by F5 for the upcoming BIG-IP releases. Do you have a suggestion for an IoT related topic that you would like for us to cover? Please leave a comment on this post if you do. References https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm-iot-administration-13-0-0/1.print.html https://devcentral.f5.com/s/articles/lightboard-lessons-iot-on-big-ip-24923?tag=lbl https://devcentral.f5.com/s/articles/the-iot-ready-platform https://f5.com/resources/white-papers/tmos-redefining-the-solution https://devcentral.f5.com/s/articles/the101-irules-101-events-amp-priorities https://en.wikipedia.org/wiki/MQTT https://mosquitto.org/Lightboard Lessons: Connecting Cars with BIG-IP
I light up how BIG-IP and Solace work together in a MQTT connected car infrastructure. ps Related: Using F5 BIG-IP and Solace Open Data Movement technology for MQTT message routing and delivery Lightboard Lessons: What is MQTT? Solace and F5: Partnering for Better IoTUsing F5 BIG-IP and Solace Open Data Movement technology for MQTT message routing and delivery
The "Internet of Things" (IoT) has the potential to not just impact the way we live and work, but to disrupt entire industries. Some of the major industries that are driving innovation in IoT include manufacturing, transportation, and utilities. In this post I’ll talk about an example of how digital manufacturing is helping automobiles be more intelligent. There are millions of connected automobiles that have software running on them: from programs that keep GPS maps up to date to sensors sending vehicle speeds or temperature information. This information typically is being sent to a centralized management system that enables intelligent decisions, and those back-end systems also need to send data, instructions, and updates to the cars. We are actively working with a leading multinational automotive company. In this deployment the management system uses MQTT to send notifications to cars that a new software update is available for download. It is critical that these messages are received by the connected cars, but message delivery can be delayed if the car can’t be reached because it’s out of range, or driving through a tunnel, or even if cellular networks are congested. Using the Solace message broker To ensure eventual delivery in such situations, you can have the management system send messages to a message broker that in turn distributes them to intended recipients. This decouples your architecture so the many elements of the management system don’t need to worry about the specific destinations to which each message must be delivered, nor track delivery status. They simply send a message to the broker which A) takes care of routing it to requisite destinations, and B) stores a copy of it until every vehicle that needs it has confirmed receipt. Basic IIoT architecture for connected automobiles Before diving deeper, let’s look at the architecture that is used in the above use case. The idea is the same, but to support the numbers of connected vehicles and volume of data involved, it’s necessary to implement multiple message brokers. This architecture is provided by Solace which uses a novel architecture that enables efficient publish/subscribe distribution of data and can scale to millions of connected devices. (see Fig 1.). Back-end applications connect to a single broker (core message broker) to send a message to any vehicle, and can use messaging techniques preferred by IT applications such as JMS, AMQP and Node.js. The core message broker is communicating to several edge message brokers through internal messaging. The data for a particular vehicle is routed through one edge message broker and delivered to the vehicle in the matter conducive for it, in this case MQTT. In the diagram, Vehicle 1 connects to Message Broker 1. That means that messages sent from the management system to the core message broker that are intended for Vehicle 1 only need to go to Message Broker 1 Fig. 1 – Basic IIoT architecture for connected automobiles This two-tier approach provides a simple and powerful scaling architecture for north-south communication, since more connected cars can be added by adding more edge message routers without affecting the existing system. BIG-IP and persistence For this architecture, we’re aiming to achieve two things: First, we want to relieve the management system of the burden of keeping track of message delivery to potentially millions of vehicles. The Solace message broker uses MQTT QoS Level 1 which guarantees that a message will be delivered at least once to the receiver. The message is persisted in the edge message broker which allows the backend IoT application to publish messages with confidence that they will eventually reach their destination, but without needing to keep track of message delivery status. Second, we need to make sure that the vehicle connects to the right message broker so it can receive its messages. This can be achieved by managing a 1:1 mapping between the edge message broker and the vehicle. Fig. 2 – F5 BIG-IP in the IoT architecture BIG-IP version 13+ helps to achieve the 1:1 mapping with support for MQTT message parsing. It sits in front of the message broker and parses the MQTT messages – using iRules – exchanged between the vehicle to the message broker. Data groups on the BIG-IP are used to map a vehicle using a unique Identifier to a pool of message brokers. Using iRules every MQTT message is parsed, then the data group lookup routes the data to the correct pool of message brokers. This helps to achieve a “sticky” load balancing solution so that when a vehicle comes back online it will always get routed and connected to the correct message brokers where the connection state for that vehicle is known to be maintained. We say a “pool of message brokers” in order for the brokers to provide a highly-available service. Here is some sample code of an iRule that helps with achieving persistence or sticky load balancing: when MQTT_CLIENT_INGRESS { log local0. "CLIENT [MQTT::type]" if {[MQTT::type] eq "CONNECT" } { # Select pool based on vehicle ID if { [MQTT::username] eq "D1111111" } { log local0. "Load balancing to pool MQTT-BrokerPool1" pool MQTT-BrokerPool1 } if { [MQTT::username] eq "D1111112" } { log local0. "Load balancing to pool MQTT-BrokerPool2" pool MQTT-BrokerPool2 } } } Conclusion IoT use cases which benefit from having a guaranteed delivery of messages can take advantage of F5’s persistence capabilities. The architecture described above means that specific edge broker instances will maintain 'state' for specific connected vehicles. This requires the MQTT load-balancing to be sticky across sessions. BIG-IP version 13+ supports MQTT message parsing using iRules and thereby provides a key functionality in achieving the end-to-end goal of the solution for this type of architecture. The industries that are embracing “Internet of Things” applications are still in the early stages of innovation, but market leaders are architecting now for scale and resilience so they can automate more and more processes with a single infrastructure. The automaker in this example has chosen F5’s BIG-IP solution for 1:1 mapping of cars to message brokers, and Solace’s Open Data Movement technology as the MQTT infrastructure for message routing and delivery. The combination provides an elegant solution that can scale to millions of connections to support an increasing number of message exchanges between the vehicles and back end systems as their telematics and in-car services get more sophisticated.Lightboard Lessons: IoT on BIG-IP
As more organizations deploy IoT applications in their data centers and clouds, they're going to need their ADC to understand the unique protocols these devices use to communicate. In this Lightboard Lesson, I light up how IoT protocol MQTT (Message Queuing Telemetry Transport) works on BIG-IP v13. iRules allow you to do Topic based load balancing along with sensor authentication. And if you missed it, here is the #LBL on What is MQTT? ps Related: Lightboard Lessons: What is MQTT? Security Trends in 2016: Securing the Internet of Things The Intruders of Things The IoT Ready Platform Using F5 BIG-IP with IBM MessageSightLightboard Lessons: What is MQTT?
The mad dash to connect virtually every noun to the internet or The Internet of Things, is creating a massive M2M network for all the devices, systems, sensors and actuators to connect & communicate on the Internet. With that, they need a communications protocol to understand each other. One of those is Message Queue Telemetry Transport (MQTT). MQTT is a “subscribe and publish” messaging protocol designed for lightweight machine-to-machine (or IoT) communications. In this episode of Lightboard Lessons, I light up how MQTT works. ps Related: IoT Ready Infrastructure IoT Effect on Applications IoT Influence on Society What are These "Things? The Intruders of Things
