Load balance messaging protocols like AMQP, MQTT, STOMP over TLS for ActiveMQ Artemis

Messaging protocols like AMQP, MQT, STOMP, OpenWire, HornetQ can all be load balanced trough the F5. Making the setup I encountered some issues. There is some specific configuration that needs to be apllied, which i will try to clearify. 

The setup: devices sending AMQP messages trough the F5 to a backend server over TLS. The F5 is configured as full proxy. The Virtual Server was pretty basic config. Standard Virtuals server, TCP, HTTP, SSL client & server profiles. Also websocket auto SNAT automap enabled.

I selected the virtual server which had the correct certificate in the Clientssl profile. The client initiating the request has our root /& issuing CA trusted in their truststore. Using a tcpdump and wireshark, I saw the corect TLS handshakes, Application data over TLS is being send, but not much data and due an error *** the client sends a TCP reset.

The F5 will only apply a profile, when that specific data is being detected (trigger). So yeah, there are TCP profiles to handle TCP and i have an SSL Client & server profile to handle TLS. Strangly it doesn't work. In the capture i took on the F5, wireshark sees the TLS application data as http-over-tls. hmm.. what if the F5 did the same? Then it would apply the http profile on the AMQP data, and that might screw things up. Disable the HTTP processing for that hostname, bingo. 

The next problem: the hostname, part off [HTTP::host] is not our event scopes. It is only activated when http profile is triggered. So the solution is to get the hostname, in this case the SNI (server name indication) from the TLS Client hello in event CLIENTSSL_HANDSHAKE. And check the SNI value to disable http processing. caveat: this will only work for TLS 1.2. When TLS 1.3 is used with encrypted SNI, another solution is needed.

Solution in short: it should work using a seperate virtual server with only TCP, SSL client & server profiles and have a load balancing default pool. If your setup is more complex and are reusing an existing VS, do the following.

I use a combination of an iRule and datagrouplists to extract the SNI, disable HTTP processing and send it to the right pool. For a current setup the SNI is also inserted server-side. I think this step is optional but i'll paste the code too.
datagrouplist  dgl_vs01_sni_targetpool: messaging.company.local and value "the targetpool"
datagrouplist dgl_vs01_disable_http_procesin: messaging.company.local without a value

when CLIENTSSL_HANDSHAKE {
   if { [SSL::extensions exists -type 0] } {
        set dgl "dgl_vs01_sni_targetpool"
        set dgl_nohttp "dgl_vs01_disable_http_processing"
        
        # read SNI value and place into variable sni_value
        binary scan [SSL::extensions -type 0] {@9A*} sni_value
        log local0. "sni value: [expr {[info exists sni_value] ? ${sni_value} : {not found} }]"
        
        #disable HTTP processing for AMQP, MQTT, STOMP, etc
        if { [class match $sni_value equals $dgl_nohttp] }{
            HTTP::disable
            log local0. "HTTP Disabled for $sni_value"
        }
        
        if { [class match $sni_value equals $dgl] }{
            set pool_target [class match -value [string tolower $sni_value] equals $dgl]
	        pool $pool_target 
	        log local0. "pool chosen for $sni_value, pool $pool_target "
	    }
    }
}

when SERVERSSL_CLIENTHELLO_SEND {
    #Inject SNI serverside
    if { [class match $sni_value equals $dgl] }{
        SSL::extensions insert [binary format SSScSa* 0 [expr { [set sni_length [string length $sni_value]] + 5 }] [expr { $sni_length + 3 }] 0 $sni_length $sni_value]
    log local0. "SNI inserted $sni_value"
   }
}

For those less comfterable with iRule, you can do SNI-based load balancing like this https://community.f5.com/t5/technical-articles/sni-routing-with-big-ip/ta-p/282018
but I'm not sure how to disable HTTP processing with a policy in the ssl client hello. If someone knows please post below, thx.

Now you are all set! Good luck with your F5 adventures.

Documentation used: 
https://clouddocs.f5.com/training/community/irules/html/class1/module1/iRuleEventsFlowHTTPS.html
https://community.f5.com/t5/crowdsrc/serverside-sni-injection-irule/ta-p/286745
https://community.f5.com/t5/crowdsrc/extracting-the-sni-server-name/ta-p/288029
https://activemq.apache.org/components/artemis/documentation/1.5.1/protocols-interoperability.html