Load Balancing based off TCP Payload entity

I added the log statement as recommended and also added MBLB profile as well. still the output from the ltm logs shows garbled. I am including the TCP payload from the packet capture as this data s text based and there is no reason the ltm should show garbled text. Here is the first push-ack from the GUI to the LTM after the 3 way handshake. the string I need to load balance is right there.

ltm virtual test1 {
    destination 5.5.5.5:https
    ip-protocol tcp
    mask 255.255.255.255
    pool test_Pool
    profiles {
        SSL_GUI {
            context clientside
        }
        RT_TCP { }
        mblb { }
    }
    rules {
        Test_Irule
    }
    source 0.0.0.0/0
    source-address-translation {
        pool snat_pool
        type snat
    }
    vlans {
        test_vlan
    }
    vlans-enabled
    vs-index 2199
}

just to be safe i tried viewing the log output from a different ssh client. both ssh tectia and Putty show slightly different formats of the payload. I am wondering if the LTM is simply not formatting the output correctly but is actually parsing correctly ?

SSH Tectia Client:

Jul 7 18:08:43 info tmm1[13133]: Rule /Common/Entity_Irule : o:t=)¾Ã¯c^¬ö|¤£ Jul 7 18:08:43 1 info tmm1[13133]: Rule /Common/Entity_Irule : À¾ÀÀºS»kWgâ ¬Â[ûÀÀ8à ÃÀ5ÃÃÀ9À8à ÃÀ/ÃÃÀ3À2ÃÃÀÃà ÃÀ Ãà ÀÀÀÀÿÀÀYÀ À4À2ÀÀÀÀÀÀÀÀ À ÀÀ À ÀÀ ÀÀÀÀÀÀÀÀÀÀÀ ÀÀÀÀÀÀÀÀqems.qa2.fxall.com Jul 7 18:08:48 1 info tmm5[13134]: Rule /Common/Entity_Irule : ÀÃÀÀÃS»pÃYÃJÃà ëþ·Â·Dü,K\¥µXmÂN «lì?oøáÃ}Iì)ELÃBõo à 1ÂYC$¿ÃÀ8à ÃÀ5ÃÃÀ9À8à ÃÀ/ÃÃÀ3À2ÃÃÀÃà ÃÀ Ãà ÀÀÀÀÿÀÀYÀ À4À2ÀÀÀÀÀÀÀÀ À ÀÀ À ÀÀ ÀÀÀÀÀÀÀÀÀÀÀ ÀÀÀÀÀÀÀÀqems.qa2.fxall.com Jul 7 18:08:48 info tmm5[13134]: Rule /Common/Entity_Irule : Jul 7 18:08:48 debug tmm5[13134]: Rule /Common/Entity_Irule : Entity Session routing and setting persistence ^[[?1;2cVT102VT102VT102^[[?1;2cVT102VT102VT102

Putty SSH Client Jul 7 17:46:22 info tmm7[13134]: Rule /Common/Entity_Irule : ▒Þ▒▒ÚS».a÷�sZ»®Õ÷J "°8vÉ�Þ�Âä� � «lì?oÀ�Ü}Iì)@Lèã�àXhÉ<1�YC$¿�▒8À À▒5ÀÀ▒9▒8À À▒/ÀÀ▒3▒2ÀÀ▒ÀÀ À▒ ÀÀ ▒▒▒▒ÿ▒▒Y▒ ▒4▒2▒▒▒▒▒▒▒▒ ▒ ▒▒ ▒ ▒▒ ▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒xxx.yyy.com

Jul 7 18:08:48 info tmm5[13134]: Rule /Common/Entity_Irule : ▒Þ▒▒ÚS»pÑYÉJßÄÏ«Ûþ·�·DÙ¼,K\¥µXm��N «lì?oÁ¸áÜ}Iì)ELßBõo� à1�YC$¿Ö▒8À À▒5ÀÀ▒9▒8À À▒/ÀÀ▒3▒2ÀÀ▒ÀÀ À▒ ÀÀ ▒▒▒▒ÿ▒▒Y▒ ▒4▒2▒▒▒▒▒▒▒▒ ▒ ▒▒ ▒ ▒▒ ▒▒▒▒▒▒▒▒▒▒▒ ▒▒▒▒▒▒▒▒xxx.yyy.com

Jul 7 18:08:48 debug tmm5[13134]: Rule /Common/Entity_Irule : Entity Session routing and setting persistence PuTTYPuTTYPuTTY

Ahhhh. I may have missed this earlier, but I see you're using a client SSL profile, which explains why the TCP data is garbled. TCP is OSI layer 4, but SSL offload doesn't happen until layer 6. You can generally replace the TCP:: commands with SSL:: commands and accomplish the same thing (but after decryption). Take a look at the SSL::collect wiki page for an example.

Let me give that a go, will keep everyone posted.

I made some good progess with the SSL irule and now the entity is correctly parsed out, but i think im missing something still as the connection never truly completes.

Here is the current irule:

when CLIENTSSL_HANDSHAKE {
   log local0. "[IP::client_addr]:[TCP::client_port]: SSL handshake completed, collecting SSL payload"
   SSL::collect
}
when CLIENTSSL_DATA {
   set payload [SSL::payload]
   set entityID [findstr $payload "@" 1 ","]
   log local0. "$entityID: $entityID"
   pool xxx_yyyy_Pool
   log local0. "blha Entity Session routing $entityID and setting persistence"
   persist uie $entityID 64000
   SSL::release
   }

Here are the log output when logging in: (at times I commented out the persist line since it throws tcl errors below)

Jul 8 17:57:56 va-bigip01 info tmm2[13133]: Rule /Common/x_Entity_Irule : 10.65.59.202:56070: SSL handshake completed, collecting SSL payload Jul 8 17:57:56 va-bigip01 info tmm2[13133]: Rule /Common/x_Entity_Irule : QA-xxx-CUST1: QA-xxx-CUST1 Jul 8 17:57:56 va-bigip01 info tmm2[13133]: Rule /Common/x_Entity_Irule : x Entity Session routing QA-xxx-CUST1 and setting persistence Jul 8 17:58:01 va-bigip01 info tmm7[13134]: Rule /Common/x_Entity_Irule : 10.65.59.202:56074: SSL handshake completed, collecting SSL payload Jul 8 17:58:01 va-bigip01 info tmm7[13134]: Rule /Common/x_Entity_Irule : : Jul 8 17:58:01 va-bigip01 info tmm7[13134]: Rule /Common/xEntity_Irule : x Entity Session routing and setting persistence Jul 8 18:04:22 va-bigip01 info tmm5[13134]: Rule /Common/x_Entity_Irule : 10.65.59.202:53658: SSL handshake completed, collecting SSL payload Jul 8 18:04:22 va-bigip01 info tmm5[13134]: Rule /Common/x_Entity_Irule : QA-xxx-PROV1: QA-xxx-PROV1 Jul 8 18:04:22 va-bigip01 info tmm5[13134]: Rule /Common/x_Entity_Irule : x Entity Session routing QA-xxx-PROV1 and setting persistence Jul 8 18:04:26 va-bigip01 info tmm4[13134]: Rule /Common/x_Entity_Irule : 10.65.59.202:53668: SSL handshake completed, collecting SSL payload Jul 8 18:04:26 va-bigip01 info tmm4[13134]: Rule /Common/x_Entity_Irule : : Jul 8 18:04:26 va-bigip01 info tmm4[13134]: Rule /Common/x_Entity_Irule : x Entity Session routing and setting persistence Jul 8 18:04:26 va-bigip01 err tmm4[13134]: 01220001:3: TCL error: /Common/x_Entity_Irule - attempt to use empty persistence key (line 1) invoked from within "persist uie $entityID 64000"

I also noticed the unviersal persistence when set doesn't seem to send the next entity to the next pool member even though the connection is not completing. in the output below I expect that all future uses matching CUST2 stay on 10.36.35:9820 but entity PROV1 be put on the other pool member 10.36.6.35:9820 not shown here

@(bigip01)(cfg-sync Changes Pending)(Active)(/Common)(tmos) show ltm persistence persist-records | grep universal universal QA-xxx-PROV1 10.195.127.9:443 10.36.6.35:9820 (tmm: 5) universal QA-xxx-CUST2 10.195.127.9:443 10.36.6.35:9820 (tmm: 2)

Okay, so given the log data (cleaned up for viewing):

10.65.59.202:56070: SSL handshake completed, collecting SSL payload 
QA-xxx-CUST1: QA-xxx-CUST1 
x Entity Session routing QA-xxx-CUST1 and setting persistence 
10.65.59.202:56074: SSL handshake completed, collecting SSL payload 
: 
x Entity Session routing and setting persistence 
10.65.59.202:53658: SSL handshake completed, collecting SSL payload 
QA-xxx-PROV1: QA-xxx-PROV1 
x Entity Session routing QA-xxx-PROV1 and setting persistence 
10.65.59.202:53668: SSL handshake completed, collecting SSL payload 
: 
x Entity Session routing and setting persistence 
TCL error: /Common/x_Entity_Irule - attempt to use empty persistence key (line 1) invoked from within "persist uie $entityID 64000"

Two things are evident:

1. Every other TCP session is missing the entityID data in the payload. The source port is different each time, so is this a single client interaction, or are you testing repeatedly?

2. You should probably check that the $entityID variable has something in it before attempting to use it for persistence

It may be more helpful at this point to spin up Wireshark to observe these interactions through a native connection (without LTM SSL offload). You can configure it to decrypt the SSL payload by inserting the private key into the RSA list of the SSL configuration. At a minimum you want to verify that the entityID is indeed in every request packet.

Hello Kevin

I ran a few GUI logins at the same time. I have a saved capture so let me pull that one up and ensure the entity is in each packet from the GUI.

OK i pulled the packets and see the following:

after initial 3 way handshake the GUI does send the entity ID as indicated by the logging result. subsequent packets from the GU are small 64 byte len and contain no readable payload until a few seconds later where the GUI send HB events that contain the entity ID. during this time the F5 is sending to the GUI all different data related to the application.

i guess what I am looking for is a way to persist the connection once the entityID is parsed and identified and stop collecting right after that point?

the entityID does have the string I need in it. right after we set the variable and log it i have a LB decision to be made but never see it complete, so i think its keeps collecting further and thats why it shows up emtpy the next time around.

Two thoughts:

1. Only attempt to persist if the entityID value is there. In your capture, when the entityID doesn't exist, is it within the same TCP session (same source port) as an original packet that did contain the entityID?

    

2. Try using standard SSL sessionID persistence. Assuming this is not an HTTP/browser-based application, does the client initiate SSL renegotiations? If not, and the initial SSL session is maintained for the life of the application session, then the sessionID should remain consistent.