Forum Discussion

paolodaniele's avatar
paolodaniele
Icon for Altostratus rankAltostratus
Sep 19, 2022

WSS LTM not passing data correctly

Hi,

i'm having some trouble about running an Virtual Server in front of a K3s ingress.

I've setted up 4 virtual server, all point to the same destination address, but different port (different pool but final address is the same, only change the port)

For Http/s, MQTT/s zero problem, but when i'm trying to redirect the http traffic using websocket i've several problem.

Destination return several errors based on the Virtual Server tuning.

Actually i've the virtual server configured in TCP with HTTP profile (Websocket enable both client and server) and in security profile i've created a wss security to allow * addresses (HTTP,HTTPS,WS,WSS)

But when we try the application it return ERR_TRUNCATE_HEADERS. I've also create a new http profile based on the default, to increse the size of che Header Payload.

Also no i'm not checking the payload, and ignore any possible malformed packet (just to let everything pass).

On the application event log i can see all the request and i see the green thick so i suppose that everything is fine on the F5 side.

How can i be sure of that? Is there a way to do extra check or am i missing something?

Thank you guys.

Paolo

error
security
virtual server
websocket
ws
wss
  • mdevlin's avatar
    mdevlin
    Oct 06, 2022

    Hi Paolo, 

    Thank you for your time today, im glad we could determine what was wrong.

    A few steps we took that helped us confirm first off, that no truncating was happening in the headers, was to capture the connection in a tcpdump with the f5 high detail peer options (:nnnp), that showed us the payload on the backend as being perfect.  No truncating happening.

    tcpdump -vvv -s0 -ni 0.0:nnnp host <f5 VIP> and port <f5 virtual server port>

     

    One of the key factors to identifying what was the issue here was something you sent me in a private message.  Removing the http profile, which subsequently resultied in the websocket, waf and clientssl profile to be removed, allowed the connection to work.  The wss:// call, not ws://.  

    Because this was a WSS or WebSocket Secure call, the server was expecting a TLS session, and our configuration was doing TLS offloading at the f5, and sending the traffic plain text to the server.  Why the application generated an "ERR_TRUNCATE_HEADERS" message is purely speculation at this point, but i suspect it was because the headers were all encrypted and thus exceeded the maximum for the application.

    I would normally expect to see 400 errors with HTTP, so the backend encryption wasnt jumping out at me sooner.

    Im glad that we could resolve this matter, and please let myself and f5 know if there is anything else we can do to assist you in the future.

    Cheers

    Mike

  • mdevlin's avatar
    mdevlin
    Icon for Employee rankEmployee
    Sep 22, 2022

    paolodaniele What version of code are you running?

    Have you uploaded a qkview to ihealth? 

    Can you provide us the configurations of your virtual server, iRules (if applicable), policies and profiles (if you cant upload a qkview)

    • mdevlin's avatar
      mdevlin
      Icon for Employee rankEmployee
      Sep 22, 2022

      Additionally, if you remove the ASM security profile (for testing purposes), does it function as required/desired?

      • paolodaniele's avatar
        paolodaniele
        Icon for Altostratus rankAltostratus
        Sep 22, 2022

        I've tried to remove the ASM and it's not work.

        Btw the version 16.1.2.2 so i've not the problem that had older version (i've done some digging before writing you!)

        As i told to buulam i'm pretty noob with f5 and i didn't know the iHealth portal. I'm uploading the file to check with the tool.

        If i cannot find anything i'll post you the config part (i've to blur some ip and name for gdpr privacy)

        Thank you

  • mdevlin's avatar
    mdevlin
    Icon for Employee rankEmployee
    Oct 06, 2022

    Hi Paolo, 

    Thank you for your time today, im glad we could determine what was wrong.

    A few steps we took that helped us confirm first off, that no truncating was happening in the headers, was to capture the connection in a tcpdump with the f5 high detail peer options (:nnnp), that showed us the payload on the backend as being perfect.  No truncating happening.

    tcpdump -vvv -s0 -ni 0.0:nnnp host <f5 VIP> and port <f5 virtual server port>

     

    One of the key factors to identifying what was the issue here was something you sent me in a private message.  Removing the http profile, which subsequently resultied in the websocket, waf and clientssl profile to be removed, allowed the connection to work.  The wss:// call, not ws://.  

    Because this was a WSS or WebSocket Secure call, the server was expecting a TLS session, and our configuration was doing TLS offloading at the f5, and sending the traffic plain text to the server.  Why the application generated an "ERR_TRUNCATE_HEADERS" message is purely speculation at this point, but i suspect it was because the headers were all encrypted and thus exceeded the maximum for the application.

    I would normally expect to see 400 errors with HTTP, so the backend encryption wasnt jumping out at me sooner.

    Im glad that we could resolve this matter, and please let myself and f5 know if there is anything else we can do to assist you in the future.

    Cheers

    Mike

    • paolodaniele's avatar
      paolodaniele
      Icon for Altostratus rankAltostratus
      Oct 07, 2022

      Thank you Mike, Bu and all F5 Team for your help. It was very hard to find a solution because we do not know the application behaviour.

      Best Regards,

      P.

  • buulam's avatar
    buulam
    Icon for Admin rankAdmin
    Sep 21, 2022

    Hi paolodaniele , I'm not finding anything on the error you've noted. I'm asking around to see if there's any more thoughts on this. Seeing the requests in your application event log seems to be a positive sign for the BIG-IP passing along the traffic.

    • buulam's avatar
      buulam
      Icon for Admin rankAdmin
      Sep 22, 2022

      paolodaniele one thought was if you've tried to pass the traffic with an http profile attached at all to see if the traffic can do a simple pass through first?

  • paolodaniele's avatar
    paolodaniele
    Icon for Altostratus rankAltostratus
    Sep 22, 2022

    Hi Buulam,

    thank you for your answer. All i can say is that i'm quite noob with F5(i've always used other solutions) but i was able to learn quickly how it works and how to configure all things (the LTM HTTPS works fine without any problem)

    All i can tell is that if i configure to pass not only TCP but instead all protocols (so no configuration of websocket or anything else) when i'm try to wscat to wss://myaddress:myport i get the Connected message and its open the wss shell. Using this configuration when i run the wss://myaddress:myport i get the answer that it expects the request ad HTTP.

    This is why i'm thinking that it's me who's miss something in the config. Let me know if you want to see part of the configuration.

    Thanks for your help.

    Regards,

    P.

    • buulam's avatar
      buulam
      Icon for Admin rankAdmin
      Sep 22, 2022

      Right, so the test I'm suggesting where you remove the http profile will still actually let you pass the websocket traffic. It will do so without passing through the http proxy and can isolate things further for you.