For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

jrmorris_151361's avatar
jrmorris_151361
Icon for Nimbostratus rankNimbostratus
Sep 26, 2016

NATd traffic from internet not being load balanced

I have a pair of LTMs in my DMZ and am trying to setup a simple VS with pool for http traffic. I use a public IP externally which gets NATd at the firewall to a private address in my VS subnet. I then use my server pool, which uses an address from a subnet which is inline with the f5 (no SNAT). My issue is that I cannot reach the server from the internet. I see the NAT is working and the firewall is allowing it. Traffic makes it to the VS, but never gets load balanced. This all works internally (no NAT). I also see the http monitor working properly to the server. v11.6.1.

ltm virtual vs_businesscard {
    destination 10.244.250.29:http
    ip-protocol tcp
    mask 255.255.255.255
    persist {
        source_addr {
            default yes
        }
    }
    pool braid_http_pool
    profiles {
        tcp { }
    }
    source 0.0.0.0/0
    vs-index 48
}
ltm pool braid_http_pool {
    members {
        dmzvlpbraid:http {
            address 10.244.252.82
            session monitor-enabled
            state up
        }
    }
    monitor http 
}

7 Replies

  • I would run a tcpdump on your internal interface and see what is happening to the traffic when the F5 sends it to the server. Is it possible the server has a default gateway for internet IP's that goes somewhere other than F5?

     

    If "internal" is the name of your server-vlan , just do "tcpdump -ni internal port 80 and not host "

     

    replace internal-self-ip with whatever the self IP of the BIG-IP is on that interface. This will filter out your monitor traffic from the tcpdump.

     

  • Thanks. I had previously tried this but tried again. I see no traffic on that interface going to my node. Although I do see the hits on the 'external' interface.

     

  • Can you do a tcpdump like this: "tcpdump -ni 0.0 port 80 and not host " and post the lines of one of your failed requests?

     

    If you know your client's IP adddress that you are testing from, then put it in the tcpdump to make it more specific.

     

    tcpdump -ni 0.0 port 80 and host X.X.X.X

     

  • Here is the output. 10.244.250.29 is the virtual server. I even tried again adding the 'p' flag hoping it would show both sides, but nothing.

    [root@SJ505DMZF51:Active:In Sync] config  tcpdump -ni 0.0 port 80 and host 70.215.138.162 and not host 10.244.252.2
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on 0.0, link-type EN10MB (Ethernet), capture size 96 bytes
    15:28:42.901772 IP 70.215.138.162.eklogin > 10.244.250.29.http: S 875055479:875055479(0) win 55520 
    15:28:42.901816 IP 10.244.250.29.http > 70.215.138.162.eklogin: S 1034731441:1034731441(0) ack 875055480 win 4164 
    15:28:43.894852 IP 70.215.138.162.eklogin > 10.244.250.29.http: S 875055479:875055479(0) win 55520 
    15:28:44.219908 IP 70.215.138.162.prp > 10.244.250.29.http: S 1911881537:1911881537(0) win 55520 
    15:28:44.219948 IP 10.244.250.29.http > 70.215.138.162.prp: S 2060876685:2060876685(0) ack 1911881538 win 4164 
    15:28:45.214853 IP 70.215.138.162.prp > 10.244.250.29.http: S 1911881537:1911881537(0) win 55520 
    15:28:45.898836 IP 70.215.138.162.eklogin > 10.244.250.29.http: S 875055479:875055479(0) win 55520 
    15:28:45.901722 IP 10.244.250.29.http > 70.215.138.162.eklogin: S 1034731441:1034731441(0) ack 875055480 win 4164 
    15:28:47.219429 IP 10.244.250.29.http > 70.215.138.162.prp: S 2060876685:2060876685(0) ack 1911881538 win 4164 
    15:28:47.219904 IP 70.215.138.162.prp > 10.244.250.29.http: S 1911881537:1911881537(0) win 55520 
    15:28:49.906804 IP 70.215.138.162.eklogin > 10.244.250.29.http: S 875055479:875055479(0) win 55520 
    15:28:51.226945 IP 70.215.138.162.prp > 10.244.250.29.http: S 1911881537:1911881537(0) win 55520 
    15:28:51.901535 IP 10.244.250.29.http > 70.215.138.162.eklogin: S 1034731441:1034731441(0) ack 875055480 win 4164 
    15:28:53.219464 IP 10.244.250.29.http > 70.215.138.162.prp: S 2060876685:2060876685(0) ack 1911881538 win 4164 
    15:28:57.914814 IP 70.215.138.162.eklogin > 10.244.250.29.http: S 875055479:875055479(0) win 55520 
    15:28:59.242889 IP 70.215.138.162.prp > 10.244.250.29.http: S 1911881537:1911881537(0) win 55520 
    15:29:03.901803 IP 10.244.250.29.http > 70.215.138.162.eklogin: S 1034731441:1034731441(0) ack 875055480 win 4164 
    15:29:05.219540 IP 10.244.250.29.http > 70.215.138.162.prp: S 2060876685:2060876685(0) ack 1911881538 win 4164 
    15:29:13.947377 IP 70.215.138.162.eklogin > 10.244.250.29.http: S 875055479:875055479(0) win 55520 
    15:29:15.291316 IP 70.215.138.162.prp > 10.244.250.29.http: S 1911881537:1911881537(0) win 55520 
    ^C
    20 packets captured
    20 packets received by filter
    0 packets dropped by kernel
    [root@SJ505DMZF51:Active:In Sync] config  tcpdump -nnnpi 0.0 port 80 and host 70.215.138.162 and not host 10.244.252.2
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on 0.0, link-type EN10MB (Ethernet), capture size 96 bytes
    15:29:25.972565 IP 70.215.138.162.4248 > 10.244.250.29.80: S 1490764085:1490764085(0) win 55520 
    15:29:25.972608 IP 10.244.250.29.80 > 70.215.138.162.4248: S 3443494879:3443494879(0) ack 1490764086 win 4164 
    15:29:26.967711 IP 70.215.138.162.4248 > 10.244.250.29.80: S 1490764085:1490764085(0) win 55520 
    15:29:28.971680 IP 70.215.138.162.4248 > 10.244.250.29.80: S 1490764085:1490764085(0) win 55520 
    15:29:28.972652 IP 10.244.250.29.80 > 70.215.138.162.4248: S 3443494879:3443494879(0) ack 1490764086 win 4164 
    15:29:32.979650 IP 70.215.138.162.4248 > 10.244.250.29.80: S 1490764085:1490764085(0) win 55520 
    15:29:34.972676 IP 10.244.250.29.80 > 70.215.138.162.4248: S 3443494879:3443494879(0) ack 1490764086 win 4164 
    15:29:41.003740 IP 70.215.138.162.4248 > 10.244.250.29.80: S 1490764085:1490764085(0) win 55520 
    15:29:46.972504 IP 10.244.250.29.80 > 70.215.138.162.4248: S 3443494879:3443494879(0) ack 1490764086 win 4164 
    15:29:57.037604 IP 70.215.138.162.4248 > 10.244.250.29.80: S 1490764085:1490764085(0) win 55520 
    ^C
    10 packets captured
    10 packets received by filter
    0 packets dropped by kernel
    [root@SJ505DMZF51:Active:In Sync] config  
    
  • What kind of machine are you making the test request from or what program are you using? I may be wrong, but in looking at the tcpdump, I don't see client completing the 3-way handshake. Should be syn, syn-ack, ack The first two show the syn and syn-ack from BIG-IP, but then the client sends another syn instead of the ACK.

    You should see something like this:

    17:51:06.935331 IP 192.168.1.145.53926 > 173.59.8.62.443: Flags [S], seq 2008116475, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 1637511503 ecr 0,sackOK,eol], length 0
    17:51:07.015095 IP 173.59.8.62.443 > 192.168.1.145.53925: Flags [S.], seq 1287488537, ack 3747984218, win 17896, options [mss 1388,sackOK,TS val 307343764 ecr 1637511503,nop,wscale 4], length 0
    17:51:07.015142 IP 192.168.1.145.53925 > 173.59.8.62.443: Flags [.], ack 1, win 4128, options [nop,nop,TS val 1637511581 ecr 307343764], length 0
    

    I'm assuming that the monitor is marking the server as Green (available).

    So, you currently have the VIP set up with type "Standard" , can you try changing it to Performance Layer 4 and trying?

    Try setting up FastL4 and that way BIG-IP won't do any TCP handling and you can see what a tcpdump of that looks like.

    You may want to dump tcpdump to a file with "-w" flag to see more information and look at it in wireshark.

    Wireshark should show you if the connection looks healthy.
  • Yes, my health monitor is passing and it is using port 80. This is a RHEL virtual server.

    After changing the VS to PL4, it still fails, and I get the following in the dump.

    [root@SJ505DMZF51:Active:Changes Pending] config  tcpdump -ni 0.0 port 80 and host 70.215.136.16 and not host 10.244.252.2
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on 0.0, link-type EN10MB (Ethernet), capture size 96 bytes
    07:10:04.973074 IP 70.215.136.16.62628 > 10.244.252.82.http: R 3148814787:3148814787(0) ack 4048302278 win 0
    07:10:04.973087 IP 10.244.250.29.http > 70.215.136.16.10337: R 4048302278:4048302278(0) ack 3148814787 win 0
    07:10:07.295460 IP 70.215.136.16.10357 > 10.244.250.29.http: S 3757643125:3757643125(0) win 55520 
    07:10:07.295461 IP 70.215.136.16.10355 > 10.244.250.29.http: S 2017543511:2017543511(0) win 55520 
    07:10:07.295511 IP 70.215.136.16.59824 > 10.244.252.82.http: S 3757643125:3757643125(0) win 55520 
    07:10:07.295669 IP 10.244.252.82.http > 70.215.136.16.59824: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:07.295679 IP 10.244.250.29.http > 70.215.136.16.10357: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:07.295926 IP 70.215.136.16.33897 > 10.244.252.82.http: S 2017543511:2017543511(0) win 55520 
    07:10:07.296043 IP 10.244.252.82.http > 70.215.136.16.33897: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:07.296051 IP 10.244.250.29.http > 70.215.136.16.10355: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:08.293241 IP 70.215.136.16.10355 > 10.244.250.29.http: S 2017543511:2017543511(0) win 55520 
    07:10:08.293253 IP 70.215.136.16.33897 > 10.244.252.82.http: S 2017543511:2017543511(0) win 55520 
    07:10:08.293323 IP 10.244.252.82.http > 70.215.136.16.33897: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:08.293333 IP 10.244.250.29.http > 70.215.136.16.10355: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:08.295491 IP 10.244.252.82.http > 70.215.136.16.33897: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:08.295501 IP 10.244.250.29.http > 70.215.136.16.10355: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:08.293115 IP 70.215.136.16.10357 > 10.244.250.29.http: S 3757643125:3757643125(0) win 55520 
    07:10:08.293136 IP 70.215.136.16.59824 > 10.244.252.82.http: S 3757643125:3757643125(0) win 55520 
    07:10:08.293283 IP 10.244.252.82.http > 70.215.136.16.59824: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:08.293297 IP 10.244.250.29.http > 70.215.136.16.10357: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:08.295527 IP 10.244.252.82.http > 70.215.136.16.59824: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:08.295541 IP 10.244.250.29.http > 70.215.136.16.10357: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:10.295532 IP 10.244.252.82.http > 70.215.136.16.33897: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:10.295547 IP 10.244.250.29.http > 70.215.136.16.10355: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:10.295558 IP 10.244.252.82.http > 70.215.136.16.59824: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:10.295575 IP 10.244.250.29.http > 70.215.136.16.10357: S 3885761346:3885761346(0) ack 3757643126 win 14600 
    07:10:10.297168 IP 70.215.136.16.10357 > 10.244.250.29.http: S 3757643125:3757643125(0) win 55520 
    07:10:10.297183 IP 70.215.136.16.59824 > 10.244.252.82.http: S 3757643125:3757643125(0) win 55520 
    07:10:10.297287 IP 70.215.136.16.10355 > 10.244.250.29.http: S 2017543511:2017543511(0) win 55520 
    07:10:10.297301 IP 70.215.136.16.33897 > 10.244.252.82.http: S 2017543511:2017543511(0) win 55520 
    07:10:10.297421 IP 10.244.252.82.http > 70.215.136.16.33897: S 965825031:965825031(0) ack 2017543512 win 14600 
    07:10:10.297433 IP 10.244.250.29.http > 70.215.136.16.10355: S 965825031:965825031(0) ack 2017543512 win 14600 
    

    I looked at this in wireshark and am just seeing a ton of retransmissions and spurious retransmissions. Could this have something to do with the way the firewall is performing the NAT?

  • just for the fun can you put Address Translation to Auto Map on the virtual server and try again?