BigIP exhibiting TCP zero Window behavior and closing connection
We have a VIP on our BigIP that sees a lot of connections. The TCP profile applied to the VIP is TCP Progressive with no Nagle. What we are seeing is after a very small amount of data the Window size on the F5 Self IP to Server connection is going to zero. The back-end server is able to maintain its buffer levels. The intention of applying a progressive profile was to take care of varying conditions and connections. We saw over 600 TCP Zero Window instance from the F5 to the server. all being sent by the F5. I am not sure what tweaks to the TCP profile i need to do to solve this problem.
LTM :: Zero Window Server Side :: TCP Profiles
We have a virtual server setup for our receiving mail system, and it has been configured as-is for quite some time (measured in years). Never before has an issue arisen, but recently a particular client has been having problems sending attachments to us (and as far as we are aware, ONLY that client). What they claim to see is that the connection is terminated. Normal email works fine. Small file attachments work fine. However when they send us attachments that are Mb in size, the connection will not be successful. On our side, we see the window size slowly creep down until it hits zero. The BIG-IP probes the mail system, the mail system acks the probe, but keeps the window size at zero. It does this until the zero window timeout is reached on the BIG-IP and the connection is terminated by the BIG-IP (TCP RST). This is what the window decrease looks like on the client side (tcp.stream eq 3 and ip.src eq [the mail system]): This is what the window decrease looks like on the server side (tcp.stream eq 2 and ip.src eq [the VIP]): Client side end of the connection: Server side end of the connection: My impression initially was that this is not a BIG-IP problem... but when we remove the BIG-IP from the path, the connection works fine regardless of attachment size. Again, works fine for everyone else as far as we know regardless of if the BIG-IP is in the path... which is perplexing. Things I've tried: * Switching-out TCP profiles (lan optimized, wan optimized, client and server matching and different in combinations of the above). Now on mptcp-mobile-optimized with defaults. * Moving TLS off of the F5 * Resetting TLS profile to defaults * Different mail systems (of same type/configuration) Current configuration: * VIP on port 25 * TCP profile with mptcp-mobile-optimized w/defaults * SSL Profile (defaults w/cert, optional SSL, allowed cipher suites) * SMTPS Profile (allows TLS) * Pool w/single mail system * iRule w/VIP bounceback * Source IP Persistence VIP bounceback iRule: when LB_SELECTED { if {[IP::addr "[IP::client_addr]/24" equals "[LB::server addr]/24"]} { snat automap } else { snat none } } Any ideas/thoughts/suggestions all welcome. Thanks for taking the time.
iRule to close an established connection
I have tcp (not http) based service where client connections are permanent. By that I mean that once a connection to a pool member gets established its stays there 24x7. The pool has 2 pool members configured with priority group. The first pool member has priority 2 and the second one priority 1, with a Less than '1' value for Priority Group Activation. The pool also has the setting of 'Reject' for 'Action On Service Down'. That takes care of any scenario where a pool member is marked down by health monitors. Whenever a highest priority pool is marked down by health monitors all established connections to that pool members get closed automatically. The client applications immediately try to reconnect and get established connections to the second pool member with the lower priority. So far, everything is exactly what we want to accomplish. The challenge comes when the higher priority pool member is marked up/available once again. We're looking for an automatic way to close the already established connections to the lower priority pool member as soon as the higher priority pool member becomes available. Is there a way to do so? Not sure what event I should use for an already established connection. First ones that came to mind were LB_SELECTED and CLIENT_ACCEPTED. So far, I've tried the following options without any results: when LB_SELECTED { if { [LB::status pool poolname member 10.0.0.1 80 ] equals "up" and [IP::addr [LB::server addr] equals 10.0.0.2] } { reject } } when LB_SELECTED { if { [LB::status pool poolname member 10.0.0.1 80 ] equals "up" and [IP::addr [LB::server addr] equals 10.0.0.2] } { LB::reselect pool poolname member 10.0.0.1 } }
TCP Rewrite Rule used in Syslog TCP
Hi Dev/Central community! I've a SIEM with two syslog/tcp recievers (Let's name it R1 and R2). I 've created a VS to listen a 514/TCP, recieve the Syslog TCP message and send it to R1. In case R1 is down, the VS will send the Syslog TCP message to R2. As my SIEM assign a tag to each message recieved with the client IP, I need to rewrite the syslog message before send it to the R1 or R2 receivers (because I see the f5 self ip as client IP in the recievers). So, I've writed an iRule to rewrite the header of each syslog message before send it. this is my irule so far: when CLIENT_ACCEPTED { # Tomo la IP del cliente que se conecta al VS / Get the client IP connecting to the VS set ip_original [IP::remote_addr] # Tomo el Payload y la paso al siguiente nivel / Get the tcp payload to send it to Client Data TCP::collect log local0. "Client Accepted from $ip_original" } when CLIENT_DATA { set OrgininalTCPLength [TCP::payload length] # Primer <PRI> del payload / Try to detect <PRI> header in very first payload bytes regsub {^<\d+>} [TCP::payload] "\[\]\[\]\[$ip_original\]\[[clock seconds]\]\[\] " string # CRLF 0d0a \r\n + <PRI> / Look for another syslog message in the same TCP Payload regsub -all {\r\n<\d+>} $string "\r\n\[\]\[\]\[$ip_original\]\[[clock seconds]\]\[\] " string set len [TCP::payload length] TCP::payload replace 0 $len $string set ModifiedTCPLength [TCP::payload length] # Se pasa el Payload al siguiente nivel / Send the modified payload to the next level log local0. "Forwarindg message from $ip_original \t original length: $OrgininalTCPLength \t modified length: $ModifiedTCPLength" TCP::release #Preparo una nueva recoleccion / Get ready for a new collection TCP::collect } The iRule works like a charm, but in some very little times, it seem to doesn't rewrite the message... Any clue/ideas/troubleshooting tips? Regards, Maxmptcp-mobile-optimized and Hardware SYN Cookie Protection
Does anyone know why the TCP protocol profile mptcp-mobile-optimized ships with Hardware SYN Cookie Protection disabled? It is still enabled on tcp-mobile-optimized. Here is a copy of my two profiles, which should be the default: ltm profile tcp mptcp-mobile-optimized { abc disabled app-service none congestion-control illinois defaults-from tcp delay-window-control disabled delayed-acks disabled dsack disabled ecn enabled hardware-syn-cookie disabled init-cwnd 16 limited-transmit enabled mptcp enabled nagle enabled pkt-loss-ignore-burst 0 pkt-loss-ignore-rate 0 proxy-buffer-high 131072 proxy-buffer-low 131072 rate-pace enabled receive-window-size 131072 reset-on-timeout disabled selective-acks enabled send-buffer-size 262144 slow-start enabled timestamps enabled } ltm profile tcp tcp-mobile-optimized { abc disabled app-service none congestion-control high-speed defaults-from tcp delay-window-control disabled delayed-acks disabled dsack disabled ecn enabled init-cwnd 16 limited-transmit enabled nagle enabled pkt-loss-ignore-burst 0 pkt-loss-ignore-rate 0 proxy-buffer-high 131072 proxy-buffer-low 131072 receive-window-size 131072 reset-on-timeout disabled selective-acks enabled send-buffer-size 131072 slow-start enabled timestamps enabled }LTM ignores tcp rst or fin flags and timouts connection
Posting on behalf of a member who is having trouble posting this: Hello, we have issue with one pool member. We have enabled https monitor for member. Monitor works fine and production traffic as well but we have noticed short service interruptions. After deeper investigation is seems that LTM ignores TCP packet with FIN or RST flag set and did not close TCP connection. Take a look for tcpdump from LTM: 14:58:32.285201 IP 10.211.200.243.18724 > 10.211.212.72.https: S 1154236661:1154236661(0) win 14600 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.299810 IP 10.211.212.72.https > 10.211.200.243.18724: S 2860122587:2860122587(0) ack 1154236662 win 14480 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.299961 IP 10.211.200.243.18724 > 10.211.212.72.https: . ack 1 win 115 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.300273 IP 10.211.200.243.18724 > 10.211.212.72.https: P 1:130(129) ack 1 win 115 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317345 IP 10.211.212.72.https > 10.211.200.243.18724: P 1:167(166) ack 130 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317487 IP 10.211.200.243.18724 > 10.211.212.72.https: . ack 167 win 123 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317895 IP 10.211.200.243.18724 > 10.211.212.72.https: P 130:205(75) ack 167 win 123 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317976 IP 10.211.200.243.18724 > 10.211.212.72.https: P 205:322(117) ack 167 win 123 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.333454 IP 10.211.212.72.https > 10.211.200.243.18724: . ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360053 IP 10.211.212.72.https > 10.211.200.243.18724: P 167:332(165) ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360329 IP 10.211.200.243.18724 > 10.211.212.72.https: P 322:375(53) ack 332 win 131 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360408 IP 10.211.212.72.https > 10.211.200.243.18724: P 332:385(53) ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360618 IP 10.211.212.72.https > 10.211.200.243.18724: F 385:385(0) ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.374770 IP 10.211.212.72.https > 10.211.200.243.18724: R 2860122919:2860122919(0) win 0 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 15:03:33.381555 IP 10.211.200.243.18724 > 10.211.212.72.https: R 375:375(0) ack 386 win 0 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=62E inslot=0 inport=0 haunit=0 priority=0 rst_cause="[0x1eb69cd:625] {peer} Flow expired (sweeper) (idle timeout) Any idea what is going on? In the same pool we have another pool member but with different monitor and it works fine.Huge number of TCP 3WHS rejected (bad ACK), chksum incorrect
Hi guys, Hope you can help me with this, for me, complete mystery. I'm getting lots of following: Wireshark text export from F5 tcpdump: 4815 17:27:58.597830 CLIENT_IP F5_VS_IP TCP 162 36562 → 443 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=681665250 TSecr=0 WS=128 4816 17:27:58.597846 F5_VS_IP CLIENT_IP TCP 193 443 → 36562 [SYN,ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 TSval=751619333 TSecr=681665250 SACK_PERM=1 4817 17:27:58.660439 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=1 Ack=1 Win=29200 Len=0 TSval=681665313 TSecr=751619333 4818 17:27:58.730179 CLIENT_IP F5_VS_IP TLSv1.2 380 Client Hello 4819 17:27:58.730201 F5_VS_IP CLIENT_IP TLSv1.2 4529 Server Hello 4820 17:27:58.792837 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=196 Ack=4345 Win=37648 Len=0 TSval=681665445 TSecr=751619465 4821 17:27:58.792854 F5_VS_IP CLIENT_IP TLSv1.2 706 Certificate, Server Hello Done 4822 17:27:58.855416 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=196 Ack=4866 Win=40544 Len=0 TSval=681665508 TSecr=751619528 4823 17:27:58.857719 CLIENT_IP F5_VS_IP TLSv1.2 543 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 4824 17:27:58.857731 F5_VS_IP CLIENT_IP TCP 185 443 → 36562 [ACK] Seq=4866 Ack=554 Win=4933 Len=0 TSval=751619593 TSecr=681665510 4825 17:27:58.859778 F5_VS_IP CLIENT_IP TLSv1.2 276 Change Cipher Spec, Encrypted Handshake Message 4826 17:27:58.923739 CLIENT_IP F5_VS_IP TLSv1.2 670 Application Data 4827 17:27:58.923793 F5_VS_IP CLIENT_IP TCP 185 443 → 36562 [ACK] Seq=4957 Ack=1039 Win=5418 Len=0 TSval=751619659 TSecr=681665576 4828 17:27:58.923981 F5_FLOAT_IP SERVER_IP TCP 193 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751619659 TSecr=0 SACK_PERM=1 4829 17:27:59.923626 F5_FLOAT_IP SERVER_IP TCP 193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751620659 TSecr=0 SACK_PERM=1 4830 17:27:59.923874 SERVER_IP F5_FLOAT_IP TCP 173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=1 Ack=993763571 Win=29845 Len=0 4831 17:27:59.923882 F5_FLOAT_IP SERVER_IP TCP 209 1360 → 8080 [RST] Seq=993763571 Win=0 Len=0 4832 17:28:00.923733 F5_FLOAT_IP SERVER_IP TCP 193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751621659 TSecr=0 SACK_PERM=1 4833 17:28:01.923650 F5_FLOAT_IP SERVER_IP TCP 181 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 SACK_PERM=1 4834 17:28:01.923822 SERVER_IP F5_FLOAT_IP TCP 173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=2408178215 Ack=1538671403 Win=30282 Len=0 4835 17:28:01.923845 F5_FLOAT_IP SERVER_IP TCP 209 1360 → 8080 [RST] Seq=1538671403 Win=0 Len=0 4836 17:28:02.923550 F5_VS_IP CLIENT_IP TCP 204 443 → 36562 [RST,ACK] Seq=4957 Ack=1039 Win=0 Len=0 4837 17:28:02.923561 F5_FLOAT_IP SERVER_IP TCP 204 [TCP ACKed unseen segment] 1360 → 8080 [RST, ACK] Seq=1 Ack=591246314 Win=0 Len=0 F5 tcpdump sees following (this is for different case): F5_FLOAT_IP.27216 > SERVER_IP.8080: Flags [R], cksum 0x95b2 (incorrect -> 0x0a17), seq 3911139265, win 0, length 0 out slot1/tmm10 lis=/Common/https_production flowtype=128 flowid=5618A9EEBE00 peerid=56189FD35F00 conflags=4000024 inslot=2 inport=9 haunit=1 priority=3 rst_cause="[0x2b07e6a:2314] TCP 3WHS rejected (bad ACK)" peerremote=00000000:00000000:X:X peerlocal=00000000:00000000:X:X remoteport=59656 localport=443 proto=6 vlan=4093 It is hitting constantly, and quite a lot. As per "K13223" this represent "The BIG-IP system failed to establish a TCP connection with the host (client or server) due to a failure during the TCP 3-way handshake process." In my case it is communication between F5 and server pool (all nodes affected). There is no firewall between F5 and server pool(s). It is happening with both AutoMap and SNAT. Are there any guides/cases how to debug this issue further? Mine test shows that it's not connected with client type (browser, curl, ...) or URL (same URL works in 99 percent of cases, that 1% is what's bothering me). Thank you!
FTP Proxy , user based forwarding with iRule
Im looking into using a VS as an FTP Proxy towards my backend webserver. Currently i already works for one webserver behind the VS. Im now trying to write an iRule to forward traffic to the right server based on the username the ftp-connection is initiated with. Secondly i need to rewrite the username and remove the web-paramater Example: USER1 connects to with following credential USER1@WEB1 @WEB1 is removed from the credential and forwarded towards pool WEB1 Anybody any ideas how to do this last part ?
CLIENT_ACCEPTED, SERVER_ACCEPTED and TCP::collect
Hi, That is probably obvious but I can't really figure out how TCP::collect is working depending on where it was enabled. When CLIENT_ACCEPTED is triggered it means three-way handshake was finished and we are waiting for packet from client. Now when TCP::collect is enabled what is collected: Only packets from client to VS (on clientside of the connection) Both packets from client and to client (send from VS to client) Same for SERVER_ACCEPTED (triggered after three-way handshake between VS and pool member is finished) Only packets from VS to pool member (on serverside of the connection) Both packets from VS to pool member and from pool member to VS What happens when iRule is like that: when CLIENT_ACCEPTED { set ehlo 0 SSL::disable } when SERVER_CONNECTED { TCP::collect } when CLIENT_DATA { set lcpayload [string tolower [TCP::payload]] if { $lcpayload starts_with "ehlo" } { set ehlo 1 serverside { TCP::collect } TCP::release TCP::collect } elseif { $lcpayload starts_with "starttls" } { TCP::respond "220 Ready to start TLS\r\n" TCP::payload replace 0 [TCP::payload length] "" TCP::release SSL::enable } else { TCP::release } } when SERVER_DATA { if { $ehlo == 1 and not([string tolower [TCP::payload]] contains "starttls") } { TCP::payload replace 0 0 "250-STARTTLS\r\n" } TCP::release clientside { TCP::collect } } No TCP:collect in CLIENT_ACCEPTED but instead clientside { TCP::collect } in SERVER_ACCEPTED. What is exact flow here (comparing to TCP::collect triggered in CLIENT_ACCEPTED)? What will be flow in case serverside { TCP::collect } is used? Is TCP::release (in SERVER_DATA) used to release packet from VS to pool member or as well from client to VS? My assumption for above is like that: Packet from client to VIP is not placed in the buffer - no TCP:collect in CLIENT_ACCEPTED This packet (first packet after three-way handshake on clientside) is triggering three-way handshake on serverside (between VS and pool member) Then TCP::collect is triggered so packet from VS to pool member is placed into buffer When full packet is placed in buffer SERVER_DATA is triggered In SERVER_DATA clientside { TCP::collect} is triggered - so is that mean that second packet from client to VIP is placed into buffer? After collecting full packet CLIENT_DATA is triggered for packet between VS and pool member (serverside { TCP::collect } Collected clientside packet is released to the serverside (TCP::release) Collecting of following clientside packet is enabled (TCP::collect) so next clientside packet will be collected So in short this code is just skipping first clientside packet and starts collecting clientside packets from second packet or it means first packet from client to pool member is delivered without collecting and first packet from pool member to client is placed into buffer? Or I completely misunderstand it? Piotr
