An Irule that Stops TLS (tcp) Fragmentation on Client Side
There is a requirement to implement a service that extracts Inserted traffic in extension header from subscribers. the insertion is done by PGW on packet core, and works on server side, which can successfully extract the information from the extension header. When the PGW Succesfully inserts the added extension, the solution on serverside works well. The challenge is that the PGW frequently truncates the TLS Header (TCP), and the serverside (F5), is unable to see the added extension header, since it has been cut off. A technical explanation is goes thus. "The EPG is configured for Heuristic TLS Enrichment, its Deep Packet Inspection (DPI) engine scans incoming TCP payloads on port 443. The EPG searches for the TLS ClientHello record header. It calculates the extension array offsets, dynamically shifts the remaining TLS bytes downstream, and injects Extension 17516 contains 4 bytes of TLS headers and 16 bytes of data). EPG platforms enforce strict MTU boundaries. If a subscriber's initial ClientHello packet is already 1,445 bytes, adding the 20-byte extension pushes it to 1,465 bytes. The EPG's internal network stack will instantly transmit the first 1,460 bytes and create a second TCP fragment for the remaining 5 bytes." Is there an irule extension that can detect and stop this fragmentation of TCP/TLS Data on the Client side, to ensure the TLS CLIENTSSL_HELLO is never fragmented.
Packet based load balancing instead of connection based (default)
Hi everyone, I have a requirement to load balance iso 8583 echo messages across two servers in a pool. I used a performance Layer 4 virtual server to attempt achieving this because I reckon that is the type of virtual server that could fulfill the requirements of load balancing all the requests across the 2 servers in the pool. However, requests are only being sent to one pool member. I also tried to craft an iRule (see below) to do this, still requests are only sent to one pool member. when CLIENT_ACCEPTED { log local0. "ACCEPTED !!" TCP::collect } when CLIENT_DATA { log local0. "DATA !!" #to get the length of messagein hexadecimal,the length info can be get from the first 2 byte binary scan [TCP::payload] H4 len log local0. $len #convertlentodecimal scan $len %x len log local0. $len #totalmessage length is length + 2 set len [expr { $len + 2} ] if {[TCP::payload length] < $len} { TCP::collect [expr {$len - [TCP::payload length]}] return } TCP::release $len TCP::notify request TCP::collect } when LB_SELECTED { log local0. [LB::server] } when SERVER_CONNECTED { log local0. "Server Connected !!" TCP::collect } when SERVER_DATA { log local0. "response: [TCP::payload]" TCP::release TCP::notify response TCP::collect } Has anyone done packet-based load balancing before? Any ideas?
BigIP exhibiting TCP zero Window behavior and closing connection
We have a VIP on our BigIP that sees a lot of connections. The TCP profile applied to the VIP is TCP Progressive with no Nagle. What we are seeing is after a very small amount of data the Window size on the F5 Self IP to Server connection is going to zero. The back-end server is able to maintain its buffer levels. The intention of applying a progressive profile was to take care of varying conditions and connections. We saw over 600 TCP Zero Window instance from the F5 to the server. all being sent by the F5. I am not sure what tweaks to the TCP profile i need to do to solve this problem.LTM :: Zero Window Server Side :: TCP Profiles
We have a virtual server setup for our receiving mail system, and it has been configured as-is for quite some time (measured in years). Never before has an issue arisen, but recently a particular client has been having problems sending attachments to us (and as far as we are aware, ONLY that client). What they claim to see is that the connection is terminated. Normal email works fine. Small file attachments work fine. However when they send us attachments that are Mb in size, the connection will not be successful. On our side, we see the window size slowly creep down until it hits zero. The BIG-IP probes the mail system, the mail system acks the probe, but keeps the window size at zero. It does this until the zero window timeout is reached on the BIG-IP and the connection is terminated by the BIG-IP (TCP RST). This is what the window decrease looks like on the client side (tcp.stream eq 3 and ip.src eq [the mail system]): This is what the window decrease looks like on the server side (tcp.stream eq 2 and ip.src eq [the VIP]): Client side end of the connection: Server side end of the connection: My impression initially was that this is not a BIG-IP problem... but when we remove the BIG-IP from the path, the connection works fine regardless of attachment size. Again, works fine for everyone else as far as we know regardless of if the BIG-IP is in the path... which is perplexing. Things I've tried: * Switching-out TCP profiles (lan optimized, wan optimized, client and server matching and different in combinations of the above). Now on mptcp-mobile-optimized with defaults. * Moving TLS off of the F5 * Resetting TLS profile to defaults * Different mail systems (of same type/configuration) Current configuration: * VIP on port 25 * TCP profile with mptcp-mobile-optimized w/defaults * SSL Profile (defaults w/cert, optional SSL, allowed cipher suites) * SMTPS Profile (allows TLS) * Pool w/single mail system * iRule w/VIP bounceback * Source IP Persistence VIP bounceback iRule: when LB_SELECTED { if {[IP::addr "[IP::client_addr]/24" equals "[LB::server addr]/24"]} { snat automap } else { snat none } } Any ideas/thoughts/suggestions all welcome. Thanks for taking the time.
iRule to close an established connection
I have tcp (not http) based service where client connections are permanent. By that I mean that once a connection to a pool member gets established its stays there 24x7. The pool has 2 pool members configured with priority group. The first pool member has priority 2 and the second one priority 1, with a Less than '1' value for Priority Group Activation. The pool also has the setting of 'Reject' for 'Action On Service Down'. That takes care of any scenario where a pool member is marked down by health monitors. Whenever a highest priority pool is marked down by health monitors all established connections to that pool members get closed automatically. The client applications immediately try to reconnect and get established connections to the second pool member with the lower priority. So far, everything is exactly what we want to accomplish. The challenge comes when the higher priority pool member is marked up/available once again. We're looking for an automatic way to close the already established connections to the lower priority pool member as soon as the higher priority pool member becomes available. Is there a way to do so? Not sure what event I should use for an already established connection. First ones that came to mind were LB_SELECTED and CLIENT_ACCEPTED. So far, I've tried the following options without any results: when LB_SELECTED { if { [LB::status pool poolname member 10.0.0.1 80 ] equals "up" and [IP::addr [LB::server addr] equals 10.0.0.2] } { reject } } when LB_SELECTED { if { [LB::status pool poolname member 10.0.0.1 80 ] equals "up" and [IP::addr [LB::server addr] equals 10.0.0.2] } { LB::reselect pool poolname member 10.0.0.1 } }
TCP Rewrite Rule used in Syslog TCP
Hi Dev/Central community! I've a SIEM with two syslog/tcp recievers (Let's name it R1 and R2). I 've created a VS to listen a 514/TCP, recieve the Syslog TCP message and send it to R1. In case R1 is down, the VS will send the Syslog TCP message to R2. As my SIEM assign a tag to each message recieved with the client IP, I need to rewrite the syslog message before send it to the R1 or R2 receivers (because I see the f5 self ip as client IP in the recievers). So, I've writed an iRule to rewrite the header of each syslog message before send it. this is my irule so far: when CLIENT_ACCEPTED { # Tomo la IP del cliente que se conecta al VS / Get the client IP connecting to the VS set ip_original [IP::remote_addr] # Tomo el Payload y la paso al siguiente nivel / Get the tcp payload to send it to Client Data TCP::collect log local0. "Client Accepted from $ip_original" } when CLIENT_DATA { set OrgininalTCPLength [TCP::payload length] # Primer <PRI> del payload / Try to detect <PRI> header in very first payload bytes regsub {^<\d+>} [TCP::payload] "\[\]\[\]\[$ip_original\]\[[clock seconds]\]\[\] " string # CRLF 0d0a \r\n + <PRI> / Look for another syslog message in the same TCP Payload regsub -all {\r\n<\d+>} $string "\r\n\[\]\[\]\[$ip_original\]\[[clock seconds]\]\[\] " string set len [TCP::payload length] TCP::payload replace 0 $len $string set ModifiedTCPLength [TCP::payload length] # Se pasa el Payload al siguiente nivel / Send the modified payload to the next level log local0. "Forwarindg message from $ip_original \t original length: $OrgininalTCPLength \t modified length: $ModifiedTCPLength" TCP::release #Preparo una nueva recoleccion / Get ready for a new collection TCP::collect } The iRule works like a charm, but in some very little times, it seem to doesn't rewrite the message... Any clue/ideas/troubleshooting tips? Regards, Maxmptcp-mobile-optimized and Hardware SYN Cookie Protection
Does anyone know why the TCP protocol profile mptcp-mobile-optimized ships with Hardware SYN Cookie Protection disabled? It is still enabled on tcp-mobile-optimized. Here is a copy of my two profiles, which should be the default: ltm profile tcp mptcp-mobile-optimized { abc disabled app-service none congestion-control illinois defaults-from tcp delay-window-control disabled delayed-acks disabled dsack disabled ecn enabled hardware-syn-cookie disabled init-cwnd 16 limited-transmit enabled mptcp enabled nagle enabled pkt-loss-ignore-burst 0 pkt-loss-ignore-rate 0 proxy-buffer-high 131072 proxy-buffer-low 131072 rate-pace enabled receive-window-size 131072 reset-on-timeout disabled selective-acks enabled send-buffer-size 262144 slow-start enabled timestamps enabled } ltm profile tcp tcp-mobile-optimized { abc disabled app-service none congestion-control high-speed defaults-from tcp delay-window-control disabled delayed-acks disabled dsack disabled ecn enabled init-cwnd 16 limited-transmit enabled nagle enabled pkt-loss-ignore-burst 0 pkt-loss-ignore-rate 0 proxy-buffer-high 131072 proxy-buffer-low 131072 receive-window-size 131072 reset-on-timeout disabled selective-acks enabled send-buffer-size 131072 slow-start enabled timestamps enabled }LTM ignores tcp rst or fin flags and timouts connection
Posting on behalf of a member who is having trouble posting this: Hello, we have issue with one pool member. We have enabled https monitor for member. Monitor works fine and production traffic as well but we have noticed short service interruptions. After deeper investigation is seems that LTM ignores TCP packet with FIN or RST flag set and did not close TCP connection. Take a look for tcpdump from LTM: 14:58:32.285201 IP 10.211.200.243.18724 > 10.211.212.72.https: S 1154236661:1154236661(0) win 14600 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.299810 IP 10.211.212.72.https > 10.211.200.243.18724: S 2860122587:2860122587(0) ack 1154236662 win 14480 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.299961 IP 10.211.200.243.18724 > 10.211.212.72.https: . ack 1 win 115 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.300273 IP 10.211.200.243.18724 > 10.211.212.72.https: P 1:130(129) ack 1 win 115 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317345 IP 10.211.212.72.https > 10.211.200.243.18724: P 1:167(166) ack 130 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317487 IP 10.211.200.243.18724 > 10.211.212.72.https: . ack 167 win 123 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317895 IP 10.211.200.243.18724 > 10.211.212.72.https: P 130:205(75) ack 167 win 123 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.317976 IP 10.211.200.243.18724 > 10.211.212.72.https: P 205:322(117) ack 167 win 123 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.333454 IP 10.211.212.72.https > 10.211.200.243.18724: . ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360053 IP 10.211.212.72.https > 10.211.200.243.18724: P 167:332(165) ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360329 IP 10.211.200.243.18724 > 10.211.212.72.https: P 322:375(53) ack 332 win 131 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360408 IP 10.211.212.72.https > 10.211.200.243.18724: P 332:385(53) ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.360618 IP 10.211.212.72.https > 10.211.200.243.18724: F 385:385(0) ack 322 win 243 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 14:58:32.374770 IP 10.211.212.72.https > 10.211.200.243.18724: R 2860122919:2860122919(0) win 0 in slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=626 inslot=0 inport=0 haunit=0 priority=0 15:03:33.381555 IP 10.211.200.243.18724 > 10.211.212.72.https: R 375:375(0) ack 386 win 0 out slot1/tmm0 lis= flowtype=130 flowid=5700012E5F00 peerid=5700012E6500 conflags=62E inslot=0 inport=0 haunit=0 priority=0 rst_cause="[0x1eb69cd:625] {peer} Flow expired (sweeper) (idle timeout) Any idea what is going on? In the same pool we have another pool member but with different monitor and it works fine.Huge number of TCP 3WHS rejected (bad ACK), chksum incorrect
Hi guys, Hope you can help me with this, for me, complete mystery. I'm getting lots of following: Wireshark text export from F5 tcpdump: 4815 17:27:58.597830 CLIENT_IP F5_VS_IP TCP 162 36562 → 443 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=681665250 TSecr=0 WS=128 4816 17:27:58.597846 F5_VS_IP CLIENT_IP TCP 193 443 → 36562 [SYN,ACK] Seq=0 Ack=1 Win=4380 Len=0 MSS=1460 TSval=751619333 TSecr=681665250 SACK_PERM=1 4817 17:27:58.660439 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=1 Ack=1 Win=29200 Len=0 TSval=681665313 TSecr=751619333 4818 17:27:58.730179 CLIENT_IP F5_VS_IP TLSv1.2 380 Client Hello 4819 17:27:58.730201 F5_VS_IP CLIENT_IP TLSv1.2 4529 Server Hello 4820 17:27:58.792837 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=196 Ack=4345 Win=37648 Len=0 TSval=681665445 TSecr=751619465 4821 17:27:58.792854 F5_VS_IP CLIENT_IP TLSv1.2 706 Certificate, Server Hello Done 4822 17:27:58.855416 CLIENT_IP F5_VS_IP TCP 185 36562 → 443 [ACK] Seq=196 Ack=4866 Win=40544 Len=0 TSval=681665508 TSecr=751619528 4823 17:27:58.857719 CLIENT_IP F5_VS_IP TLSv1.2 543 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 4824 17:27:58.857731 F5_VS_IP CLIENT_IP TCP 185 443 → 36562 [ACK] Seq=4866 Ack=554 Win=4933 Len=0 TSval=751619593 TSecr=681665510 4825 17:27:58.859778 F5_VS_IP CLIENT_IP TLSv1.2 276 Change Cipher Spec, Encrypted Handshake Message 4826 17:27:58.923739 CLIENT_IP F5_VS_IP TLSv1.2 670 Application Data 4827 17:27:58.923793 F5_VS_IP CLIENT_IP TCP 185 443 → 36562 [ACK] Seq=4957 Ack=1039 Win=5418 Len=0 TSval=751619659 TSecr=681665576 4828 17:27:58.923981 F5_FLOAT_IP SERVER_IP TCP 193 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751619659 TSecr=0 SACK_PERM=1 4829 17:27:59.923626 F5_FLOAT_IP SERVER_IP TCP 193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751620659 TSecr=0 SACK_PERM=1 4830 17:27:59.923874 SERVER_IP F5_FLOAT_IP TCP 173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=1 Ack=993763571 Win=29845 Len=0 4831 17:27:59.923882 F5_FLOAT_IP SERVER_IP TCP 209 1360 → 8080 [RST] Seq=993763571 Win=0 Len=0 4832 17:28:00.923733 F5_FLOAT_IP SERVER_IP TCP 193 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 TSval=751621659 TSecr=0 SACK_PERM=1 4833 17:28:01.923650 F5_FLOAT_IP SERVER_IP TCP 181 [TCP Retransmission] 1360 → 8080 [SYN] Seq=0 Win=4380 Len=0 MSS=1460 SACK_PERM=1 4834 17:28:01.923822 SERVER_IP F5_FLOAT_IP TCP 173 [TCP ACKed unseen segment] 8080 → 1360 [ACK] Seq=2408178215 Ack=1538671403 Win=30282 Len=0 4835 17:28:01.923845 F5_FLOAT_IP SERVER_IP TCP 209 1360 → 8080 [RST] Seq=1538671403 Win=0 Len=0 4836 17:28:02.923550 F5_VS_IP CLIENT_IP TCP 204 443 → 36562 [RST,ACK] Seq=4957 Ack=1039 Win=0 Len=0 4837 17:28:02.923561 F5_FLOAT_IP SERVER_IP TCP 204 [TCP ACKed unseen segment] 1360 → 8080 [RST, ACK] Seq=1 Ack=591246314 Win=0 Len=0 F5 tcpdump sees following (this is for different case): F5_FLOAT_IP.27216 > SERVER_IP.8080: Flags [R], cksum 0x95b2 (incorrect -> 0x0a17), seq 3911139265, win 0, length 0 out slot1/tmm10 lis=/Common/https_production flowtype=128 flowid=5618A9EEBE00 peerid=56189FD35F00 conflags=4000024 inslot=2 inport=9 haunit=1 priority=3 rst_cause="[0x2b07e6a:2314] TCP 3WHS rejected (bad ACK)" peerremote=00000000:00000000:X:X peerlocal=00000000:00000000:X:X remoteport=59656 localport=443 proto=6 vlan=4093 It is hitting constantly, and quite a lot. As per "K13223" this represent "The BIG-IP system failed to establish a TCP connection with the host (client or server) due to a failure during the TCP 3-way handshake process." In my case it is communication between F5 and server pool (all nodes affected). There is no firewall between F5 and server pool(s). It is happening with both AutoMap and SNAT. Are there any guides/cases how to debug this issue further? Mine test shows that it's not connected with client type (browser, curl, ...) or URL (same URL works in 99 percent of cases, that 1% is what's bothering me). Thank you!
