BIG-IP 17.1.2 FastL4 unexpectedly sending TCP RST mid-session

Hi everyone,

I am troubleshooting unexpected TCP RST packets physically sent by a BIG-IP 17.1.2 Performance L4 virtual server using the default FastL4 profile.

The virtual server performs SNAT and forwards traffic through a single next-hop pool member (the default gateway), so there is no load balancing across multiple servers. There are no traffic policies, iRules, AFM, ASM, or HA/mirroring configured.

TCP connections establish successfully, complete TLS, and exchange application data. After that, BIG-IP sends TCP RST packets and terminates the established session. Packet captures suggest that the resets do not appear to be caused by the configured idle timeout.

Has anyone seen similar behavior with FastL4 on TMOS 17.1.2, or knows of a configuration, FastL4/TMM behavior, or known issue that could generate RST packets mid-session?

Thanks in advance.

Additional information: the issue is intermittent and occurs randomly throughout the day. Most connections establish and complete normally, while only some established TLS sessions are unexpectedly reset by BIG-IP after exchanging application traffic. This is not consistently reproducible for every connection.

Thanks in advance.

Hi jcbr2002,

A couple of things to nail down before diving into a capture, and then the capture itself as the key diagnostic step.

1. Confirm the platform first

Before deciding whether hardware acceleration is even a factor: is this BIG-IP running on a physical appliance (with an ePVA/PVA ASIC - i5000/i7000/i10000/i11000/i15000 series, VIPRION blades, VELOS/rSeries) or on a Virtual Edition (VE)? VE has no ePVA chip at all, so if you’re on VE the whole PVA/ePVA acceleration angle can be ruled out immediately and we go straight to a software-path capture. If it’s physical hardware, it would also help to know what PVA Acceleration mode is set on the FastL4 profile (“show ltm virtual <name>” reports the live acceleration status per connection: None / Partial (Assisted) / Dedicated).

2. Take the capture with the nnnp modifier - this is the key step

Since the profile is default FastL4 with no SSL profiles involved (Performance L4 is pure L4 forwarding, so BIG-IP never terminates the TLS session), there’s no master secret or client-ssl/server-ssl handshake to capture - decryption doesn’t apply here and isn’t needed to diagnose an L4-level RST anyway. The reset is a transport-layer decision made by TMM, and F5’s internal tcpdump extension already exposes the exact reason inline:

tcpdump -ni 0.0:nnnp -s0 -w /var/tmp/capture.pcap host <client-ip>

0.0 captures across all VLANs at the TMM forwarding plane. nnn pulls the maximum noise level (low+medium+high internal TMM detail), which - since BIG-IP 11.2.0 - includes the Reset Cause field directly in the packet trailer. p follows the full end-to-end flow (client-side and server-side legs) even with SNAT in play, so you get both sides of the connection in one capture.

Since the RST happens mid-session (not at setup), start this capture before reproducing the failure and let it run through the exact moment the reset occurs, so that specific event’s Reset Cause lands in the trailer.

Once you have the pcap, open it in Wireshark, enable the f5ethtrailer dissector (Analyze > Enabled Protocols > f5ethtrailer), and inspect the Ethernet II trailer on the RST packet for the Reset Cause field, along with VIP name, flow id/peer id, and which TMM/slot handled it.

3. Optional cross-check via logs

If you also want the cause logged in plain text (handy if pulling the exact packet is inconvenient), temporarily enable:

tmsh modify sys db tm.rstcause.log value enable
tmsh modify sys db tm.rstcause.pkt value enable

then check /var/log/ltm and “tmsh show /net rst-cause” after reproducing. Disable both afterward - tm.rstcause.pkt embeds the cause string in the RST payload itself, which you don’t want left on in production.

4. Once you have the Reset Cause

Common causes worth ruling out given your setup (no idle timeout involved): “TCP RST from remote system” (BIG-IP is just relaying a reset that actually originated at the client or the next-hop/pool member - quite possible in a SNAT + single-gateway topology), “TCP retransmit timeout”, “TCP zero window timeout”, or “No flow found for ACK”. If the cause points at the hardware path and this turns out to be physical hardware with PVA Acceleration enabled, that’s when it’s worth temporarily setting PVA Acceleration to None on the FastL4 profile to force everything through software and see if the resets stop - that isolates whether ePVA offload is involved, but I’d treat it as a confirmation step after you have a cause, not a first move, since it has a performance cost.

Share the Reset Cause value (and platform type) once you have it and we can narrow this down further.

jcbr2002,

FastL4 bypasses the TCP proxy.  The BIG-IP is not doing any thing with the TCP segments.  The BIG-IP is acting as NATing IP forwarder. The TCP resets are very likely being sent by one of the TCP connection end points.

You can take a filtered packet capture to determine which side of the TCP connection is sending the TCP reset.

The TCP reset logging may not log any thing. I believe that the logging will only work when the TMM TCP proxy is used.

Why is a TCP reset being sent? Why is it intermittent and occurs randomly?

My first guest would be due to congestion.  One side of the TCP connection at random time throughout the day runs out of TCP resources. It’s TCP buffers may be come full. It may not be able to process it’s TCP buffers in a timely manner.

Is there any thing between the end points other than the BIP-IG that could be causing TCP congestion?

Have you done a packet capture of the TCP sessions and reviewed the capture for other TCP issues?

TCP reset can be a symptom or bigger TCP of network or server issues.

-------------------

Some articles about packet captures.

When you just want to view a summary of the packet headers for do not need the write to file argument, -w,  to tcpdump.

tcpdump -ni 0.0:nnnp  host <client-ip>

K411: Overview of packet tracing with the tcpdump utility

K13637: Capturing internal TMM information with tcpdump

Just a slight note of clarification…the proxy is not bypassed, but access is limited in scope. You can still use iRules against a FastL4 profile for primitive use cases, though typically not recommended unless the conditions met are a significantly small percentage of the traffic.

JRahm,

Thank you for the clarification.

FastL4 lets you play with the HTTP headers.

K16446: The BIG-IP system now allows a Performance (Layer 4) virtual server to have an associated HTTP profile

Ok, thanks to Quiroman81 and everyone else for your help.

Update: This issue affects a single Performance L4 virtual server.

For troubleshooting, we created a custom FastL4 profile with ePVA acceleration disabled so we could capture the traffic. We used:

tcpdump -vi 0.0:nnnp -s0 -w /var/tmp/captura.pcap

In the captured BIG-IP-generated RSTs, the F5 Reset Cause was Flow expired (sweeper) (bias oldest). This indicates that the connection sweeper terminated those flows; the captures did not point to FastL4 idle-timeout expiration.
1394

We also observed peaks of approximately 2 million active connections, 70% memory utilization, and 70% CPU utilization. About a week ago, we enabled Eviction Protected on this virtual server. Since then, similar peaks have occurred, lasting less than 10 minutes, but we have not observed further BIG-IP-generated RSTs in the captures reviewed. The application-side errors have also stopped.

Based on the Reset Cause and the behavior observed after enabling Eviction Protected, connection eviction appears to have been responsible for the RSTs we investigated. We are continuing to monitor the virtual server.

F5 BIG-IP 17.1.2 Tenant 32 gb ram, 8 cpu cores

thank all