Faster Small Downloads: TCP Early Retransmit and Tail Loss Probe

I’d like to welcome you to a new blog about F5® TCP Express™ in BIG-IP® Local Traffic Manager™ (LTM). It will focus on new features and the product’s more subtle behaviors. Today I’ll discuss two improvements introduced in F5® TMOS® 11.6.0 that are meant to work together for small file transfers, or any application where the server sends out small amounts of data over TCP. They are Early Retransmit (ER) and Tail Loss Probe (TLP), two separate features meant to work together to recover from losses faster. Although they’re not on by default, many users would improve performance by enabling them in their TCP profiles.

What it does

For connections with very little data sent, a single packet loss results in costly timeouts, a significant percentage of the total transfer time spent idle. ER and TLP can save on the order of hundreds of milliseconds for file downloads. For connections with multiple requests and responses, the savings can be several times that.

To see the benefit, download lots of small files (on the order of 3-5 packets) in an environment that often loses a single packet from those transfers. In the aggregate, you should notice a decrease in mean transfer time with these two features enabled.

How it Works

When a TCP receiver detects a gap in received sequence numbers, it issues a duplicate acknowledgment (dupack). TCP senders normally assume a packet is lost if they receive three consecutive dupacks, meaning that at least four packets are outstanding (one lost, and three to trigger the dupacks).

ER covers the case where there is not enough outstanding data to receive three dupacks, because of a small advertised receive window or lack of data offered by the application. Traditional TCP senders wait for the retransmit timer to expire, well above the connection Round Trip Time (RTT) and usually on the order of at least hundreds of milliseconds. During this time the connection is idle.  Furthermore, timeouts result in a much more aggressive rollback of sending rate compared to losses detected by dupacks.

ER looks at the transmit queue and computes the maximum number of dupacks it could generate given one packet loss. If a sender has received that many dupacks, it retransmits the putatively lost packet as if three dupacks had arrived, allowing data transfer to resume earlier.

If the last data packet is lost, and there are no other gaps in arriving packets, there will be no dupacks and ER can’t help. That’s where TLP comes in. TLP adds a second retransmission timer, considerably faster than the conventional one. Loss of the last sent data packet causes a TLP timeout. TLP then transmits the last sent packet.

If only the last packet is lost, that resolves the problem immediately. Otherwise, if there is additional application data, the resulting dupack allows “limited transmit recovery” of new data (if enabled in the profile, as it is by default). The table below summarizes what can happen.

Burst Loss Size (packets)	TLP Outcome (No more application data)	TLP Outcome (More application data)
1	Loss recovered	Loss recovered
2	Early Retransmit, loss recovered	New data sent, triggering more dupacks
3	Dupack received, but enough for retransmission	New data sent, triggering more dupacks
4+	“Forward ACK (FACK)”: SACK arrives with a gap three packets long, triggering fast retransmit	New data sent, triggering more dupacks

In only one case above does TLP not allow faster recovery from loss. In all cases, the TLP timeout doesn’t trigger the usual congestion response: if there is a packet loss, the fast retransmit mechanism will adjust the congestion parameters appropriately.

Note that our TLP works a bit differently than the latest (expired) internet-draft specifies. As this is a sender-side only enhancement, there is no risk of interoperability problems due to incomplete compliance.

Deployment Pointers

Hotfix Rollup 5. We added some improvements to ER in TMOS 11.6.0 Hotfix 5, so if you’re running an earlier iteration of 11.6 we strongly recommend an upgrade to get the most out of this feature. In any case, F5 recommends obtaining the most recent hotfix for the latest security updates.

SACK enabled. TLP only works when the connection endpoints have negotiated Selective Acknowledgments (SACK).

Packet reordering. The reason TCP waits for 3 dupacks is to filter out dupacks that result from mild packet reordering. In environments that experience a lot of such reordering, due to frequent routing changes or link-layer ARQ schemes, ER can result in spurious retransmissions that waste bandwidth and unnecessarily reduce the send rate. To identify your network’s tendency to reorder, examine a packet capture for cases where the data receiver sends a SACK, but then acknowledges the “missing” data before the sender has a chance to retransmit. If this happens a lot, ER is a poor choice for your network.

If you have any questions, or recommendations for future TCP-related topics, please say so in the comments. Thanks!