Explore TCP and TLS Profiles for Optimal S3 with MinIO Clusters

A lab-based investigation was conducted to observe measurable performance differences when using different profiles, including TCP and TLS settings, in both local and simulated wide-area implementations. Testing at a high-traffic scale was not carried out. That may be something to look for in the future. Rather, simply observing the nuances of TCP and TLS in support of modern S3 flows, think AI data delivery for model training exercises, led to interesting strategic findings. Top of mind throughout the exercise were new available configurations of both TCP and TLS BIG-IP profiles, with the express interest in improving the speed and resilience of S3 data flows. Suggested tweaks to the rich set of TCP and TLS parameters will be touched upon.

 

Lab Setup

A fully software-based configuration was created using virtualization and a modern hypervisor, where S3 clients were routed to tunable BIG-IP virtual servers, which proxied S3 traffic to a virtualized AIStor single-node single-drive object solution. The routing of client traffic was chosen so as to simulate first a high-speed local area network (LAN) experience, followed by, second, a wide area network (WAN) examination. The router offered various network impairments, simulating cross-North American latency, shaping of traffic to a reasonable but constrained maximum bandwidth, and the introduction of packet loss throughout the S3 activities.

The investigation was primarily around changes in performance for a representative S3 transaction, such as a 10-megabyte object retrieval by a client, as different profiles in BIG-IP were exercised.

 

LAN Baseline for S3 Traffic and BIG-IP

To establish what our lab setup is capable of delivering, some standard S3 transactions were carried out. This started without a simulated WAN. S3 was put through its paces by both downloading and uploading of objects using both Microsoft Edge and the S3Browser clients.

The LAN setup, instantiated on an ESXi host, is designed to be highly responsive. As seen by quick measurements of ICMP latency between a client and a BIG-IP VE virtual server, the round-trip time is sub-1 millisecond. The first Ping slightly exceeds 1 millisecond due to a client-side ARP request/response between client and router, and subsequent responses settle to approximately 400 microseconds. The measurements were taken with Wireshark on the S3 client running on Windows. The command prompt results do not have the fidelity of Wireshark and simply report latencies of sub-1 milliseconds for the bulk of response times. With performance protected by limiting Wireshark capture to the first 128-bytes of all packets, we use the Wireshark delta times highlighted to confirm more accurately the low round-trip time (RTT) latency.

To support local on-premises S3 clients, the virtual server at 10.150.92.202 was configured to use one of the BIG-IP available LAN-oriented TCP profiles “tcp-lan-optimized”, available with all fresh BIG-IP installs. As pointed out in the diagram, the S3 traffic will also benefit from SSL/TLS security between the client and the BIG-IP.

The TCP profile “tcp-lan-optimized” exhibits settings that are beneficial for low-latency TCP sessions only traversing LANs, where intuitively low packet loss and high throughput are both very likely. Just some of the characteristics include aggressive (“high speed”) congestion control, smaller initial congestion windows (suited for lower RTT), and disabling of Nagle’s algorithm such that data is sent immediately, even if not filling out a TCP maximum segment size (MSS).

There are a breadth of LAN profiles also available with their own tweaks, such as “f5-tcp-lan”, the selected profile "tcp-lan-optimized" was simply chosen as a starting point.

With MinIO AIStor configured with buckets allowing public access, meaning no S3 access key is necessary, simple browsing of buckets with something like Microsoft Edge and its Developer Tools can give an estimate of S3 retrieval times for a sample 10-megabyte object.

 

Determining Throughput for LAN Delivered S3 Objects

We noted that Edge Developer Tools, with caching disabled, suggested our 10-megabyte object required between 155 and 259 milliseconds to download. Some variances in time can be expected as an early test might require full TCP 3-way handshakes on port 9000 (the MinIO S3 port used) and a full TLS negotiation. Later downloads can benefit from features like TLS resumption.

To drill deeper, and to estimate the actual traffic rate provided in the LAN only setup, in terms of bits per second, one can again turn to Wireshark. It is important to restrict capture to only the necessary bytes to decode TCP, as it’s the study of progressive TCP sequence numbers and issues like TCP retransmits that lead to an estimate of S3 transfer rate. Trying to capture all the payload on a consumer Windows virtual machine will not see all the packets stored. It is also good practice to filter out capture of extraneous traffic, such as Remote Desktop Protocol (RDP), which often operates on TCP and UDP ports 3389.

Three successive downloads from the BIG-IP, serving the MinIO AIStor solution in the backend, appear as follows when using the TCP Stream Graphs, specifically the Sequence Number (Stevens) plot, where one clearly sees the rapid rise in sequence numbers as the 3 downloads complete at high speed.

Interestingly, zooming in upon any one of the downloads, one notes there is room for improvement, which is simply to say we could have driven S3 faster with client adjustments. The Windows client periodically sends TCP “zero window” advertisements to BIG-IP, essentially halting S3 delivery for some number of milliseconds while buffers on the client are serviced.

A quick filter on the client’s address and zero window events can show the activity “on the wire” during these moments (double-click to enlarge).

We see that the Windows S3 client periodically shuts down the delivery of TCP segments by reducing its TCP receive (Rx) window size to zero, after which it can be seen to take 15.5 milliseconds to re-open the window. This is not critical but means our measured bit rate for a simple set of basic transactions, even in a virtualized environment, could easily be increased with more performant clients in use. The objective is not to do benchmarking but rather a comparison of TCP profiles (double click for high resolution).

As seen above, the typical S3 10-megabyte object, even with a client throttling delivery periodically, was still in the 85 Mbps range. This with the TCP-LAN-Optimized profile in use.

The last baseline measurement undertaken was to push S3 data (technically an HTTP PUT command) from the client to the MinIO object storage, via the BIG-IP virtual server. The value here was to measure the TCP response times observed by the client as it moved TCP segments to BIG-IP. To this end, S3Browser, available here, was used as it fully supports S3 user access keys and their corresponding secrets, and allows a user a graphical “File Explorer” experience to manipulate S3 objects. Uploading a 20-kilobyte object from our local client, we see the following response times in Wireshark (double-click for detailed image).

Note, response times with something like ICMP Ping Echo Request/Reply are simple, each transaction provides an estimate of round-trip time. With TCP and data in flight, most TCP stacks use a delayed ACK approach where two times MSS (often meaning two times 1,460 bytes) are waited for, or a short timeout in case no additional data arrives, before the data is acknowledged. With large bursts of traffic, such as the upload of a sizable object with S3, the response time is likely not to deviate a lot as delayed ACKs add negligible time.

We see in the chart that the data moved from client to BIG-IP, over the LAN, with typical TCP acknowledgment times in the 400 to 800 microsecond range.

 

Baseline S3 Performance Across WAN Using Different BIG-IP TCP Profiles

The router used in the lab, OPNSense, has an ability to emulate network impairments, including those consistent with the realities of traffic traversing long distances. Objectives for testing include:

• Base round-trip delays, in our case, 70 milliseconds, will be introduced as this is in line with best-case optical delays for round trips between New York City and Los Angeles.

• Shaping of traffic to emulate an achievable bandwidth of 10 Mbps, simulating normal limiting factors such as IP hops, queuing delays, and head-of-line blocking on oversubscribed intermediate network gear. The expectation is such a bandwidth cap will add variance to the overall round-trip delays of packets and occasional drops, forcing TCP into phases such as TCP slow start with congestion avoidance.

• Experimenting with packet loss, as anything below 1 percent packet loss generally is considered not altogether unexpected. A value of 0.5 percent packet loss will be set for both directions.

As with LAN TCP profiles, BIG-IP has a number of WAN options. The one to be contrasted with the existing LAN profile was selected as “tcp-wan-optimized”. The objectives of this profile include maximizing throughput and efficient recovery from packet loss over less performant WAN links. The expectation is to encounter higher latency and lower bandwidth end-to-end network experience. Note, on the inside of BIG-IP, where MinIO AIStor continues to be co-located at LAN speeds, the server side will continue to use the tcp-lan-optimized profile.

A rule of thumb is that packet loss in a WAN environment, even as little as 0.5 percent, will impede network quality of service more than latency. As such, we started with no packet loss, simply 70 milliseconds of round-trip latency and policing of bandwidth to a 10 Mbps maximum.

Continuing to use the lan-tcp-optimized profile on BIG-IP still saw decent results. As expected, the bandwidth now falls under 10 Mbps, with the measured value, over three downloads, appearing to hover just under 7 Mbps.

Using Edge’s developer tools, it shows, even with a profile optimized for LAN traffic, that total download times are fairly consistent, averaging just under 13 seconds.

The BIG-IP virtual server was then updated to use the tcp-wan-optimized profile, just for traffic involving the external, client-side. The TCP profile for the internal-side, co-located MinIO server was left with an LAN profile.

Using S3 from the client to retrieve the very same 10-megabyte objects, and the results were, to a degree, better.

The object download times were in the same order of magnitude as with the LAN profile, however, drilling into the actual bit rates achieved, one can see a marginal increase in overall S3 performance. The next step would be to introduce another real-world component; packet loss equally applied in both directions. Since both directions were subjected to loss, beyond the need to retransmit TCP segments in the direction of the client, TCP acknowledgments from the client to the BIG-IP can also be dropped. The resulting behavior will exercise the TCP congestion control mechanisms.

With even low packet loss in our simulated WAN environment, the outcome with the tcp-wan-optimized profile on BIG-IP was markedly better. As seen in the tables above, this is far from a scientific, in-depth analysis, as three 10-megabyte S3 retrievals is not a rigorous baseline. However, simply using these numbers above to guide us, we come to these findings:

• Average S3 download with WAN profile: 33.7 seconds

• Average S3 download with LAN profile: 40.2 seconds

• Percentage reduction in S3 transaction time using WAN profile: 16 percent

Comparing one sample S3 transaction using each of the two profiles, visually, we see modest differences that can help to explain the increased quality of service of the WAN profile.

For the purpose of a quick investigation, the WAN profile has been seen to offer benefits in a lossy, higher-latency environment such as the lab emulation. Two specific TCP features to call out, and are worth enabling in such environments are:

1. Selective ACKs (SACK option)

   Without SACK, a normal approach for a client to signal back to a sender that a TCP segment appears to have been lost in flight, is to send simple duplicate ACKs. This means for every newly received segment after the missing data, an ACK is sent but only acknowledging the last contiguously received data, there is no positive indication of the subsequent data received. Simply receiving duplicate ACKs can let the sender infer there is a gap in the data stream delivered.

   With SACK, there is no inferring, should both ends (e.g. the client and BIG-IP in our case) support this TCP option, agreed upon during the 3-way TCP handshake, then the edges of the discontinuity in data are clearly sent by the client.

    

   

    

   Most TCP profiles will have SACK enabled, but it is worth confirming it has not been disabled and is active with your key S3 clients, as seen in the following screenshot.

   

    

2. TCP Receive Window Exponential Scaling

    

   Original implementations of TCP only allowed for approximately 65,535 bytes to be in flight, without the sender having received acknowledgments of reception. This number can be a bottleneck with highly performant devices exchanging TCP over higher latency WAN pipes, networks with so-called high-bandwidth delay products (BDP). The workaround is for each end to advertise an exponent, for instance 2, in which case the peer device will understand that advertised receive windows are interpreted as four times (2^2) the encoded value. An exponent of 4 would indicate a 16-time multiplier (2^4), and so on.

   To enable this on a BIG-IP stack, per this KB article, we simply adjust the buffer size to the desired value in the profile setup. Without this adjustment, there will be no windows scaling used.

 

Introducing the New S3 TCP Profile for BIG-IP

The rise in S3 traffic has implications specific to networking and traffic in-flight. Some of the characteristics that are relevant include:

• There can be vast differences in network load, transaction by transaction. Consider an IoT device generating a 40-kilobyte sensor reading, followed immediately by a 500-megabyte high-definition medical ultrasound. Both are valid and sample payloads in S3 delivery today.

• S3, being transported by an HTTPS conduit, employs extensive parallelism for many types of larger transactions, for instance multi-part uploads or using HTTP ranges for large object downloads. Essentially, large transactions such as a 60-megabyte upload become, as an example, 12 smaller 5-megabyte writes. This parallelism is particularly advantageous with clustered HDD nodes, as spinning media still is estimated to provide 60 percent of all storage, yet the input/output (IOPS) rates of HDD technology frequently peaks at rates of only 100 to 150. As such, there is value around using many smaller, but parallel transactions.

• Solutions supporting S3 storage are focused upon strong read-upon-write consistency, meaning the correct versioning of objects must be served immediately after writing, frequently from any number of storage sites. As such, the immediate replication of asynchronously connected sites through S3 protocol, is something that must happen very quickly for an effective solution.

With version 21 and later of TMOS, the BIG-IP LTM module has provided a starting point for a S3 TCP profile, a profile with logical settings aligned with S3 network delivery. The s3-tcp profile is based upon the parent “tcp” profile of BIG-IP, with a number of tweaks now to be described.

With the s3-tcp profile, management of the receive window and send buffer is turned over to the system, which will monitor network behavior to auto-adjust. The one striking note are maximum values for items like the largest possible receive window to advertise are much bigger, moving from 65,535 bytes into the millions of bytes range. Similarly, send buffers are much larger, although the client side will ultimately have control over throttling the amount of sent traffic "in flight".

The other major difference is that two different congestion control algorithms are in use, CUBIC for the s3-tcp profile and high-speed for the standard tcp-profile.

Congestion control is the art of operating a TCP connection with the highest possible congestion window (CWND) over time, while minimizing segment loss due to saturated networking or peer-perceived loss due to fluctuations in delivery latency. TCP is designed to back off how many segments may be in flight, meaning the CWND, when loss is detected.

The two algorithms that dictate the ensuing behavior are slow start and congestion avoidance. TCP slow start aggressively opens the congestion window when starting a TCP connection, or during a connection when recovering from a segment loss.  Don't be fooled by the term slow, it's actually aggressive and fast off the mark!   The "slow" is from historical convention, so named because it is slow only in comparison to the original, 1980s TCP behavior of immediately transmitting a full window of data at wire speed.

Meanwhile, congestion avoidance will normally try to slowly, continually and usually linearly further open the congestion window once arriving at a fraction of the last fleetingly reached CWND, perhaps half of the last CWND before loss was detected. From there the congestion avoidance algorithm will inch upwards at a controlled rate trying to achieve the optimal end to end bandwidth before segment loss sets in once again.  Think of it as a constant fine-tuning exercise trying to maximize throughput while being wary of the network’s saturation point.

CUBIC stems from Linux implementations where high bandwidth, coupled with substantive network latency, are prevalent. CUBIC, as the name suggests, uses a cubic function to calculate congestion window growth based on time elapsed since the last packet loss.

The bonus of BIG-IP is that adjustments to TCP profiles are quite easy to achieve. Simply create a new profile, often based upon an existing “parent” profile and make “before” and “after” observations.

A good candidate for experimentation is the Westwood+ profile.  Westwood+ congestion control is a server-side only implementation that studies the return of TCP ACKs, to infer optimal transmission rates. A more basic TCP approach is to half the congestion window, a full 50 percent reduction of unacknowledged bytes in flight, when three duplicate ACKs are received. The presumption is that three are supporting evidence that a TCP segment has been lost over the network. Westwood+ has a more advanced approach to studying the ACKs to arrive at a less coarse value for a new congestion window.

 

New S3 TLS Profile for BIG-IP

Similar to the new s3-tcp profile, there is now an S3 TLS profile. The normal practice is to make a copy of this profile, to allow the setting of specific certificates, keys and certificate chains for the application in question. One of the most important aspects of the s3 profile is that it by default supports TLS 1.3. There are a number of aspects to 1.3 that make it a good option. One, it removes some of the antiquated ciphering and hashing algorithms of older TLS revisions. For another thing, TLS 1.3 is a mandatory baseline for supporting post-quantum computing (PQC) shared key establishment between peers. BIG-IP, using TLS 1.3 as a building block, supports NIST FIPS 203 – ML-KEM key encapsulation as per the standard found here.

A S3 immediate win around TLS 1.3 is the ability for entirely new TLS sessions, not resumptions of previously negotiated sessions, to start delivery of application data in one round-trip time (RTT). This is due to a simplification in the protocol exchange between peers in TLS 1.3.  Among other things, in non-mutual TLS sessions, only the server is required to provide a certificate, which is already encrypted when delivered. In other words, no later TLS 1.2-style quick validation testing of encryption is required; all necessary tasks required of the server, including if the client can successfully decipher the provided certificate, are achieved in one logical step.

Take the following example of a S3 object retrieval through an encrypted HTTP GET.  As seen in the diagram, there are two round-trip times and the latency, run across the wide-area network emulator used early, required 300 milliseconds from Client Hello to the first application data.

It is noted above that the time from the TLS Client Hello to the first transmission of encrypted data, carrying a S3 transaction, was 300 milliseconds. Contrast this with the same lab setup, now using a copy of the BIG-IP S3 TLS profile, which offers out-of-the box default advertising for TLS 1.3, something other TLS profiles typically require a custom adjustment to make happen in prior BIG-IP releases.

As with this overall exploration, the results are suggestive of a notable performance increase, but much higher-load testing should be considered more decisive evidence. What one can say, in this case of single S3 encrypted transactions, a 50 msec or 17 percent savings in latency was observed, with respect to getting a transaction on the wire. Extrapolating over thousands of TLS sessions and the savings, let alone the security-posture improvement of TLS 1.3, make it appear like a logical configuration choice.

 

Summary

The amount of attention now being paid to S3 data delivery, including delivery for AI initiatives like model training, is heightened across the industry. With network considerations top of mind, like the effect of both WAN latency and non-zero packet loss ratios, a lab oriented simple exploration of S3 performance between clients and MinIO AIStor was carried out.

New configurations of both TCP and TLS profiles, with the express interest in improving the speed and resilience of S3 data flows, were investigated. The impact of using existing LAN or WAN TCP profiles on BIG-IP was measured for small-scale, sampled lab tests. The outcomes suggested that although a LAN profile performed well with significant latency applied, the WAN oriented profiles were demonstrably better in environments with just 0.5 percent packet loss.

TLS 1.3 configurations for S3 traffic were also tested, with a noticeably quicker transition to the data plane being setup when the 1RTT of TLS 1.3 was in use.  Extrapolating to enterprise loads and the win should be significant, beyond just preparing the foundations for PQC-infused TLS in the future.

The recommendation from this exercise is to investigate, using tools with network visibility, what options are already actively in use by both TCP and TLS in support of your S3 applications today. Awareness and benefits from TCP windows, scaling, selective acknowledgments and the use of TLS1.3 wherever possible, are all likely to add to the most robust S3 performance possible.