How OneConnect Profile works with Cookie Persistence
1. Preliminary Information This is a hands-on test based on whatK7964explains regarding interaction between OneConnect®profile and Cookie persistence on Keep-Alive connections. OneConnect® changes the default behaviour of making a load balancing decision based on TCP connection to one based on HTTP requests. The fact that back-end server connection can potentially be kept alive and reused for different client requests can cause issues for those who don't completely understand OneConnect®. Whilst working for F5 Support, most cases I had were related to Keep-alive connections, in particular when used with Cookie Persistence. I hope to clarify OneConnect® behaviour with Cookie Persistence in this article in a practical hands-on way. 2. Scenario → Real IP address of Client1 and Client2 does not matter as we'll be focusing on connection between BIG-IP1 and BIG-IP2 Note: BIG-IP is using Cookie persistence (insert) in our example. If you don't understand cookie persistence please refer toK83419154: Overview of cookie persistencefirst. For all tests below, 1st request (from Client1) always create new TCP connection between BIG-IP1 and BIG-IP2 and then 2nd request (from Client2) goes through same TCP connection! Keep this in mind while reading, please! 3. Lab test Results Cookie Persistence only (without OneConnect®) BIG-IP only reads cookie persistence entry once for TCP connection (at first HTTP request) If no cookie is sent to BIG-IP, BIG-IP creates one and hands back to client for first request only over a TCP connection BIG-IP also ignores subsequent cookie persistence entries in subsequent HTTP requests and do not hand further cookies over same TCP connection OneConnect®+ Cookie persistence BIG-IP reads cookie persistence entry for every HTTP request over same TCP connection If no cookie is sent (e.g. new request), BIG-IP takes new load balancing decision and hands cookie back to client every time If cookie is sent BIG-IP persists based on cookie information for every request over same TCP connection 4. OneConnect®and Cookie Persistence over the same TCP connection 4.1 New requests BIG-IP takes load balancing decision after each new HTTP request over same TCP connection: 4.2 Subsequent requests BIG-IP reads cookie persistence entry for each HTTP request and persists: 5.Cookie Persistence ONLY (without OneConnect®) over the same TCP connection 5.1 New requests BIG-IP creates cookie persistence entry and hands to client after 1st HTTP request and no longer creates further entries for subsequent requests: BIG-IP also creates one cookie record for the TCP connection and hands it back to Client1. New request from Client2 goes to Server2 as no new load balancing decision is taken. 5.2 Subsequent requests BIG-IP only reads cookie persistence entry once for TCP connection (at first HTTP request) but ignores entries for subsequent requests: If there were multiple clients with different cookie persistence records pointing to other servers, they would all be ignored and BIG-IP would keep sending requests to Server2 as part of first persistence decision when TCP connection was first established.How OneConnect Profile's max-size works
1.Preliminary Information This setting controls the maximum number of idle server-side TCP connections that BIG-IP holds in the connection reuse poolsplit per tmm,i.e.if BIG-IP has 2 tmmssetting max-size to 2 gives each tmm 1 idle connection. If HTTP requests reach the same TMM, then max-size of 1 and 2 will have the same effect in practice. This is a quick walk through to show how OneConnect max-size works. For more general overview and description of each profile option individually please refer to one of the following resources: What is HTTP Part VII - OneConnect How OneConnect with Cookie Persistence works K7208: Overview of the OneConnect profile K2055: The BIG-IP system may not load balance traffic evenly when OneConnect is in use K85357500: Managing connection reuse using OneConnect source mask (12.x) Think of this article as a quick lab test so we can quickly understand what oneconnect max-size is along with corresponding tmsh show command output 2. Lab Scenario My BIG-IP (VE) box has 2 tmms: I've set max-size to 2 and left everything else with default settings (setting to 1 would have the same effect):: This roughly makes max-size = 1 per tmm. We can now play with 'show' commands and understand a bit more about OneConnect using fewer requests. This is the scenario: I added the following iRule to the virtual server to make it simple and confirm whether connection has been re-used or not: 3. Lab test results BIG-IP adds eligible connections to OneConnect reuse pool max-sizevalue set is split per tmm Ifmax-sizehas been reached in a given tmm, new eligible connection is not added to connection reuse pool In such case, such connection isclosed (FIN/ACK)when we receive another request that could potentially reuse it and BIG-IP creates a new one Note: In this lab test I am usingCurrent Idleoutput fromtmsh show ltm profile one-connect as a reference to understand connection reuse pool because it is a controlled environment and this is the only kind of traffic BIG-IP is receiving. However,Current Idleactually shows current number of any idle server-side connection, even if it is not in connection reuse pool, i.e. either not eligible because ofsource-maskconfiguration or because it hasn't been claimed to be reused yet for the first time. 4. Explanation of show command output Current Idle: number of current idle server-side connections regardless if it is eligible or not to be reused Maximum: number of maximum concurrent connections a single tmm has ever handled Total Reuses: Goes up when any idle connection is reused New: Goes up when BIG-IP establishes any new connection server-side 5. Lab test details I sent first HTTP requests via curl from client (10.199.3.135): And it goes to Server1: At this moment, server side connection is immediately added to connection reuse pool (Current Idle): I then send 2nd request and it goes to Server2: Becausemax-sizeis set to 1 then there is no room for another eligible idle connection and old connection is closed to give room to a new connection. This explains why New counters increases to 2 (since this is the 2nd new connection this profile handled since the counter started) and Current Idle remains at 1 (because max-size only allows 1 idle connection server-side, remember?): 3rd request comes in and round-robin method determines it should go to Server1 but as there is an idle connection in connection reuse pool this time BIG-IP reuses it (notice port 38742 is the same port): And fair enough Total Reuses go up: 4th request to Server2 also creates a new connection that is also closed because connection reuse pool still contains Server1 connection entry: New entry increases againas it is a new server-side connection butCurrent Idle (connection reuse pool) does not: I have now set max-size to 4 and that gives 2 max entires per tmm and now I see connection being reused once for both pool members because BIG-IP now can leave up to 2 idle connections open: The 'tmsh show ltm profile oneconnect myoneconnect' output reflects that: Because the same tmm instance (tmm0) handles all the requests, a max-size of 4 gives tmm0 2 spots for idle connections which also explains why a 2nd new connection is never added to Connection Reuse pool and Current Idle didn't go above 2.What is HTTP Part VII - OneConnect
In the last article in this What is HTTP? series we finished covering the settings in the HTTP profile itself. This week, we begin the transition to technologies that support, enhance, optimize, secure, and (insert your descriptive qualifier here…) the HTTP traffic flowing through the BIG-IP, starting with OneConnect. What is OneConnect? What comes to mind when you hear the phrase “Reduce Reuse Recycle”? Does a little green pseudo-triangle with arrows flash before your eyes, or do you think if they really cared about the reduction, the phrase would be “re[duce|use|cycle]”? But I digress. OneConnect reduces the number of server-side connections between the BIG-IP and application servers by reusing the existing server-side connections. So whereas you might be serving thousands of connections on the client-side, there will be a reduced number of server-side connections to offload the overhead of tcp connection setup and teardown. You can see in the image above that before HTTP/1.1, single requests per connection on client and server sides resulted in inefficiencies. With HTTP/1.1, you can multiplex requests on single connections client-side, and as shown below, can multiplex multiple client-side connections onto single server-side connections. This is the power OneConnect offers. Both clients multiple requests in a single tcp connection for each client are pooled together on a single tcp connection between the BIG-IP and the server. There’s even hope for HTTP/1.0 clients via the OneConnect Transformations I mentioned in part five of this series. In this case, the four individual connections on the client-side of the BIG-IP are transformed into a single connection on the server-side thanks to OneConnect. The OneConnect Profile Let’s start with a screen shot of the fields available in the OneConnect profile. As this is the default parent, you don’t have the option to name it and set the parent, but when creating a custom profile, those settings would be available for configuration as well in the General Properties. Source Prefix Length Also known as theSource Maskfield in earlier versions, this is where you specify the mask length for connection reuse. It’s important to understand that this is the contextual source IP of the server-side connection. This means that if you are using a SNAT address, it is applied before the mask length is evaluated for OneConnect reuse eligibility. The masking behavior details by version are maintained on AskF5 in knowledge base articles K5911 (10.x - 11.x) andK85357500 (12.x.) In my current lab version of 12.1.2, the prefix length can be set to IPv4 or IPv6 mask lengths, and are supplied in CIDR notation instead of traditional netmasks. There’s no rocket science to the mask itself, a mask-length of 0 (default) is most efficient as OneConnect will look for any open idle tcp connection on the destination server. A length of /32 for IPv4 or /128 for IPv6 is a host match, so only an open connection between the source host IP (or SNAT) and the destination server would be eligible for reuse. Maximum Size This setting controls the maximum number of idle server-side tcp connections that BIG-IP holds in the connection reuse pool. This value is divided by the number of TMMs on BIG-IP, so for the default of 10,000 on a system with 10 TMMs, the maximum number of idle connections per TMM would be 1,000. This can be tested with an artificially low maximum-size of two, which means one a two TMM system only one idle connection is available per TMM in the connection reuse pool. Let's use this iRule to make it simple to confirm whether connection has been reused or not: when HTTP_REQUEST_RELEASE { log local0. "BIG-IP: [IP::local_addr]:[TCP::local_port] sent [HTTP::method] to [IP::server_addr]:[serverside {TCP::remote_port}]" } If we have two servers and send two requests to them, each request goes to each server respectively (everything except max size here is default with round-robin LB method.) Rule /Common/check_oneconnect : BIG-IP: 172.16.199.234:38648 sent GET to 172.16.199.31:80 Rule /Common/check_oneconnect : BIG-IP: 172.16.199.234:38650 sent GET to 172.16.199.32:80 The server2 connection (from second request) is not added to the connection reuse pool because max size is set to 2 (1 per TMM) and server1 is already there. The third request goes to server1 and reuses the connection from the first request (src port 38648): Rule /Common/check_oneconnect : BIG-IP: 172.16.199.234:38648 sent GET to 172.16.199.31:80 The fourth request goes to server2 and because there are no idle connections open to server2 then a new connection is created: Rule /Common/check_oneconnect : BIG-IP: 172.16.199.234:38646 sent GET to 172.16.199.32:80 The fifth request again goes to server1 still reusing the idle server1 connection: Rule /Common/check_oneconnect : BIG-IP: 172.16.199.234:38648 sent GET to 172.16.199.31:80 A sixth request to server2 also creates a new connection because server1 is still occupying the only spot available in the connection reuse pool: Rule /Common/check_oneconnect : BIG-IP: 172.16.199.234:38658 sent GET to 172.16.199.32:80 Maximum Age This is the maximum number of seconds that a connection stays in the connection reuse pool before the BIG-IP marks a connection ineligible for reuse. Two key points on the max age: It is triggered after the first time the connection is reused, not when the connection is used for the initial request. It is only after the first reuse is completed and the connection is idle that the max age countdown begins. The max-age setting does not close connections when expired, it marks connections ineligible for future reuse. They are then closed in one of two ways. The first way is the BIG-IP will close an ineligible idle connection when a new request comes in. The second way is to wait for the server’s HTTP keep alive to time out. Both sides of the connection have the ability to close down the connection. Let’s take a look at the max age behavior by setting it artificially low on the BIG-IP to 5 seconds, and setting the test apache server to 60 seconds to prevent apache from closing the connection first. If we send one request (frame 10) only the connection is closed 60 seconds later (frame 46) honoring the keep alive we set on apache: The max age did not kick in here as the connection was not reused. However, let’s make another initial request (frame 10.) This time we make a second request (frame 51) 20 seconds later but before the 60 second timeout. Notice the connection has been reused Now that the max age setting has kicked in, we wait another 20 seconds (well after the 5 second expiration we set) and BIG-IP closes the previous connection (frame 77) and opens a new one (frame 92.) One the new connection is established the client-side request is sent to the server (frame 96.) You can also see that after another minute of idleness, the apache server keep alive timeout has expired as well, and it closes the connection (frame 130.) If we had opened a new connection within 5 seconds after frame 115, then the connection would have been reused, and then max-age would kick in again immediately after the connection was idle again. Maximum Reuse This setting controls themaximum number (default: 1,000) of times a particular server connection can be reused before it is closed. So for example, if you set the maximum reuse to two connections, a server-side tcp connection (172.16.199.234:63826 <-> 172.16.199.31:80) would be used for three requests (one initial in frame 10 and two reuses in frames 49 and 90,) and then BIG-IP, unlike with max age in marking a connection ineligible, will immediately close the connection (frame 119.) Idle Timeout Override When enabled (disabled by default,) this setting overrides any timeout set by protocol profiles such as tcp which has a default timeout of 300 seconds. So with the BIG-IP tcp idle timeout at 300 seconds and apache (from earlier) set to 60 seconds on the keep alive, the idle connection will be closed by the server. If we set the keep alive to 301 seconds, BIG-IP would reset the connection. For the sake of testing the idle timeout override, however, let’s enable the idle timeout override in the OneConnect profile by setting it at 30 seconds. You can see in the capture above that after the request is completed (frame 33) and the connection is now idle, it is reset after 30 seconds (frame 44,) occurring before the server timeout and the BIG-IP tcp timeout. You can also set the idle-timeout-override to infinite, which will ignore the other BIG-IP protocol timers but not overcome the settings of the server keep alives. Limit Type This setting controls how connection limits are managed in conjunction with OneConnect. None (default) - OneConnect is not involved with the connection limits. Idle - Specifies that idle connections will be dropped as the tcp connection limit is reached. For short intervals, during the overlap of the idle connection being dropped and the new connection being establish, the tcp connection limit may be exceeded. Strict - Specifies that the tcp connection limit is honored without exception. This means that idle connections will prevent new tcp connections from being made until they expire, even if they could otherwise be reused. This is not a recommended configuration, but might prove useful in exceptional cases with very short expiration timeouts. Understanding OneConnect Behaviors OneConnect does not override load balancing behavior. If there are two pool members (s1 and s2) and we use the round-robin load balancing method with OneConnect applied, the behavior is as follows: First request goes to s1 Second request goes to s2 Third request goes to s1 and reuses idle connection Fourth request goes to s2 and reuses idle connection Applying a OneConnect profile to a virtual server without a layer seven transactional (read: clearly defined requests and responses) protocol like HTTP is generally considered a misconfiguration, and you may see odd behavior when doing so. There may be exceptions, but this configuration should be considered carefully and tested thoroughly before deploying to a production environment. Without OneConnect on a virtual server with HTTP, you will find that persistence data does not appear to be honored. This is because by default, the BIG-IP system performs load balancing for each tcp connection, not each HTTP request, so further requests on the same client-side connection will follow suit from the original decision. By applying OneConnect to the virtual, you effectively detach the server-side from the client-side connection after each request, forcing a new load balancing decision and using persistence information if it available. Resources OneConnect profile overview OneConnect and persistence OneConnect HTTP headers OneConnect and Traffic Distribution OneConnect and Source Mask Thanks to Rodrigo Albuquerque for the fantastic test source material!
