Forum Discussion
James_Michalak_
Nimbostratus
Nimbostratuspersist none for all pool connections when only one member is active (tcp only)
I've reviewed several posts with similar questions, but didn't see a direct parallel.
I'm using BIG-IP 9.3.1 Build 69.0, and don't have the ability to updgrade in the short term.
MY NEED IS THIS:
When using source address persistence, with a 2 node cluster of backend servers to which I'm load-balancing with the "Least Connections" method, I need an irule that will remove persistence on already established connections.
NEEDED BECAUSE:
If one of the offending servers fail, I am left with all connections going to a single server. Assuming that nearly all of the connections "persist" for hours at a time, the load balancing method won't rebalance the connections immediately.
CHALLENGES:
- persistence is required (although sessions can failover between pool members, it is an expensive operation with a lot of overhead).
- one of the two servers can't handle all of the traffic for a sustained period of time (I can't wait for the load balancing method to rebalance connections naturally over time...purchasing a third server would be optimal, but is not possible in the short term).
- there's a certificate on the server, encrypting all traffic, so I'm not able to use an http profile (Importing the certificate on the LTM to unencrypt each request seems cumbersome, but maybe necessary).
WHAT I'VE CONSIDERED:
I know I'll need to use "persist none" at some point, and will need to evaluate the number of pool members when requests are received, but I'm not sure in which EVENT I would place this logic.
Please assist.
James_Michalak_I didn't mention that once the 2nd server is brought back online, the fact that all connections continue to use the already established source address persistence, rebalancing doesn't occur immediately. Running "b persist [ | all] delete" is a manual solution, but I'm looking for an irule to remove the manual dependency.
L4L7_53191I'd strongly suggest terminating that SSL as you'll be able make decisions that are orders of magnitude more intelligent than source address affinity. Believe it or not, it's not really a big deal to move the cert over to the BigIP at all. The LTM uses hardware offload for all of its crypto (handshakes *and* bulk crypto) so this isn't a big deal load wise. I'd expect that your servers will appreciate not dealing with it as well. If this is an option, I'd definitely encourage you to test it. If you do this, you can use cookie inserts for persistence which is (to me, at least) always ideal if you can use it.
Anyhow, you're right: once your server comes back online, the current connections will persist to the server they've been talking to. This is by design because in this case we've configured the BigIP to persist...if your app can tolerate re-binding to a new system - and it sounds like it can, as a last resort - you may be able to pull this off via iRules by clobbering the persistence record, although this may add a few challenges. How would you decide which connections to re-bind and which not to? It may not matter, but it's worth considering...
-Matt- James_Michalak_Thank you, Matt. Unfortunately we're migrating the established application and backend infrastructure to our data center, and suggesting changes to the application architecture has been a bit of a challange.
- James_Michalak_To answer your question: "How would you decide which connections to re-bind and which not to? It may not matter, but it's worth considering..".
I plan to evaluate the total number of connections, and compare that to the number of connections on each pool member. Once the proprortion is acceptable, the irule should not trigger "persist none". - If your goal is to send about half the connections to the second server after it comes up, then I would seriously consider using the Election Hash iRule (http://devcentral.f5.com/wiki/default.aspx/iRules/ElectionHashLoadBalancingAndPersistence.html) for all of your persistence for this appliction.
Since you're not on 10.x, you'll need the iRule, and since you're not even on 9.4, you'll need to hardcode the list of pool members instead of using "active_members -list". Also, since you don't have any L7 data to key off of, you'll need to be using the client IP instead of the HTTP uri. But overall this would probably the easiest solution to code.
I would echo the above sentiment that setting up SSL offloading onto the BIG-IP would be something to push for trying soon. It is not that difficult to setup by itself, though your application could very well introduce some complexities as it may have some assumptions about getting data over HTTPS. It very likely will help reduce load on your existing servers, as they would no longer have any SSL processing to contend with. - James_Michalak_Thank you, Spark. The Election Hash irule is extremely useful in that it would only break persistence if the number of active members changed (eliminating overhead for session failover on the server in most cases). Also, it works when considering source IPs as well. My only question would now be, in what event would I place the logic? I completely agree that offloading SSL on the BIG-IP is useful and preffered. We typically do this in most cases. Unfortunately this isn't a typical application, and I've already lost this battle. As a result, I can't use the HTTP_REQUEST event. Should I use CLIENT_DATA (Is this too expensive)? CLIENT_ACCEPTED only triggers when the connection is first created, so it wouldn't allow me to rebalance existing connections. Any recommendations?
- I would think that CLIENT_ACCEPTED is the appropriate choice here.
I don't think there's any way to migrate an existing connection, especially on 9.3.1 and without being able to see the application-layer traffic. The only thing that *might* work is to reset a connection that should go to the other server when that server comes back up. Will your clients automatically reconnect? I can't think of any other way to do that. - James_Michalak_I had feared as much. Thank you for your time and consideration. Hopefully I can persuade our application folks to terminate SSL at the load balancer or to purchase another server.
- L4L7_53191Remember that you can always re-encrypt back to the servers if they've drawn a hard line in the sand. Not ideal, but it'll let you get at the HTTP information before sending it back to the app.
-Matt - You could put the logic in a CLIENT_DATA event. This would allow you to make a new pool member deicision every time the client sends more data over. Note that you would need to precede the new pool selection with an LB::reselect statement since you already have a pool member selected
