Forum Discussion
10.2.3 Persistence Profile inconsistencies
I have a web portal deployed that is behind a pair of F5s. There are 3 web servers being load balanced that require persistence because there's no central session broker for the portal system.
Because of the way the web guys wanted to deploy the portal, it was done in mixed-mode SSL, meaning, the users go to are redirected to and are redirected back to after they authenticate correctly.
Something like 90-95% of the time, this seems to work and their sessions stay together. The other 5-10%, they start on one web server and end up on a different one. Then their sessions get locked on the web servers and they can no longer login until they time out (long time outs too).
To do this, I have to have 2 pools and 2 virtual servers (http and https) and match across virtual servers.
The number of calls about this problem showed up pretty quickly and it appears the upgrade to 10.2.3 from 10.2.0 may be the culprit. I found this in the release notes prior to 10.2.3:
Behavior changes in 10.2.2
ID 226971, CR142479
In previous releases, the system reused client session IDs from previous sessions to reestablish SSL connections. Now, in situations where security changes in the BIG-IP configuration, for example, an iRule changes the security parameter to request or require client certificates, the system establishes a new SSL connection with the client and does not reuse the previously established session ID.
So...my question. Could that be the cause of a Source IPAddress Affinity profile not working?
I plan on rolling back to 10.2.0 and seeing if the problem goes away. I just wanted some more eyes on the problem to assist since I've not gotten much feedback from Support over the past many weeks and numerous tcpdumps showing the behavior. They've really tried though-
Thanks,
Scarpozzi
11 Replies
- Marek_S_64473
Nimbostratus
I don't see any reason why SSL renegotiation would affect Source IP based persistance.The client IP stays the same all the time. Can you try disabling CMP to see if the problem goes away? You can do it just for one virtual server. - scarpozzi_82104
Nimbostratus
At this point, I'm just trying to get a handle on why persistence isn't working and looking at anything that could be the cause. I don't know how the F5 handles the source ip sessions and whether or not it links them to a session id in the tables or if it does it the other way around... I don't know how long we've experienced the problems because it's so infrequent and I've only had it happen to me a few times in the past year and I use the web servers every day. Some folks have it happen much more frequently, for some reason.
I just read up on CMP. I didn't realize that was even an option/problem. I always make the assumption that a persistence profile defined by a vendor is going to fine-tune the device so it actually keeps persistence...
I'll disable it for the virtual servers in question and see what happens in the server logs.
Thanks for the suggestion. - hoolio
Cirrostratus
Hi Scarpozzi,
Can you post your case number so I can take a look?
Thanks, Aaron - scarpozzi_82104
Nimbostratus
C1109834
I disabled CMP on the virtual servers just after my last post. I had one of our analysts check the webserver logs and he noticed that it happened again after making the change.
I was hopeful that it might be the silver bullet we've been looking for. - hoolio
Cirrostratus
Make sure you have match across services enabled on the source address profile. And make sure that each virtual server's pool has the same node IP addresses. See SOL5837 for details:
sol5837: Match Across options for session persistence
https://support.f5.com/kb/en-us/solutions/public/5000/800/sol5837.html
The Match Across Services option is used in the following two configurations:
Configurations that have multiple virtual servers with the same IP address but have different services specified
Configurations that have pool members sharing the same address but have different services specified
And ensure that the timeout on the persistence profile is long enough so that the persistence record is preserved for as long as the application session.
While you're troubleshooting the source address persistence issue, you could try using an iRule to implement cookie based persistence across services:
https://devcentral.f5.com/wiki/iRules.Cookie_Encryption_across_pools_and_services.ashx
Aaron - hoolio
Cirrostratus
Also, if you were interested in serving all of the content over HTTPS (which would be far more secure than mixed HTTP/HTTPS), you could use a simple iRule to redirect requests to the HTTP VS to HTTPS. On the HTTPS VS, you could use a blank stream profile and an iRule that configures the stream filter to rewrite http:// references to https://.
Here are a couple of related links:
https://devcentral.f5.com/wiki/iRules.STREAM__expression.ashx
HTTPS offload rewriting
https://devcentral.f5.com/wiki/iRules.HTTPS-offload-rewriting.ashx
You could then use cookie insert persistence or source address persistence (without match across services enabled) on the HTTPS VS.
Aaron - scarpozzi_82104
Nimbostratus
I went back and forth over which option would suit me best....match across services or match across vs. In production, since we've had this problem, I've tried both match across virtual servers and match across services. I'm see the error using either one. As for the timeout....I did have that set to 13 hours so it would last the length of a work-day. Then in reading, I found that keep-alives are supposed to extend the timeout, so if they're logged in, it should never be an issue. With it set to 3 minutes (1800 seconds), I can go back 20 minutes later and it still works for me... I'm able to see that I'm still getting these errors in the web server logs.
What's kept me from moving to all HTTPS content is that I don't control the content we're service. The site is a portal app with iFrames. Everything that needs to be encrypted is https....but there is a lot of http content as well. I was afraid to tangle with the workings of the site too much since it's based on decade-old practices... The next version of this app will be using similar tactics, but won't give us the session problems we're having now. I'm kind of waiting it out until we deploy the new version to hopefully "clean up" the way it works.
I'm tempted to try setting a cookie from an iRule as you suggested. It doesn't look like the persistence profile works well enough by itself. I was hoping the persistence tables would be able to track sessions, etc....but at this point, I just want this working.
Thanks for your suggestions. - scarpozzi_82104
Nimbostratus
I used the persistence-based cookie iRule as noted here:
https://devcentral.f5.com/wiki/iRules.Cookie_Encryption_across_pools_and_services.ashx
It appeared to help reduce the issue. (I still got errors in my logs, but they were cookie-based errors...perhaps from mis-configured browsers) I was unable to confirm any persistence problems in the logs.
Unfortunately, adding the cookie somehow broke the login sequence. This is a uPortal application and it started causing uPortal errors until I removed the iRule. The errors occurred just after the auth post when the session was to be redirected back to http and view the login session. Pressing F5 refreshed the screen into an active session....but I can't alter the web portal application. So I guess I'm back to square one trying to figure out why the iRule cookie persistence appears to work and source IP persistence profile doesn't. Considering what we paid for these things, I'm surprised this is that difficult to make it work....though I'm not a fan of the application we're stuck working with either...it's definitely half of the problem. - scarpozzi_82104
Nimbostratus
I used the persistence-based cookie iRule as noted here:
https://devcentral.f5.com/wiki/iRules.Cookie_Encryption_across_pools_and_services.ashx
It appeared to help reduce the issue. (I still got errors in my logs, but they were cookie-based errors...perhaps from mis-configured browsers) I was unable to confirm any persistence problems in the logs.
Unfortunately, adding the cookie somehow broke the login sequence. This is a uPortal application and it started causing uPortal errors until I removed the iRule. The errors occurred just after the auth post when the session was to be redirected back to http and view the login session. Pressing F5 refreshed the screen into an active session....but I can't alter the web portal application. So I guess I'm back to square one trying to figure out why the iRule cookie persistence appears to work and source IP persistence profile doesn't. Considering what we paid for these things, I'm surprised this is that difficult to make it work....though I'm not a fan of the application we're stuck working with either...it's definitely half of the problem. - scarpozzi_82104
Nimbostratus
I used the persistence-based cookie iRule as noted here:
https://devcentral.f5.com/wiki/iRules.Cookie_Encryption_across_pools_and_services.ashx
It appeared to help reduce the issue. (I still got errors in my logs, but they were cookie-based errors...perhaps from mis-configured browsers) I was unable to confirm any persistence problems in the logs.
Unfortunately, adding the cookie somehow broke the login sequence. This is a uPortal application and it started causing uPortal errors until I removed the iRule. The errors occurred just after the auth post when the session was to be redirected back to http and view the login session. Pressing F5 refreshed the screen into an active session....but I can't alter the web portal application. So I guess I'm back to square one trying to figure out why the iRule cookie persistence appears to work and source IP persistence profile doesn't. Considering what we paid for these things, I'm surprised this is that difficult to make it work....though I'm not a fan of the application we're stuck working with either...it's definitely half of the problem.
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com
