Forum Discussion
cf_525_pain_310
Nimbostratus
NimbostratusCloudFlare 525 Errors - f5 fix?
I recently started routing assets called via https through Cloudflare, a proxy service, amongst other things.
I have an interesting issue where, once i did that, i saw that around 1% of browser requests get presented with a "525" cloudflare ssl handshake error (their generated error), which seems to have this issue calling our origin some .5-1.0% of the time. We did a bunch of packet captures - while its ongoing at this time, I'm curious if other folks have run into an issue like this, and if any setting on their f5 might have helped fix it.
I tried a few things like: - increasing the ssl handshake timeout in my ssl profile - enabling "no session resumption on renegotiation" in my ssl profile
All they tell me is to check my f5 for any settings that might account for why this happens (the RST/reset messages seen ssl streams in the packets). I see stories all over the internet about various "fixes" when someone starts using cloudflare. But many are related to something not working 100% of the time, and in my case, its around 1% error rate.
I have an issue with Cloudflare presenting around 1% of client browser requests with this 525. On our end, we have a public IP natted to an internal vip configured for ssl, with an ssl profile and the certificate applied to the VIP itself - so SSL terminates right on the f5 device. the requests then flow to a pool of proxy servers running nginx, but by that time, there's no encrypted traffic. I've got a PAN firewall in front of the f5s.
I've spent a lot of time trying to resolve this over the last week, including a ticket with f5 that hasn't yet revealed anything. I'm appealing to devcentral to try and find someone who might have dealt with this strange problem. the hard part is the frequency - its enough of an error rate that it needs fixing.
Vijay_ECirrus
Basic questions:
- Have you verified that the SSL cipher suite between cloudflare and F5 are compatible ?
- SNI enabled/disabled ?
- Have you tested with SSL terminating directly on the server(s) or a test server ?
- Any weird logs on the F5 ?
- Did this error exist since the initial set up or was noticed recently after a change like code upgrade/downgrade ?
cf_525_pain_310-
the cipher suites are compatible. I made comparisons and even matched CFs entire list and applied it to the f5s. didn't change the error condition
-
SNI isn't being used. rather, none of the checkboxes or the server name field have any data. as an FYI, there are 3 websites/domains that go to this VIP, but they share one ssl cert with all 3 domain names in it. the cert I'm using does have all 3 domain names (san) in it.
-
Yes and no. I tried but the nginx proxy isn't configured to handle that ssl traffic and would require other resources to do it, which I can do if necessary.
-
There are some weird logs on the f5 that I see; but they don't seem to correlate with the 525s.
- cf_525_pain_310
I actually split up the sites into 3 vips (all using the same ssl profile settings but I created 3 separate ssl client profiles) - specifically so that I could change up any ssl profile settings that were suggested, and only impact one site at a time (the least important one).
- cf_525_pain_310
- I noticed it right away. we actually saw a browser display that 525 error, then had trouble repro'ing it...it happens less than 1% of requests. but I have Catchpoint, I see a few of them in those automated tests each day and many on the cloudflare dashboard. so this could simply be exposing a preexisting condition that now manifests in a browser error. but, I wasn't seeing basic websites tests fail just trying to hit an https page this like does at times.
-
- cf_525_pain_310
Connection error: ssl_select_suite:6737: TLS_FALLBACK_SCSV with a lower protocol (86) Connection error: ssl_passthru:4015: not SSL (40) No shared ciphers between SSL peers 141.212.122.129.31995:10.20.223.37.443. Connection error: ssl_null_parse:3103: record protocol version incorrect (47)
msud_307411Hi cf_525_pain!
Did You manage to solve the problem?
For several days I have the same problem in my infrastructure. With the Cloudflare in between and SSL terminates right on F5 too.
Regards.
JQBSame exact scenario here. Anyone have a fix?
- JQB
Still having the issue. No help from F5 or Cloudflare.
Daniel_VarelaEmployee
Have you ever checked on your firewall? In my case App-id was messing with my application (it wasn’t ssl but http). What I did was disable app-id and just allow access on port 443, you don’t actually need app-id for inbound traffic.
- JQB
That's precisely what I am looking at now, thanks for the feedback. So you disabled app-id for the "web-browsing" application in the PA?
- Daniel_Varela
Yes but that was because my application was http, it you case I assume it should be ssl. You need to make your PAN to look only into the tcp port and nothing else.
- JQB_349735
Still having the issue. No help from F5 or Cloudflare.
- Daniel_Varela
Have you ever checked on your firewall? In my case App-id was messing with my application (it wasn’t ssl but http). What I did was disable app-id and just allow access on port 443, you don’t actually need app-id for inbound traffic.
- JQB_349735
That's precisely what I am looking at now, thanks for the feedback. So you disabled app-id for the "web-browsing" application in the PA?
- Daniel_Varela
Yes but that was because my application was http, it you case I assume it should be ssl. You need to make your PAN to look only into the tcp port and nothing else.
- Philippe_Page_2
I have an virtual server created with many websites hosted in it. It has one public IP assigned to it and is NATed to a private IP. The websites are accessible via HTTP and HTTPS. However one website uses Cloudflare, we experience an error with cloudflare (error 521). I have whitelisted the cloudflare IP addresses onto F5 via Network ›› Packet Filters : Rules
I am running software version 12.1.3
We no longer have blocking both in the F5 and the internal server. This only occurs with websites having Cloudflare configured with it. Is there anything I can check between F5 and Cloudflare?
I have tried applying an iRule that redirects traffic to see if the traffic from Cloudflare IPs are really reaching F5. Sometimes it does redirect but most of the times it servers error 521 immediately.
- Philippe_Page_2
Can anyone provide an insight regarding this issue. Anything will do. Thank you.
- Vijay_E
Throwing a few options:
You said there is no other blocking in place. So, no packet-filters on F5 or any intermediate device ?
When you run a tcpdump, do you see any RST/FIN from F5 or from the pool member to the cloud flare IP ?
Have you tried using OneConnect profile ?
boneyardMVP
another idea would be to contact F5 support, this is an issue which pops up from time to time, so they might already have some ideas.