Forum Discussion

cf_525_pain_310's avatar
cf_525_pain_310
Icon for Nimbostratus rankNimbostratus
Feb 16, 2017

CloudFlare 525 Errors - f5 fix?

I recently started routing assets called via https through Cloudflare, a proxy service, amongst other things.

 

I have an interesting issue where, once i did that, i saw that around 1% of browser requests get presented with a "525" cloudflare ssl handshake error (their generated error), which seems to have this issue calling our origin some .5-1.0% of the time. We did a bunch of packet captures - while its ongoing at this time, I'm curious if other folks have run into an issue like this, and if any setting on their f5 might have helped fix it.

 

I tried a few things like: - increasing the ssl handshake timeout in my ssl profile - enabling "no session resumption on renegotiation" in my ssl profile

 

All they tell me is to check my f5 for any settings that might account for why this happens (the RST/reset messages seen ssl streams in the packets). I see stories all over the internet about various "fixes" when someone starts using cloudflare. But many are related to something not working 100% of the time, and in my case, its around 1% error rate.

 

I have an issue with Cloudflare presenting around 1% of client browser requests with this 525. On our end, we have a public IP natted to an internal vip configured for ssl, with an ssl profile and the certificate applied to the VIP itself - so SSL terminates right on the f5 device. the requests then flow to a pool of proxy servers running nginx, but by that time, there's no encrypted traffic. I've got a PAN firewall in front of the f5s.

 

I've spent a lot of time trying to resolve this over the last week, including a ticket with f5 that hasn't yet revealed anything. I'm appealing to devcentral to try and find someone who might have dealt with this strange problem. the hard part is the frequency - its enough of an error rate that it needs fixing.

 

  • Basic questions:

     

    1. Have you verified that the SSL cipher suite between cloudflare and F5 are compatible ?
    2. SNI enabled/disabled ?
    3. Have you tested with SSL terminating directly on the server(s) or a test server ?
    4. Any weird logs on the F5 ?
    5. Did this error exist since the initial set up or was noticed recently after a change like code upgrade/downgrade ?
    1. the cipher suites are compatible. I made comparisons and even matched CFs entire list and applied it to the f5s. didn't change the error condition

       

    2. SNI isn't being used. rather, none of the checkboxes or the server name field have any data. as an FYI, there are 3 websites/domains that go to this VIP, but they share one ssl cert with all 3 domain names in it. the cert I'm using does have all 3 domain names (san) in it.

       

    3. Yes and no. I tried but the nginx proxy isn't configured to handle that ssl traffic and would require other resources to do it, which I can do if necessary.

       

    4. There are some weird logs on the f5 that I see; but they don't seem to correlate with the 525s.

       

    • cf_525_pain_310's avatar
      cf_525_pain_310
      Icon for Nimbostratus rankNimbostratus

      I actually split up the sites into 3 vips (all using the same ssl profile settings but I created 3 separate ssl client profiles) - specifically so that I could change up any ssl profile settings that were suggested, and only impact one site at a time (the least important one).

       

    • cf_525_pain_310's avatar
      cf_525_pain_310
      Icon for Nimbostratus rankNimbostratus
      1. I noticed it right away. we actually saw a browser display that 525 error, then had trouble repro'ing it...it happens less than 1% of requests. but I have Catchpoint, I see a few of them in those automated tests each day and many on the cloudflare dashboard. so this could simply be exposing a preexisting condition that now manifests in a browser error. but, I wasn't seeing basic websites tests fail just trying to hit an https page this like does at times.
  • Connection error: ssl_select_suite:6737: TLS_FALLBACK_SCSV with a lower protocol (86) Connection error: ssl_passthru:4015: not SSL (40) No shared ciphers between SSL peers 141.212.122.129.31995:10.20.223.37.443. Connection error: ssl_null_parse:3103: record protocol version incorrect (47)

     

  • Hi cf_525_pain!

     

    Did You manage to solve the problem?

     

    For several days I have the same problem in my infrastructure. With the Cloudflare in between and SSL terminates right on F5 too.

     

    Regards.

     

  • JQB's avatar
    JQB
    Icon for Nimbostratus rankNimbostratus

    Same exact scenario here. Anyone have a fix?

     

  • JQB's avatar
    JQB
    Icon for Nimbostratus rankNimbostratus

    Still having the issue. No help from F5 or Cloudflare.

     

    • Have you ever checked on your firewall? In my case App-id was messing with my application (it wasn’t ssl but http). What I did was disable app-id and just allow access on port 443, you don’t actually need app-id for inbound traffic.

       

    • JQB's avatar
      JQB
      Icon for Nimbostratus rankNimbostratus

      That's precisely what I am looking at now, thanks for the feedback. So you disabled app-id for the "web-browsing" application in the PA?

       

    • Yes but that was because my application was http, it you case I assume it should be ssl. You need to make your PAN to look only into the tcp port and nothing else.

       

    • Daniel_Varela's avatar
      Daniel_Varela
      Icon for Employee rankEmployee

      Have you ever checked on your firewall? In my case App-id was messing with my application (it wasn’t ssl but http). What I did was disable app-id and just allow access on port 443, you don’t actually need app-id for inbound traffic.

       

    • JQB_349735's avatar
      JQB_349735
      Icon for Nimbostratus rankNimbostratus

      That's precisely what I am looking at now, thanks for the feedback. So you disabled app-id for the "web-browsing" application in the PA?

       

    • Daniel_Varela's avatar
      Daniel_Varela
      Icon for Employee rankEmployee

      Yes but that was because my application was http, it you case I assume it should be ssl. You need to make your PAN to look only into the tcp port and nothing else.

       

  • I have an virtual server created with many websites hosted in it. It has one public IP assigned to it and is NATed to a private IP. The websites are accessible via HTTP and HTTPS. However one website uses Cloudflare, we experience an error with cloudflare (error 521). I have whitelisted the cloudflare IP addresses onto F5 via Network ›› Packet Filters : Rules

     

    I am running software version 12.1.3

     

    We no longer have blocking both in the F5 and the internal server. This only occurs with websites having Cloudflare configured with it. Is there anything I can check between F5 and Cloudflare?

     

    I have tried applying an iRule that redirects traffic to see if the traffic from Cloudflare IPs are really reaching F5. Sometimes it does redirect but most of the times it servers error 521 immediately.

     

    • Philippe_Page_2's avatar
      Philippe_Page_2
      Icon for Cirrus rankCirrus

      Can anyone provide an insight regarding this issue. Anything will do. Thank you.

       

    • Vijay_E's avatar
      Vijay_E
      Icon for Cirrus rankCirrus

      Throwing a few options:

       

      You said there is no other blocking in place. So, no packet-filters on F5 or any intermediate device ?

       

      When you run a tcpdump, do you see any RST/FIN from F5 or from the pool member to the cloud flare IP ?

       

      Have you tried using OneConnect profile ?

       

    • boneyard's avatar
      boneyard
      Icon for MVP rankMVP

      another idea would be to contact F5 support, this is an issue which pops up from time to time, so they might already have some ideas.