CloudBleed: Guess What? There was 0-day protection
About CloudBleed
If you aren’t familiar with CloudBleed, take a moment to read the following articles to get an understanding how it was found, what happened, and what PII/PCI data was (possibly) leaked:
- Vulnerability Disclosure from Tavis Ormandy (@taviso), a security researcher at Google
- Incident Response from CloudFlare
- 3rd party assessment from Ryan Lackey (@octal)
In some of the examples shown by Tavis, or by digging into cached sites on less popular search engines, you can see how usernames, passwords, sessions, credit card info, etc could be seen in clear text despite the use of HTTPS (TLS1.2). A simplification of the flow would look like this:
Request: Client >=====> CloudFlare >=====> Origin Web Server
Response: Client <=====< CloudFlare <=====< Origin Web Server
Despite the fact that the client is utilizing HTTPS, CloudFlare has the ability to terminate the secure connection since it has the private key. This is essentially the content (PII or PCI) that was being cached if certain conditions occurred on the CloudFlare reverse proxy.
0-day Protection Against CloudBleed
So, what is the 0-day protection and what is it all about? Application Layer Encryption is a feature from F5 that would have left researchers scratching their heads when looking at the cached content.
First things first, let’s dynamically encrypt sensitive parameter names using Application Layer Encryption. But really, what does that mean? Let’s take a deeper look.
In all examples I will be using Google Chrome.
Let’s say you have login page without Application Layer Encryption.
If I were to right click on the Password box and choose Inspect, I would see something like this:
As you can see from the red boxes, the parameters for my username and password are log and pwd. We can all agree that this is in human readable format (HRF).
If I were to do the exact same thing with Application Layer Encryption, I would see this:
In the screen shot above there are 3 things I would like to focus your attention on:
- Both name=log and name=pwd are no longer in HRF.
- Notice in the first red box that there is a slight purple hue over the value? That’s the dynamic part of Application Layer Encryption. Every few seconds the values will change. This is done on the client side, not the server side. That value will never repeat and is completely unique to the user’s session.
- We also removed the element IDs, id=”user_login” and id=”user_pass”.
With Application Layer Encryption turned on, every user will have a unique value for name=log and name=pwd moving forward.
We can also turn up the security a notch by inserting decoy fields to further confuse the bad guys.
Decoy fields are only seen in the raw code and do not change the end users experience. As you can see above, there are additional IDs which makes it very difficult to pull apart in a replay attack.
But Brian, what about the values being sent!?!!?
Let’s look at a data submission without Application Layer Encryption from the browser via the network tools in Chrome:
As you can see from the picture above, the username is briandeitch and the password is test. Although this is sent via HTTPS, CloudFlare has the private key and this is the type of data that could have been cached by search engines. Yikes!
Same exercise but this time with Application Layer Encryption:
Decoy fields turned off:
Decoy fields turned on:
As you can see from the form data, both the parameter names and values have been dynamically encrypted on the client side. Let’s say this data was leaked by CloudFlare and cached by a search engine. Couple of points:
- These substituted and encrypted values are single use, meaning a replay attack wouldn’t work.
- Good luck reverse engineering any of this. Application Layer Encryption uses private/public key technology and this key, unlike the private key for the SSL Certificate, isn’t hosted on CloudFlare. This means the payload (form data shown above, you know the values for the username and password) is completely encrypted to CloudFlare (or any other proxy for that matter).
- In addition, the decoy fields are sent as well. Good luck deciphering which fields are mapped to the actual application(s).
Personally, my favorite part of Application Layer Encryption is that is requires no change to my back end application. This solution requires no installation or modification to the end users web browser, is 100% client-less, and transparent to the end user. At the end of the day, you can still rely on HTTPS for transport layer security however Application Layer Encryption steps up your overall security posture by using clever obfuscation and encryption to protect your data when there is a proxy (MITM device) between your customers and application(s).
Lastly, Application Layer Encryption is a feature within F5 WebSafe. F5 has taken a complex security enhancement and created a point and click solution to easily protect your application(s).
Configuring Application Layer Encryption
Step 1: By creating a profile, specify the URIs you want protected.
*Note: You are able to use wildcard URIs as well.
Step 2: Add the parameters you want protected.
The Encrypt checkbox is how we secure the username and password values that are being sent. By encrypting the values, this will prevent replay attacks as the values can only be used once.
The Substitute Value checkbox substitutes the parameter’s value with a random value in the web application while the form is being filled. This also prevents browser based malware from capturing keystrokes as they are being entered.
The Obfuscate checkbox encrypts the parameter’s name attribute.
Step 2a: If you would like to insert decoy fields to further mess with the bad guys, you don’t have to be a rocket scientist. Relax, it is just a checkbox.
Step 3: Associate the new profile to the application.
That’s it! You have successfully configured and secured your application with Application Layer Encryption. Special thanks to BAM, Todd Morton, Chad Fazio, Jeffery McFerson, Joe Martin, and Ted Byerly for their contributions on this article.
Published Feb 27, 2017
Version 1.0
Brian,
Thanks for taking the time to write this article. Making people aware of the capabilities of Application Layer Encryption and the continued compromise and/or attempted compromise of user credential is essential.
Carl_BrothersEmployee
Brian, I think one distinction that is missing for some might be where this solution fits in to the original equation.
Unprotected -
Request: Client >=====> CloudFlare >=====> Origin Web Server
Response: Client <=====< CloudFlare <=====< Origin Web Server
Protected -
Request: Client >=====> CloudFlare >=====> F5 proxy with WebSafe >=====> Origin Web Server
Response: Client <=====< CloudFlare <=====< F5 proxy with WebSafe <=====< Origin Web Server
HTH
chris888_147181Nimbostratus
Great article and write up. Presumably then when this is encrypted client side, the appropriate decryption happens within Websafe?
I am curious about this with regards to how it is set up for specific web apps. Would it be correct to say that the device needs to be told about all such fields?
Also, does this fully mitigate all issues noted with Cloudbleed? As in, we can see that clearly it can prevent the clear text caching of credentials, but this is only a small part of the issue...can you tell Websafe about session cookies and does mitigate session hijacking in the same way and prevent replay with randomisation?
Thanks :)
Carl-
Great point, I will incorporate that into my article.
Regards, BD
Hi Chris888-
Replying from my mobile, hopefully I can articulate some decent thoughts here.
WebSafe is telling the client to encrypt the credentials on the fly via public/private keys. When an encrypted parameter is sent to WebSafe, it will decrypt it and send it (credentials in my example) to the web server. Does that make sense? And yes, you will need to call out which fields that need to be encrypted.
As for your question regarding Session replay is really two fold. We might be able to encrypt sessions via WebSafe (iRule possibly), I would have to lab it up to know for sure. I would recommend using WebSafe to protect credentials as it would be trivial to grab the clear text credentials and sign in.
As for mitigating a session replay, I know for sure that's a feature of ASM. We can prevent session hijacking. Any attempt to replay a session that's already been used will stick out like an orange hat with a green bill.
Good dialog, let's keep this going :)
- chris888_147181
Hey Brian,
Thanks for that. That certainly makes sense regarding the WebSafe part and the need to call out the fields which need to be encrypted (I expected that to be the case).
- chris888_147181
I'd actually not come across this feature before. I was also thinking about the other aspect of Cloudbleed as I was not quite sure how this would have been able to mitigate session/auth tokens being captured and re-used. Just in terms of say actually passing a username and password - when it comes to overall transaction volume it's probably quite low, but then subsequent authenticated requests would be done with the cookie and so that's the bit I was wondering how to mitigate with F5.
So a quick for search for ASM session hijacking search turned up a page on the support site which talks about exactly how this is achieved. The Client-ID is a very elegant solution. I am not overly familiar with ASM but it certainly addresses my concern of it just being the same cookie passed every time.
It seems we need more Application Layer Encryption in the world!
PS - Sorry for separate comment, but for some reason it was flagged as spam otherwise.
Chris888-
I would assume you are 100% correct about the actual volume of credential vs sessions that were inadvertently exposed based on basic math and this from CloudFlare:
The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).
Although, they do have 5 million websites--so how many HTTP requests do they actual see per second?
WebSafe's Application Layer Encryption is more about a risk based approach to security and not around compliance. You have to ask yourself what would have happened to your business had credentials been exposed? Is it a slap on the wrist? Does the CISO lose his or her job? The list goes on and on.
Did you mean Device-ID by chance? :) I would agree with you, Device-ID is a very elegant solution. If we put ourselves in the shoes of the bad guys and try to do a replay attack (session hijacking), it would fail. There are some 20+ environmental elements that Device-ID is able to hone in on. Unfortunately, I am not at liberty to discuss all of them (in fact, they won't tell me, ha!) but we look the browser version, screen resolution, time zone, plugins, fonts etc to make an exact finger print of the client. Take all of that into account, you'd be hard pressed to execute a successful replay attack.
I've personally attempted to circumvent Device-ID and WebSafe by using Burp Suite and have zero luck. Obviously, I am not the Alpha and Omega, but it is challenging enough that if I had hair I would have pulled it all out and given up.
Best regards, BD
