What is Web Cache Exploitation?

Let’s talk about Web Cache Exploitation. There was a presentation done at BlackHat/DefCon 2024 discussing this, and here is the link to a writeup done by the presenter: https://portswigger.net/research/gotta-cache-em-all

That article details how different HTTP servers and proxies react when presented with specially crafted URLs. These discrepancies have the potential to be used for use in different types of web cache attacks. My goal here is to give a brief overview and discuss further about how NGINX can be involved in this as well as mitigations that are possible. As such, it is a good idea to reference that article as I am only summarizing pieces of it here. Especially since the researcher did such a great job of writing this up.

Definitions:

First, here are a few terms that will be used in this article:

Web caching — the process of storing copies of web files either on the user’s device or in a third-party device such as a proxy or Content Delivery Network (CDN).

The purpose of this is to speed up the serving of static content by presenting it from the store instead of the backend server. This saves time and resources.

Web caches use keys to determine which responses should be stored or not. These usually use the URL in some fashion, then map to the stored response.

Web Cache Poisoning — the act of inserting fake content into the cache, causing clients to pull content they were not intending to inadvertently.

Web Cache Deception — the act of tricking the backend server to place dynamic content into a cache thinking that it was static. This can be especially bad if the data is intended for an authenticated user.

Delimiters — one or more characters in a sequence that indicate a separation (end/beginning) of the elements in a stream of text or data.

An example of this could be the question mark in a URI indicating that a query is starting.

Normalization - concerning web traffic, the process of standardizing data for consistency across network paths.

We see this a lot with web traffic using % notation for certain characters, such as %20 for a space.

Detecting Delimiters and Normalization:

The article describes that the RFC (https://datatracker.ietf.org/doc/html/rfc3986) states which characters are used as delimiters. The issue is that the RFC is very permissive and allows each instance to add to that list. They then give a few examples of how to detect the delimiters that backend servers or caches use. This can then help to determine if there is a discrepancy between them.

For example: the article shows sending a request for /home and then a request for /home$abcd to see if the response is the same or not.

This can also be used to see if the cached request is served up when specific delimiters are used.

The second discrepancy that the article discusses is with normalization. Using delimiters, the path is extracted and then it is normalized to determine any encoded character or dot-segments that may be used.
I will explain what those are:

Encoding is used sometimes when a delimiter character needs to be interpreted by the application rather than the HTTP parser. For example: %2F used instead of a forward slash /.

Dot-segment normalization is a way to reference a resource from a relative path. Also referred to as a path traversal a lot of the time. For example: ../ used to move back to one directory.

The RFC says how to code URLs and handle dot-segments. But it doesn’t say how a request should be forwarded or changed, which makes it hard to tell which vendors agree with each other. Similar to what was done in the delimiter section, the article gives different examples of how to detect discrepancies in decoding behavior.
For example: the article gives a table that lists different cache proxies as well as HTTP servers and how each treats a request for /hello..%2fworld.

NGINX resolves this to /world whereas Apache does not normalize it at all.

Deception:

Cache rules are used to determine if a resource is static and should be stored or not. The discrepancies mentioned in the last section can be leveraged to exploit cache rules possibly leading to dynamic content being stored. The article describes different data attributes that cache proxies may use to determine if a resource is static or not. These include static extensions, static directories, and static files.

Static extensions may include file types such as .css, .js, .pdf, and more. Some proxies may have rules setup that cause these extensions to allow caching. An example given in the article is where the dollar sign is a delimiter on the backend server but not the proxy. This can cause the response to a specific path to be cached when it should not be. Normalization discrepancies can be used to exploit this as well by encoding a delimiter.

Example: request for /account$static.css will be stored by the proxy due to the .css extension, but due to the delimiter, the response from the backend is for /account which may be a client's authorized account data.

Static directory rules are those that match the path used for the request. Some common examples are /static, /shared, /media, and more.. This is similar to static extensions, where delimiter discrepancies and normalization discrepancies can be used for exploitation. This involves hiding a path traversal after a character that is a delimiter on the backend server. The static directory is then placed after the path traversal, causing the proxy to resolve it but not the backend server.

Example: request: /account$/..%2Fstatic/any cache proxy sees: /static/any backend server sees: /account

Static files are files that may not necessarily be in a static directory or have a static extension but are expected to stay static on every site. Examples of these files are /robots.txt or /favicon.ico. Exploiting these types of rules is similar to how static directories are exploited. In other words, this example would look like the previous except replace 'static/any' with 'robots.txt'.

Poisoning:

If the attacker can get a cache to store a specific response to the key that the cache is using, then they can steer users to that response when they visit. Delimiters and normalization can be exploited to carry out cache poisoning. By combining these with cache poisoning, it could be possible to modify a cache key to point to a highly visited site. There are many ways to combine these to try and use this. These include key normalization and delimiters used by both the backend server and the cache on the frontend.

Key normalization may happen before the cache key is generated. This can allow for poisoning of the mapped resource if the backend server is interpreting the path differently. This is similar to our above example for static directories. If a path traversal is placed between the path for the backend server and the path you want cached, you may be able to map one to the other.

Example: URL: /path/../../home Cache Key: /home Backend Server: /path

As this shows, it is possible to create the cache with a key pointing to /home but returns the response for /path. So, when a user visits /home they will not receive the page expected, but instead they will get the page that the malicious actor wanted them to get.

Server delimiters can be used for this when the cache is not using the same delimiter. This allows for the creation of a key for the response as the delimiter will prevent the backend server from fully resolving the path. This is similar to the last example, but with the delimiter placed before the path traversal.

Example: URL: /path$/../home Cache Key: /home Backend Server: /path

Cache delimiters are harder since special characters that the browser will allow are harder to find for web caches. The pound sign can do this, though, as some caches use it as a delimiter. This is similar to the previous example but would be the other way around as the backend server path would be last after the traversal.

Example: URL: /path#../home Cache Key: /path Backend Server: /home

Mitigation/Defense:

The first thing to note is that none of this means that vendors are doing anything wrong with their products. The differences in how each handles normalization and delimiters is expected given the freedom to add their own options.

Also, I mentioned that I would further discuss how NGINX could be involved in these kinds of attacks. Naturally, as NGINX can be used as a proxy and a web server, it can be involved in these types of transactions. So it really falls on how NGINX handles normalization and delimiters when compared to a web cache being used in the same path. The author of that article does a great job of comparing multiple vendors for backend servers, CDNs, and frameworks.

The first defense would be to try and use products that will align in how they parse data to try and prevent as many opportunities as possible for this to happen.

The next defense and probably the best design choice would be to add a cache control to your pages to prevent caching of pages that should never be cached. This would mean adding a 'Cache-Control' header with values of 'no-store' and 'private' to any dynamically generated responses. Then also ensure that any of the cache rules cannot override the header that is set.

Another option would be to add a WAF into the path of the traffic. Just looking at a lot of the requests used in these examples, I can see that ASM/Advanced WAF or NGINX App Protect would be pretty effective at stopping a lot of these requests. Path traversal and meta-character

One thing that was discussed in the article in regard to NGINX was how it handles the newline-encoded byte (%0A) in a rewrite rule. This byte is used as a path delimiter in NGINX. A common use of the rewrite rule is to use the regex of (.*) to write the rest of the path to then new location.

For example: rewrite /path/.(*) /newpath/$1 break;

This will work in most situations, but if the newline byte is added then it will stop at that delimiter.

For example: /path/test%0abcde ---> /newpath/test
You can see how it gets cut off after the encoded byte is hit.

I did some research on this and found a similar situation with the return rule in NGINX. https://reversebrain.github.io/2021/03/29/The-story-of-Nginx-and-uri-variable/ This blog shows how the Carriage Return Line Feed (CRLF) can be used to inject a header into the response. I tested this by firing up an NGINX container, and adding a location configuration to my nginx.conf file like this:

    server {
        location /static/ {
            return 302 http://localhost$uri;
        }

I then send a request with the encoded CRLF (%0D%0A) and then the header I want injected after that:

  curl "http://127.0.0.1:8081/static/%0d%0aX-Foo:%20CLRF" -v
  *   Trying 127.0.0.1:8081...
  * Connected to 127.0.0.1 (127.0.0.1) port 8081
  > GET /static/%0d%0aX-Foo:%20CLRF HTTP/1.1
  > Host: 127.0.0.1:8081
  > User-Agent: curl/8.6.0
  > Accept: */*
  >
  < HTTP/1.1 302 Moved Temporarily
  < Server: nginx/1.27.0
  < Date: Thu, 15 Aug 2024 18:15:46 GMT
  < Content-Type: text/html
  < Content-Length: 145
  < Connection: keep-alive
  < Location: http://localhost/static/
  < X-Foo: CLRF                               <-----header injected
  <
  <html>
  <head><title>302 Found</title></head>
  <body>
  <center><h1>302 Found</h1></center>
  <hr><center>nginx/1.27.0</center>
  </body>
  </html>
  * Connection #0 to host 127.0.0.1 left intact

That blog also describes how to avoid that happening by changing the return directive to use $request_uri instead of $uri or $document_uri.

This made me wonder if it was possible to similarly modify the rewrite directive to avoid the issue with the newline-encoded byte being used as a path delimiter. After searching, I found this page in GitHub: https://github.com/kubernetes/ingress-nginx/issues/11607

Which then links to: https://trac.nginx.org/nginx/ticket/2452

These pages are discussing this issue with using the newline-encoded byte as a delimiter. The response in the ticket was to use this regex (?s) to enable single-line mode. I re-configured my NGINX container to add another couple of locations so I could test this:

    server {
        location /static/ {
            return 302 http://localhost$uri;
        }
        location /user/ {
            rewrite /user/(.*) /account/$1 redirect;
        }
        location /test/ {
            rewrite /test/(?s)(.*) /account/$1 redirect;
        }

So now I have two rewrite directives, one for testing the issue and one for testing the workaround. Now send a request and see if it works.

  curl "http://127.0.0.1:8081/user/%0d%0aX-Foo:%20CLRF" -v
  *   Trying 127.0.0.1:8081...
  * Connected to 127.0.0.1 (127.0.0.1) port 8081
  > GET /user/%0d%0aX-Foo:%20CLRF HTTP/1.1
  > Host: 127.0.0.1:8081
  > User-Agent: curl/8.6.0
  > Accept: */*
  >
  < HTTP/1.1 302 Moved Temporarily
  < Server: nginx/1.27.0
  < Date: Thu, 15 Aug 2024 18:56:48 GMT
  < Content-Type: text/html
  < Content-Length: 145
  < Location: http://127.0.0.1/account/%0D     <---Newline delimiter was hit.
  < Connection: keep-alive
  <
  <html>
  <head><title>302 Found</title></head>
  <body>
  <center><h1>302 Found</h1></center>
  <hr><center>nginx/1.27.0</center>
  </body>
  </html>
  * Connection #0 to host 127.0.0.1 left intact

For the first test, it cutoff at the newline-encoded byte as expected. Now to test the workaround.

  curl "http://127.0.0.1:8081/test/%0d%0aX-Foo:%20CLRF" -v
  *   Trying 127.0.0.1:8081...
  * Connected to 127.0.0.1 (127.0.0.1) port 8081
  > GET /test/%0d%0aX-Foo:%20CLRF HTTP/1.1
  > Host: 127.0.0.1:8081
  > User-Agent: curl/8.6.0
  > Accept: */*
  >
  < HTTP/1.1 302 Moved Temporarily
  < Server: nginx/1.27.0
  < Date: Thu, 15 Aug 2024 19:32:50 GMT
  < Content-Type: text/html
  < Content-Length: 145
  < Location: http://127.0.0.1/account/%0D%0AX-Foo:%20CLRF      <-------Appears to have worked.
  < Connection: keep-alive
  <
  <html>
  <head><title>302 Found</title></head>
  <body>
  <center><h1>302 Found</h1></center>
  <hr><center>nginx/1.27.0</center>
  </body>
  </html>
  * Connection #0 to host 127.0.0.1 left intact

Changing regular expressions to enable single-line mode prevents the possibility of any confusion being introduced by newline characters. This is just an FYI as I thought it was interesting to see issues raised in the past by others and what suggestions were given.

Last Thoughts:

First of all, I would like to thank Michael Hedges and Parker Green, both from F5 Networks for bringing this to our attention.

As shown in the examples and the article written by the researcher, these types of attacks are not extremely difficult to carry out and can have very significant ramifications in specific scenarios.

As such, taking this into account when setting up a site is key. This would include the configuration of pages to use cache controls and which vendors to use for both web servers as well as web caching proxies. The article I referenced at the beginning does a good job of breaking down how each vendor handles different scenarios. That makes for a great reference point to start with.

Published Sep 09, 2024

Version 1.0

security