Apache CVE-2022-31813 and the hop-by-hop header mechanism

I’ve had a few questions about this particular CVE, and I think it is worth talking about because I think hop-by-hop header handling is not something many people are all that familiar with, and also because I personally think this is a very dubious CVE – I would argue that it should never have been granted a CVE in Apache at all!

 

What are hop-by-hop headers?

We all know what HTTP request headers are, right? They are metadata set in an HTTP request by a client and sent to a server – simple. Except HTTP isn’t that simple at all, the protocol has to deal with proxies and caches, be robust and yet still able to communicate the right things to the right systems in the chain.

 

For that reason, RFC2616 actually defines two different kinds of headers (see https://datatracker.ietf.org/doc/html/rfc2616#section-13.5.1

• End-to-end headers, which must be transmitted to the ultimate recipient of a request or response, stored in intermediate caches and transmitted in any response served from a cache. [emphasis mine]
• Hop-by-hop headers, which are meaningful only to a single transport-level connection, and which must not be stored by caches or forwarded by proxies.

Why the distinction? As you can probably work out from the list of default hop-by-hop headers in the RFC, these headers describe things about the request or response that are only meaningful for the system you are directly talking to – for example, a client might want to tell a proxy that the connection should be kept alive (“Connection: Keep-Alive”), but it is for the proxy to determine how it wishes to handle the connection to the next hop not the client. In contrast, Cookies set by a client are End-to-End because the client needs those to be delivered to the target server regardless of intermediate proxies.

 

Great, so what is the problem?

 

Flexibility and future-proofing

In order to make the RFC somewhat future-proof, RFC2616 includes the following innocent-looking note:

Other hop-by-hop headers MUST be listed in a Connection header, (section 14.10) to be introduced into HTTP/1.1 (or later).

What that means in practice is that you can list any header in the Connection header, and an intermediate proxy should strip it; this is the desired behaviour for hop-by-hop headers, but having a user-defined way to alter proxy behaviour means that the RFC has also unwittingly introduced a mechanism that can be abused.

 

To use the example from CVE-2022-31813, a request from the client would look like this:

GET / HTTP/1.1
Host: 192.0.2.1
X-Forwarded-For: 198.51.100.1
Connection: Close, X-Forwarded-For

When the intermediate proxy (Apache, in this example) handles the request, it will strip any headers defined in the Connection header as being hop-by-hop and forward the request on looking like this:

GET / HTTP/1.1
Host: 192.0.2.1
Connection: Close

(Note that the proxy is quite free to also use Connection: Keep-Alive, if it wishes!)

 

And that means?

What CVE-2022-31813 says is that if the ultimate target system uses X-Forwarded-For (or any other X-Forwarded-*) header for the purposes of authentication then there is the possibility that this authentication may be bypassed if Apache strips the headers when acting as a proxy.

So why do I think this is controversial?

 

For one, hop-by-hop header handling is defined by the RFC; it is a desired, useful, feature of HTTP/1.1 without which proxies would be far less flexible and potentially broken.

For another, in my opinion, the vulnerability here is not in Apache but rather the target application which must be making some assumptions and ‘failing open’ if a header it relies on for authentication is not present. That is bad design, but bad design on the part of the application, not Apache!

 

Finally, an example..

One last thing I need to describe is at least a fictitious real-world example of how this might be abused.

Imagine a web application which allows access based on the IP address of the user, and imagine that the application was built to handle the possibility of proxied requests where the true client IP would be in X-Forwarded-For but also (for reasons) also implicitly trusts the internal network where everyone has access.

In this scenario, the attacker, with an IP address of 203.0.113.10, sends a request to the proxy at 192.0.2.1, the proxy forwards it to the server with a source IP address of 10.10.10.10 and a destination of 10.10.10.20, and the application sees a request which looks like this:

GET / HTTP/1.1
Host: www.example.com 
X-Forwarded-For: 203.0.113.10
Connection: Close

The application examines the ‘client IP’ (taken from X-Forwarded-For), compares it to the ACL and denies access.

If the attacker specifies that X-Forwarded-For is a hop-by-hop header, however, the application might only see the following request as originating from 10.10.10.10 (the proxies internal address):

GET / HTTP/1.1
Host: www.example.com 
Connection: Close

In this case, the application examines the client IP as sent at Layer 4, compares 10.10.10.10 against the ACL and allows access.

Clearly this is terrible application design in something facing the internet, so why the CVE against Apache? (Rhetorical question!)

 

Other, possibly more realistic scenarios would be an application which builds the client IP address into a cookie or access token and later checks that the token is being sent by the same client IP (to prevent cookie stealing/impersonation attacks) using X-Forwarded-* header contents, but which assumes the token to be valid in the absence of such headers. Again, this is poor application design, not poor design on the part of Apache, in my humble opinion.

 

What to do about it

If you are a BIG-IP ASM, BIG-IP Advanced WAF or F5 NGINX App Protect customer, make sure you have signature 200014019 enabled in your policies, which will alert or block (depending on configuration) requests with custom hop-by-hop headers defined.

 

If you’re a BIG-IP LTM user, iRules are your friend here. With iRules it is relatively trivial to sanitize headers – just remember that, by default, many HTTP::header operations happen only on the last header if there are multiple, identically named headers, and so you need to handle the possibility of multiple Connection headers somehow. Something like this should get you well under way:

when RULE_INIT {
  set static::debug 1
}

when HTTP_REQUEST {
  if { [HTTP::header count "Connection"] > 1 } {
    # More than one conneciton header found, strip them all and assume Close
    if { $static::debug eq 1 } {
      log local0. "HTTP request containing more than one Connection header found"
    }

    HTTP::header remove "Connection"
    HTTP::header insert Connection Close
  }

  if { [HTTP::header value "Connection"] contains "," } {
    if { [string tolower [HTTP::header value "Connection"]] contains "close" } {
      # Connection header contained Close, sanitize to _only_ Close
      if { $static::debug eq 1 } {
        log local0. "HTTP request containing Close and hop-by-hop directive found"
      }

      HTTP::header remove "Connection"
      HTTP::header insert Connection Close
    }

    if { [string tolower [HTTP::header value "Connection"]] contains "keep-alive" } {
      # Connection header contained Keep-Alive, sanitize to _only_ Keep-Alive
      if { $static::debug eq 1 } {
        log local0. "HTTP request containing Keep-Alive and hop-by-hop directive found"
      }

      HTTP::header remove "Connection"
      HTTP::header insert Connection Close
    }
  }
}

 

 

Summary

My belief is that this is a rather spurious CVE against Apache given that the vulnerability must exist in software behind the proxy server, and Apache is operating precisely as per the RFCs. But, if you are concerned, you can protect your services using BIG-IP Advanced WAF or NGINX App Protect signatures, BIG-IP LTM iRules or one of F5 Distributed Cloud Services offerings.

If you want to read more about how hop-by-hop header handling can be abused then I recommend reading this research from 2019 – it covers far more than just the scenario outlined by this particular CVE, and dives deeper into the topic than I have here: https://nathandavison.com/blog/abusing-http-hop-by-hop-request-headers

If you have any questions, don't hesitate to reach out!