iRule Security 101 - #06 - HTTP Referer

In this article, I'm going to talk about the HTTP "Referer" header, how it's used, and how you can use iRules to ensure that an access request to a website is coming from where you want it to come from. Other articles in the series:

• iRule Security 101 – #1 – HTTP Version
• iRule Security 101 – #02 – HTTP Methods and Cross Site Tracing
• iRule Security 101 – #03 – HTML Comments
• iRule Security 101 – #04 – Masking Application Platform
• iRule Security 101 – #05 – Avoiding Path Traversal
• iRule Security 101 – #06 – HTTP Referer
• iRule Security 101 – #07 – FTP Proxy
• iRule Security 101 – #08 – Limiting POST Data
• iRule Security 101 – #09 – Command Execution

First, let me say that I know that "Referer" is misspelled.  For some reason, the authors of the HTTP specification (RFC 2616, section 14.36) didn't run a spell checker on the specification and now that every browser and web server has implemented this with the wrong spelling it's too late to change it.  Take a look at the definition for it on dictionary.com and you'll see for yourself.  Nothing like a dictionary 'dissing an Internet spec...

Once you can get past the misspelling, the HTTP "Referer" header is defined as the following (RFC 2616, section 14.36)

14.36 Referer 

The Referer[sic] request-header field allows the client to specify,
for the server's benefit, the address (URI) of the resource from
which the Request-URI was obtained (the "referrer", although the
header field is misspelled.) The Referer request-header allows a
server to generate lists of back-links to resources for interest,
logging, optimized caching, etc. It also allows obsolete or mistyped
links to be traced for maintenance. The Referer field MUST NOT be
sent if the Request-URI was obtained from a source that does not have
its own URI, such as input from the user keyboard.

So basically, when ever you click on a link from a website causing new HTTP request to be made, the URI of the website you are on will be passed in the HTTP request in the form of a HTTP header with the name of "Referer" and a value containing the source URI.  Why is this important from a security perspective?  I'll give just one example attack and a way to use Referer headers to help block against it. 

With the massive uptake of blogging by users on the Internet, comments are a useful way to get feedback on your ideas.  Unfortunately blog spam, as it's called, has been on the rise.  Several ways have been developed to protect against blog spam including comment moderation, CAPTCHA (you know, when you type out the text that is displayed in randomly generated images), as well as online dynamic services such as Akismet that process the content of comments in a very similar way to common email SPAM services.  CAPTCHA is the most common form of defense but it is not fool proof and spammers have found ways to build programs to defeat this system.

So how does this fit with Referers?  I'll get to that in just a minute...

If you set a policy on your blog that only the comment form can be accessed by clicking on a feedback link on your blog, then you can make use of this fact by denying all requests that do not contain the URI of your blog post in the Referer header.  Sure, there are ways to bypass this since HTTP headers are easily programmed into any HTTP client program.  But, there are ways to trick the client into thinking that the post succeeded when it really didn't.

Let's take a look at an example. 

http://www.mycoolblog.com/ - blog site in question
http://www.mycoolblog.com/first_post - blog post page that is to be commented on.
http://www.mycoolblog.com/PostComment.aspx - Comment post form.

Legitimate commenter's will first vist the blog post page and then fill in the comment information and submit it to the PostComment.aspx form.  Spammers will try to bypass this step by pulling in these images into a client program, try to determine the CAPTCHA image's text, and then formulate a HTTP POST command directly to the PostComment.aspx page.  By enforcing that a Referer header from the same blog site comes in the PostComment.aspx request, we can block out those spammers.

when HTTP_REQUEST {
  switch -glob [HTTP::header "Referer"] {
    "http://www.mycoolblog.com/*" {
      # Allow Request to go through...
    }
    "" {
      HTTP::respond 200 content ""
    }
    default {
      HTTP::redirect [HTTP::header "Referer"]
    }
  }
}

Basically any request coming from http://www.mycoolblog.com will be allowed through.  Any request with a empty Referer header will be immediately returned with a HTTP 200 response to trick the client that a successful attempt was made, and any other Referer's will be redirected back to the referral site.

Caveats:

This is by far not a universal blog spam solution as each blogging engine handles comments differently.  Some have a different URI for comment posting (as illustrated above) and others use POST data values on the same application page as the blog posting to indicate comment submissions.  Also, it is easy for clients to spoof referer values by manually adding the header in the requests.  But, it is a good start for those automated bots out there that are just searching for blogs to send their unwanted content to.  Also, this solution does not support Trackback/Pingback spam as those solutions typically are programmatic submissions from references in other blogs.

Conclusions:

Blog Spam was just an example of the type of application security issue that could be addressed by making use of the HTTP Referer header.  Hopefully this article has provided some food for thought into how you can use the Referer header to your advantage in protecting your applications.