Header name with no header value

[DrewW]

Hi,

We had to create a new App Security policy because technical support couldn't tell us why our old one wasn't doing anything after several days of working with them.

I am seeing some events for a rule that says: 'Header name with no header value'

Basically the request looks like this:

Cache-Control: max-age=88544
Connection: keep-alive
Accept: text/css,*/*;q=0.1
From: 
User-Agent: AdsBot-Google (+http://www.google.com/adsbot.html)

I'm not entirely certain how this poses a risk and it seems like it's blocking Google from crawling our website which makes it suboptimal. Are there a list of things that F5 thinks are security issues that just break your website that you have to disable?

[Erik_Novak]

In terms of RFC2616 compliance, the empty From: header in your example is probably harmless, but in some cases headers with empty values can cause errors in some parsers. That is why it triggers a violation. You can turn off the block flag for the violation "Header name with no header value" if you determine it is causing a false positive. You have control over the blocking action for every single violation on the Learning and Blocking Settings page. According to RFC, the From request-header field, if given, SHOULD contain an Internet e-mail address for the human user who controls the requesting user agent. The address SHOULD be machine-usable, as defined by "mailbox" in RFC 822 [9] as updated by RFC 1123. Again, probably not malicious but informative about the clients that are accessing your app.

[DrewW]

So does Google's crawler actually send it with an empty From: or not? Any clue? It could just be another scraper saying that its Google.

[Marvin]

Hi Drew had similar issue client is buying these Google services but for some Google crawlers the from header was empty.

https://support.google.com/webmasters/thread/110658424/adsbot-google-does-not-include-from-googlebot...

This was of course spoiling the WAF logs with false alerts so what we did is to strip the empty header with Irule when it comes in (bit nasty)

The another thing that should be improved is to mark this BOT as trusted as by default it is untrusted and you cannot overwrite that according to F5 support. So I agree with you this should be improved cause when you acquire their services it should be marked as trusted in my opinion.

[Erik_Novak]

Without some forensic data, it is hard to say based on that single example. The User-Agent string looks legit, but is easily spoofed. I am not an expert on Google's bots, but sending an empty header like that is certainly atypical from what we would consider normal browsing behavior. You could try implementing a bot defense profile, and then allow bots at your discretion. Bot defense will challenge all bots for which you don't specify an exception and prevent them from scraping your application.

[DrewW]

Unfortunately even though we pay F5 a huge sum of money we dont have bot defense.