We had to create a new App Security policy because technical support couldn't tell us why our old one wasn't doing anything after several days of working with them.
I am seeing some events for a rule that says: 'Header name with no header value'
I'm not entirely certain how this poses a risk and it seems like it's blocking Google from crawling our website which makes it suboptimal. Are there a list of things that F5 thinks are security issues that just break your website that you have to disable?
In terms of RFC2616 compliance, the empty From: header in your example is probably harmless, but in some cases headers with empty values can cause errors in some parsers. That is why it triggers a violation. You can turn off the block flag for the violation "Header name with no header value" if you determine it is causing a false positive. You have control over the blocking action for every single violation on the Learning and Blocking Settings page. According to RFC, the From request-header field, if given, SHOULD contain an Internet e-mail address for the human user who controls the requesting user agent. The address SHOULD be machine-usable, as defined by "mailbox" in RFC 822 [9] as updated by RFC 1123. Again, probably not malicious but informative about the clients that are accessing your app.
This was of course spoiling the WAF logs with false alerts so what we did is to strip the empty header with Irule when it comes in (bit nasty)
The another thing that should be improved is to mark this BOT as trusted as by default it is untrusted and you cannot overwrite that according to F5 support. So I agree with you this should be improved cause when you acquire their services it should be marked as trusted in my opinion.
Without some forensic data, it is hard to say based on that single example. The User-Agent string looks legit, but is easily spoofed. I am not an expert on Google's bots, but sending an empty header like that is certainly atypical from what we would consider normal browsing behavior. You could try implementing a bot defense profile, and then allow bots at your discretion. Bot defense will challenge all bots for which you don't specify an exception and prevent them from scraping your application.
