Forum Discussion
iAPP template for creating multiple VIP's
Thanks before
- Erik_NovakMay 12, 2020
Employee
In terms of RFC2616 compliance, the empty From: header in your example is probably harmless, but in some cases headers with empty values can cause errors in some parsers. That is why it triggers a violation. You can turn off the block flag for the violation "Header name with no header value" if you determine it is causing a false positive. You have control over the blocking action for every single violation on the Learning and Blocking Settings page. According to RFC, the From request-header field, if given, SHOULD contain an Internet e-mail address for the human user who controls the requesting user agent. The address SHOULD be machine-usable, as defined by "mailbox" in RFC 822 [9] as updated by RFC 1123. Again, probably not malicious but informative about the clients that are accessing your app.
- So does Google's crawler actually send it with an empty From: or not? Any clue? It could just be another scraper saying that its Google.
Hi Drew had similar issue client is buying these Google services but for some Google crawlers the from header was empty.
This was of course spoiling the WAF logs with false alerts so what we did is to strip the empty header with Irule when it comes in (bit nasty)
The another thing that should be improved is to mark this BOT as trusted as by default it is untrusted and you cannot overwrite that according to F5 support. So I agree with you this should be improved cause when you acquire their services it should be marked as trusted in my opinion.
- Erik_NovakMay 14, 2020
Employee
Without some forensic data, it is hard to say based on that single example. The User-Agent string looks legit, but is easily spoofed. I am not an expert on Google's bots, but sending an empty header like that is certainly atypical from what we would consider normal browsing behavior. You could try implementing a bot defense profile, and then allow bots at your discretion. Bot defense will challenge all bots for which you don't specify an exception and prevent them from scraping your application.
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com