Forum Discussion

DrewW's avatar
DrewW
Icon for Nimbostratus rankNimbostratus
May 12, 2020

Header name with no header value

Hi, We had to create a new App Security policy because technical support couldn't tell us why our old one wasn't doing anything after several days of working with them. I am seeing some events ...
ASM Advanced WAF
security
DrewW's avatar
DrewW
Icon for Nimbostratus rankNimbostratus
So does Google's crawler actually send it with an empty From: or not? Any clue? It could just be another scraper saying that its Google.
Marvin's avatar
Marvin
Icon for Cirrocumulus rankCirrocumulus
May 09, 2022

Hi Drew had similar issue client is buying these Google services but for some Google crawlers the from header was empty.

https://support.google.com/webmasters/thread/110658424/adsbot-google-does-not-include-from-googlebot-at-googlebot-com-header?hl=en

This was of course spoiling the WAF logs with false alerts so what we did is to strip the empty header with Irule when it comes in (bit nasty)

The another thing that should be improved is to mark this BOT as trusted as by default it is untrusted and you cannot overwrite that according to F5 support. So I agree with you this should be improved cause when you acquire their services it should be marked as trusted in my opinion.