Forum Discussion
How can I see if Google is hitting us?
You could try rate limiting Google requests based on the user-agent header value, but I think as Steve Iveson pointed out on the codeshare example, it may affect your Google site ranking.
It might be better to send a 503 response from the iRule when you want to block access to a search engine spider:
http://googlewebmastercentral.blogspot.com/2011/01/how-to-deal-with-planned-site-downtime.html
Outages that are not clearly marked as such can negatively affect a site’s reputation. ... it’s better to return a 503 HTTP result code (Service Unavailable) which tells search engine crawlers that the downtime is temporary.
Aaron
Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com