Forum Discussion
How can I see if Google is hitting us?
You could try rate limiting Google requests based on the user-agent header value, but I think as Steve Iveson pointed out on the codeshare example, it may affect your Google site ranking.
It might be better to send a 503 response from the iRule when you want to block access to a search engine spider:
http://googlewebmastercentral.blogspot.com/2011/01/how-to-deal-with-planned-site-downtime.html
Outages that are not clearly marked as such can negatively affect a site’s reputation. ... it’s better to return a 503 HTTP result code (Service Unavailable) which tells search engine crawlers that the downtime is temporary.
Aaron
Help guide the future of your DevCentral Community!
What tools do you use to collaborate? (1min - anonymous)Recent Discussions
Related Content
* Getting Started on DevCentral
* Community Guidelines
* Community Terms of Use / EULA
* Community Ranking Explained
* Community Resources
* Contact the DevCentral Team
* Update MFA on account.f5.com