Rate limiting Search Spiders

Hi there, we are a SaaS company and therefore don't have control of all our communities submissions to Search engines. As well each of our Communities we allow the ability for them to have a robots.txt file as well. These robots.txt files are handled and server up with Code and not sitting in the root of the typical web server.

 

 

Here you can see why I need to control speed from the F5 for robots.

 

 

When searching on this site it seems this thread beginning is what I am after. Now I am just hoping someone has a working examples and can help me.

 

 

 

thanks for your input.