Forum Discussion
dla06060
Nimbostratus
NimbostratusAppend text to incoming robots.txt request
We have had problematic crawling with bingbot and would like to use the crawl directive in the robots.txt:
User-agent: bingbot Crawl-delay: 1
Since the robots.txt file varies by different websites we host and each is unique, we would like to just append the above to end of the robots.txt when we see the request.
Would like to know how to implement the append at the end of the request so the response back to the client would include the bingbot crawl-delay.
Thank you in advance!
- Paulius
MVP
dla06060 Correct me if I'm wrong but it seems like you would like the F5 to modify the file that is being sent? If so I do not believe this is something that is possible but you can have the F5 send a file of its own by using the following. This would essentially move the robots.txt file from your servers to the F5 directly. Now I do not believe this to be the best solution to your problem but I believe it is the solution that you are asking about. You should ideally resolve this on the server as to limit what the F5 has to do so it can focus on performing load balancing tasks rather than bandaids for server side issues.
