dla06060
Mar 15, 2023Nimbostratus
Append text to incoming robots.txt request
We have had problematic crawling with bingbot and would like to use the crawl directive in the robots.txt:
User-agent: bingbot Crawl-delay: 1
Since the robots.txt file varies by different websites we host and each is unique, we would like to just append the above to end of the robots.txt when we see the request.
Would like to know how to implement the append at the end of the request so the response back to the client would include the bingbot crawl-delay.
Thank you in advance!