Forum Discussion

dla06060's avatar
dla06060
Icon for Nimbostratus rankNimbostratus
Mar 15, 2023

Append text to incoming robots.txt request

We have had problematic crawling with bingbot and would like to use the crawl directive in the robots.txt: User-agent: bingbot Crawl-delay: 1 Since the robots.txt file varies by different websites ...
application delivery
dla06060's avatar
dla06060
Icon for Nimbostratus rankNimbostratus

Paulius that's correct. Ultimately all incoming F5 requests would append the text to the end of the robots.txt file request.

Paulius's avatar
Paulius
Icon for MVP rankMVP
Mar 15, 2023

dla06060 I don't believe you can add to the file on the response back from the server but alternatively you can use what I referenced above but it really should be performed on the server.