For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

dla06060's avatar
dla06060
Icon for Nimbostratus rankNimbostratus
Mar 15, 2023

Append text to incoming robots.txt request

We have had problematic crawling with bingbot and would like to use the crawl directive in the robots.txt: User-agent: bingbot Crawl-delay: 1 Since the robots.txt file varies by different websites ...
application delivery
Paulius's avatar
Paulius
Icon for MVP rankMVP
Mar 15, 2023

dla06060 Correct me if I'm wrong but it seems like you would like the F5 to modify the file that is being sent? If so I do not believe this is something that is possible but you can have the F5 send a file of its own by using the following. This would essentially move the robots.txt file from your servers to the F5 directly. Now I do not believe this to be the best solution to your problem but I believe it is the solution that you are asking about. You should ideally resolve this on the server as to limit what the F5 has to do so it can focus on performing load balancing tasks rather than bandaids for server side issues.

https://techdocs.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/ltm-implementations-11-6-0/29.html

dla06060's avatar
dla06060
Icon for Nimbostratus rankNimbostratus
Mar 15, 2023

Paulius that's correct. Ultimately all incoming F5 requests would append the text to the end of the robots.txt file request.

  • Paulius's avatar
    Paulius
    Icon for MVP rankMVP
    Mar 15, 2023

    dla06060 I don't believe you can add to the file on the response back from the server but alternatively you can use what I referenced above but it really should be performed on the server.