Forum Discussion

Parveez_70209's avatar
Parveez_70209
Icon for Nimbostratus rankNimbostratus
Oct 03, 2013

How to hide URL( Virtual-Servers) which are exposed over Internet from search engines using robots.txt?

Hi,   Kindly guide into this, we wanted to hide our Virtual-server which is exposed into internet from search engines using robots.txt?   I was also going through the below link in which Kevin ...
Parveez_70209's avatar
Parveez_70209
Icon for Nimbostratus rankNimbostratus
Oct 04, 2013

Yes,we are planning to test this with the below URL which covers all:

 

when HTTP_REQUEST { if { [string tolower [HTTP::uri]] equals "/robots.txt" } { HTTP::respond 200 content "User-agent: *\nDisallow: /" } }

 

So, nothing specific to do from server/application team right for the objective to achieve.

 

Thanks and Regards Parveez