Forum Discussion

W__Ho_172333's avatar
W__Ho_172333
Icon for Nimbostratus rankNimbostratus
Aug 17, 2015

Remove server banner when block the robots

Hi all,   I have a requirement to block the robots. I have the following code that works just fine:   when HTTP_REQUEST { if { .... } some actions.... elseif { [string ...
Michael_61033's avatar
Michael_61033
Icon for Nimbostratus rankNimbostratus
Aug 18, 2015

Use the "noserver" option inside the respond, that should do the trick

when HTTP_REQUEST { 
      if { .... }{   
            some actions.... 

     } elseif {[string tolower [HTTP::uri]] equals "/robots.txt"}{

             HTTP::respond 200 noserver content "User-agent: *\r\nDisallow: /" 
             log local0. "Attempt by [IP::client_addr] crawling our site" 
      }
}
  • W__Ho_172333's avatar
    W__Ho_172333
    Icon for Nimbostratus rankNimbostratus
    Aug 18, 2015
    Hi Michael, I did what you suggested. However, now when pointing to the url https://mysite.domain.com/robots.txt I received the error message stating: "Problem loading page" Secure Connection Failed The connection to the server was reset while the page was loading. The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. Any thoughts? Thank you.