robots metatag and nofollow attribute in Irule

You can do this at least two ways:

• You can specify the blocked pages/URIs in the robots.txt file. Example:

  when HTTP_REQUEST {     
      if { [string tolower [HTTP::uri]] equals "/robots.txt" } { 
          HTTP::respond 200 content "User-agent: *\nDisallow: /cgi-bin/\nDisallow: /tmp/"             
      }         
  }
  

Ref: http://www.robotstxt.org/robotstxt.html

• You can insert a meta tag into a page. Probably the easiest way to do that would be with a STREAM profile. Something like the following:

  when HTTP_REQUEST {        
      STREAM::disable
      HTTP::header remove "Accept-Encoding"
      if { [class match [string tolower [HTTP::uri]] starts_with my_nofollow_dg] } {
          set nofollow 1
      }
  }
  when HTTP_RESPONSE {
      if { ( [info exists nofollow] ) and ( [HTTP::header Content-Type] contains "text" ) } {
          unset nofollow
          STREAM::expression {@@@}
          STREAM::enable
      }
  }
  

where "my_nofollow_dg" is a string-based data group containing the no-follow URIs. Example:

    "/cgi-bin/" := 1
    "/tmp/" := 1

On HTTP request, if the requested URI matches a value in the data group, a temporary local variable is set. On HTTP response, if that variable exists, use the STREAM profile to replace the end HEAD tag with a new robots META tag and the end HEAD tag.