For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

randyrue's avatar
randyrue
Icon for Nimbostratus rankNimbostratus
Mar 03, 2022

robots.txt rule?

Hello, Bear with me if this has been answered elsewhere, I've found threads that seem similar but nothing exactly like what I'm trying to do and none of the related examples seem to work. I'm tryin...
application delivery
event
randyrue's avatar
randyrue
Icon for Nimbostratus rankNimbostratus
Mar 04, 2022

Not my ideal solution as it requires me to modify any iRule that my robots.txt rule will be in front of. That is, the robots.txt rule remains

when HTTP_REQUEST {
  if { [HTTP::uri] == "/robots.txt" } {
    HTTP::respond 200 content [ifile get robots.txt]
  }
}

and my second rule (in this case the gopher rule) becomes:

when HTTP_REQUEST {
  if { [HTTP::has_responded] } { return }
  if { [HTTP::uri] == "/gopher.jpg" } {
    HTTP::respond 200 content [ifile get gopher.jpg]
  }
}

Or is there a way to do this entirely in the robots rule?

  • JoshBecigneul's avatar
    JoshBecigneul
    Icon for MVP rankMVP
    Mar 08, 2022

    Well the simple answer might be that you would just put the "if { [HTTP::has_responded] } { return }" text into the robots iRule and just put that rule after any other rules, assuming the other rules wouldn't prevent the execution of the robots iRule. But in some cases you may need to edit the others to include that statement.