F5 is upgrading its customer support chat feature on My.F5.com. Chat support will be unavailable from 6am-10am PST on 1/20/26. Refer to K000159584 for details.

Forum Discussion

randyrue's avatar
randyrue
Icon for Nimbostratus rankNimbostratus
Mar 03, 2022

robots.txt rule?

Hello, Bear with me if this has been answered elsewhere, I've found threads that seem similar but nothing exactly like what I'm trying to do and none of the related examples seem to work. I'm tryin...
application delivery
event
randyrue's avatar
randyrue
Icon for Nimbostratus rankNimbostratus
Mar 04, 2022

Not my ideal solution as it requires me to modify any iRule that my robots.txt rule will be in front of. That is, the robots.txt rule remains

when HTTP_REQUEST {
  if { [HTTP::uri] == "/robots.txt" } {
    HTTP::respond 200 content [ifile get robots.txt]
  }
}

and my second rule (in this case the gopher rule) becomes:

when HTTP_REQUEST {
  if { [HTTP::has_responded] } { return }
  if { [HTTP::uri] == "/gopher.jpg" } {
    HTTP::respond 200 content [ifile get gopher.jpg]
  }
}

Or is there a way to do this entirely in the robots rule?

JoshBecigneul's avatar
JoshBecigneul
Icon for MVP rankMVP
Mar 08, 2022

Well the simple answer might be that you would just put the "if { [HTTP::has_responded] } { return }" text into the robots iRule and just put that rule after any other rules, assuming the other rules wouldn't prevent the execution of the robots iRule. But in some cases you may need to edit the others to include that statement.