Forum Discussion
MihirJoshi_3137
Nimbostratus
NimbostratusAllow user agents to scrape my website
I am trying to write a iRule which allows facebopot to scrape my webpage
when HTTP_REQUEST { if { [HTTP::header "User-Agent"] contains "Valid_Agents_for_Facebook" } { ASM::disable } else { ASM::enable /Common/abc_Sec_Policy } }
Valid_Agents_for_facebook is the list of agents. I tested but not working.
Finally got it working,
when BOTDEFENSE_ACTION { if { [class match [string tolower [HTTP::header "User-Agent"]] contains "Valid_Agents_for_Facebook"] } { BOTDEFENSE::action allow } }
Vijay_ECirrus
Are the values within the datagroup case-sensitive and have you added them as they are expected to show up from facebook ?
MihirJoshi_3137Finally got it working,
when BOTDEFENSE_ACTION { if { [class match [string tolower [HTTP::header "User-Agent"]] contains "Valid_Agents_for_Facebook"] } { BOTDEFENSE::action allow } }