bot
7 TopicsGoogle bot Unreachable on 301 redirect iRules
Hello, premise: I'm not managing directly the F5, I'm just the web designer of the company, so I'm pretty new with this kind of stuff; I’m migrating all European websites (ccTLDs) of the company I work for to a global one (.com), and I have some issues with the google boot and 301 redirects. For example, what I'm doing: Example.fr > example.com/fr Example.co.uk > example.com/en-gb …etc.. My system eng. suggested me to implement the redirect with F5 iRules. This means that the actual flow is: Local domain name DNS changed to the F5 IP, inside F5 I put the iRules to redirect the pages to the new global website: i.e. example.fr > F5 > script for the redirect 301 > example.com/fr The iRules code I used is the following: when HTTP_REQUEST { if { ( [HTTP::host] eq "www.kerrdental.co.uk") or ( [HTTP::host] eq "kerrdental.co.uk" ) } { if { ( [HTTP::uri] contains "/AccessoriesForFillingMaterials/AmalgamAccessories/AmalgamGuns" ) } { HTTP::respond 301 Location "https://www.kerrdental.com/en-uk/dental-restoration-products/amalgam-guns-accessories" } elseif { ( [HTTP::uri] contains "/AccessoriesForFillingMaterials/CompositeRestorativeSystems/CompoRoller" ) } { HTTP::respond 301 Location "https://www.kerrdental.com/en-uk/dental-restoration-products/comporoller-accessories" } } } Now, the problem I have concerns the “Change of address” feature in Google Search Console (Webmaster tool): Search Console notify me that “Cannot confirm that 301-redirects work properly, we couldn’t crawl your site. Check that Googlebot can crawl the root of your site using the Fetch as Google tool” Then, when I go under “do Fetch as Google” the system returns me the status “Unreachable”. When I use RexSwain on the HTTP version of the old URL, I get a 503. Receiving Header: HTTP/1.1·503·Service·Unavailable(CR)(LF) Content-Type:·text/html;·charset=UTF-8(CR)(LF) Content-Length:·931(CR)(LF) Connection:·close(CR)(LF) P3P:·CP="CAO·PSA·OUR"(CR)(LF) Expires:·Thu,·01·Jan·1970·00:00:00·GMT(CR)(LF) Cache-Control:·no-store,·no-cache,·must-revalidate,·post-check=0,·pre-check=0(CR)(LF) Pragma:·no-cache(CR)(LF) (CR)(LF) Can someone help me to understand why google boot cannot find anything? Thank you in advance, Best regardsSolved517Views0likes2CommentsWhy does the Local Traffic policy allow Bot profile to be selected but the iRule can't ?
When I attach DOS and BOT profiles with local traffic policy or iRule I always need a default BOT and DOS profile even when I have a default rule that catches all the traffic. That is one thing but the strangest thing is when I decide to attach a Bot profile with iRule it does not work but the Local traffic policies allow this. I will need to test this but is really strange. This is the first time something is only possible with Local Traffic Policies but I will have to test if it works 🙂Solved1.2KViews0likes2CommentsF5 blocking my webpage that works as monitor of Sites hosted behing F5
Hi Guys, we recently enable ASM module on F5 in evualtion/learning mode only and we have one website that is hosted behind the F5 lTM . Once the ASM module is activated my customer web site hosted in a different data center in Azure cannot get thought F5. this website acts as a web monitor and every 5 min it is monitoring the site hosted being the F5. I get a blank page which looks like this. My web monitor is doing HTTP web request and sometimes this happens. Ther is no disturbance of the site hosted in LTM pools. In my google Serach it relieves the problem might be in ASM module trying to block. Can you guys provide some pointer how to resolve this issue. <!DOCTYPE html> <html><head> <meta http-equiv="Pragma" content="no-cache"/> <meta http-equiv="Expires" content="-1"/> <meta http-equiv="CacheControl" content="no-cache"/> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <link rel="shortcut icon" href="data:;base64,iVBORw0KGgo="/> <script> (function(){ window["bobcmn"] = "11111011101010200000002200000005200000000289895ae4200000096300000000300000000300000006/TSPD/300000008TSPD_101300000005https3000000b0081b93fe10ab20006f0e8f1c61960cb6df13226d973e4b69e019690083a6fd29acdda2b6f1b2f5dd0805bbb5290a280019bbf7f5e3c12d280528f7ff9915458e1d0c71804c667eac9e06aa4ea740e68a5b754f765c6ef008200000000200000000"; Regards Sunil1.5KViews0likes1CommentDevice ID - Bot/Dos Profile
Will F5 generate any device id to the client if it is still in transparent mode ? Will my Device ID work in DOS profile if my Bot profile is still in transparent mode ? As for device id to work in context of DOS profile it must be configured under Bot profile as said so.Solved1.6KViews0likes8CommentsBot Profile - Transparent Mode ?
Why do we need transparent mode in Bot profile ? I do understand for signatures to be staged part but do they also learn about Trusted , Untrusted Bot , Suspicious Browser, Malicious Bot which is the reason we need to first place in under transparent mode ?Solved618Views0likes4CommentsCan ASM do something with people (or bot) who access website too rapidly.
Hi We have website for people to download file and show some report. But we found some people (Mostly bot) try to download file or see report too many time per second. (some bot access 10 time per second, some people download 10 file in 10 second.) Is ASM can help us do something about this people (or bot) ? We currently using ASM/LTM version 12. Thank you213Views0likes1CommentIP Intelligence - The Scanners category
Hi, One of the categories that can be enabled to monitor or block with IP Intelligence via the ASM is "Scanners". The definition for this reads "The Scanners category includes all reconnaissance, such as probes, host scan, domain scan, and password brute force." My question is, if this category is enabled and set to block, will it block search engine bots? Will any category in IP Intelligence block search engine traffic? As you can imagine, this is not a desired outcome. Thank you220Views0likes1Comment