Forum Discussion

Christoph_Fris1's avatar
Christoph_Fris1
Icon for Nimbostratus rankNimbostratus
Jul 10, 2017

How to allow Google to crawl my Site, when DOS Profile is active?

Hello all,

 

we activated the DDos Protection on our F5 Cluster, but after that Google is no longer able to crawl our site. Although I already set all "Google" Signatures on the "Whitelist".

 

But always when I'm checking the crawl status the Google Bot get's this response:

 

...

Please enable JavaScript to view the page content.

And here is the config from our current Profile:

 

security dos profile Homepage {
    app-service none
    application {
        Homepage {
            bot-defense {
                browser-legit-captcha disabled
                browser-legit-enabled disabled
                mode during-attacks
            }
            bot-signatures {
                categories {
                    "DOS Tool" {
                        action block
                    }
                    "E-Mail Collector" {
                        action block
                    }
                    "Exploit Tool" {
                        action block
                    }
                    "Network Scanner" {
                        action block
                    }
                    "Search Engine" {
                        action report
                    }
                    "Spam Bot" {
                        action block
                    }
                    "Vulnerability Scanner" {
                        action block
                    }
                    "Web Spider" {
                        action block
                    }
                    "Webserver Stress Tool" {
                        action block
                    }
                    Spyware {
                        action block
                    }
                }
                check enabled
                disabled-signatures {
                    "Facebook External Hit" { }
                    "Google AdsBot" { }
                    "Google Desktop" { }
                    "Google Feedfetcher" { }
"Google Page Speed Insights" { }
                    "Google Translate" { }
                    "Google favicon" { }
                    "Nokia-WAPToolkit.\* googlebot" { }
                    AppEngine-Google { }
                    Bing { }
                    Google { }
                    Google-Adwords-Instant { }
                    Google-Calendar-Importer { }
                    Google-Sitemaps { }
                    GoogleWebLight { }
                    Google_Analytics_Snippet_Validator { }
                    Java { }
                    Mediapartners-Google { }
                    YahooSeeker { }
                }
            }
            captcha-response {
                failure {
                    body "You have entered an invalid answer for the question. Please, try again.

%DOSL7.captcha.image% %DOSL7.captcha.change%

What code is in the image\?
%DOSL7.captcha.solution%

%DOSL7.captcha.submit%"
                }
                first {
                    body "This question is for testing whether you are a human visitor and to prevent automated spam submission.

%DOSL7.captcha.image% %DOSL7.captcha.change%

What code is in the image\?
%DOSL7.captcha.solution%

%DOSL7.captcha.submit%"
                }
            }
            ip-whitelist {
             xxx.xxx.xxx.xxx/xx    { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
            }
stress-based {
                mode blocking
            }
            tcp-dump {
                record-traffic enabled
            }
            tps-based {
                device-client-side-defense enabled
                device-rate-limiting enabled
                ip-client-side-defense enabled
            }
        }
    }
    partition Common
    whitelist none
}

Maybe you have a hint for me how to solve this. Current Big-IP version: 12.1.2 - ASM Signatures: v12.1.2/ASM-SignatureFile_20170403_145743

 

Thanks, Christoph

 

  • nathe's avatar
    nathe
    Icon for Cirrocumulus rankCirrocumulus

    Christoph - i don't know if this will specifically sort your issue out, but i know that BOT defence needs the BIG-IP to have a DNS server configured. Do you have one setup? Not sure whether the sympton of not having one is the same as your issue, but thought i'd throw it out there. Something you can tick off pretty quickly anyway.

     

    Let us know,

     

    N

     

  • So I opened a ticket for this topic and get this as solution from F5: 1) For this feature to be fully functional F5 recommend to configure a DNS resolver 2) We will recommend to configure a forward zone with "." in FQDN name 3) Also we will recommend to configure the DoS profile on "Bot Signatures", with "Report" enabled for the "Search Bot" and "Search Engine" categories. Remove the Bot Signature List since this overrides the configured actions for the bot signature categories. 4) To validate/verify if this change solves the issue, You can use the google tool to test your website https://support.google.com/webmasters/answer/6065812?hl=en

     

    Now everything is working. Hope this answer will also help others with similar issues

     

    Regards, Christoph

     

  • Hello Christopher,

     

    I'm having this same issue. I'm wondering if you can describe your routing for your DNS resolvers? I expect them to utilize the management interfaces routing table if no route exists in the TM route subtable, but since I can't pcap that interface I can't prove that's what's happening. If I add a route to the TM route subtable, I see packets leave in a capture as expected. Ultimately I've got this setup as F5 recommends, but I'm not having any luck resolving the issue. Oddly, DNS resolutions via bash and tmsh are working properly for crawlers. Any insight you could share would be greatly appreciated.

     

    Thanks