Forum Discussion

maximillean_953

Nimbostratus

Aug 26, 2011

F5 Ltm Asm module and google chrome problem

Hello,

Couple days ago we activate the asm module. It works nice with nice features.

But we have only one problem that we could not overcome.

Problem is related with google chrome browser and with it only. The all others works perfectly.

Random times / google chromes gets a null page response from f5 directly without dispatching the request to the pool but sometimes it does dispath and brings me the correct request.

As an example below captured from wireshark from chrome machine.Clean cookie and history first request from and response from F5. F5 added also "?srpclwjccvhocvho" string to meta url

line somehow.

Request

GET / HTTP/1.1

Host: www.test.com

Connection: keep-alive

Cache-Control: max-age=0

User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding: gzip,deflate,sdch

Accept-Language: en-US,en;q=0.8

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

Response

HTTP/1.1 200 OK

Connection: Close

Pragma: no-cache

Cache-Control: no-cache

Server: HTTP Server 1.0

Content-Type: text/html; charset=UTF-8

Content-Length: 222

html head

meta http-equiv="refresh" content="0;url=http://www.test.com/?srpclwjccvhocvho"

meta http-equiv="pragma" content="no-cache"

meta http-equiv="expires" content="-1"

/head body /body /html

On the other hand another request and response 10 minutes later.

request

GET / HTTP/1.1

Host: www.test.com

Connection: keep-alive

Cache-Control: max-age=0

User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding: gzip,deflate,sdch

Accept-Language: en-US,en;q=0.8

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

response

HTTP/1.1 200 OK

Date: Fri, 26 Aug 2011 14:56:32 GMT

X-Powered-By: PHP/5.3.3-1ubuntu9.5

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

Vary: Accept-Encoding

Keep-Alive: timeout=15, max=800

Connection: close

Content-Type: text/html; charset=UTF-8

Content-Encoding: gzip

The host i try this is not connected to site all day long can not be blocked did not even connect any other pool on f5 either. It doesnot get recorded in non of the logs nothing.

I try everyting cookies,chunck behaviour on profile anything i mean we try anything. But somehow this only occurs on google chrome and nothing else. 30 people tested site with 6 different browsers for more then 40 hours. Client side all done. clear caches try without clearing tried. This behavior is only seen the pool that asm applied and not on the non asm applied pools.

Also the browser requests from chrome and seamonkey

Chrome

Chrome gets this some of the time and some of the time not.

GET / HTTP/1.1

Host: www.test.com

Connection: keep-alive

User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Encoding: gzip,deflate,sdch

Accept-Language: en-US,en;q=0.8

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

HTTP/1.1 200 OK

Connection: Close

Pragma: no-cache

Cache-Control: no-cache

Server: HTTP Server 1.0

Content-Type: text/html; charset=UTF-8

Content-Length: 222

html head

meta http-equiv="refresh" content="0;url=http://www.test.com/?srpclwjccvhocvho"

meta http-equiv="pragma" content="no-cache"

meta http-equiv="expires" content="-1"

/head body /body /html

Seamonkey

Seamonkey gets this all of the time. Which is correct respond from vserver.

GET / HTTP/1.1

Host: www.test.com

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.18) Gecko/20110412 SeaMonkey/2.0.13

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

HTTP/1.1 200 OK

Date: Fri, 26 Aug 2011 14:56:32 GMT

X-Powered-By: PHP/5.3.3-1ubuntu9.5

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

Vary: Accept-Encoding

Keep-Alive: timeout=15, max=800

Connection: close

Content-Type: text/html; charset=UTF-8

Content-Encoding: gzip

tcpdump has no joy. When i get this response on chrome request never reaches the vserver vlan/interface.

Please anyone had same issue as us? Help us.

For tcpdump screenshot. http://i53.tinypic.com/1qhklk.png

app sec

BIG-IP Access Policy Manager (APM)

security

hooleylist
Cirrostratus
Aug 30, 2011
Hi maximillean,

I'd try opening a case with F5 Support on this.

Aaron
maximillean_953
Nimbostratus
Aug 30, 2011
Hi Aaron,

Now the meta refresh F5 sends is not accepted by chrome and for example curl on console but w3m and links named console browsers works perfectly. This problem only occurs on curl and Google Chrome. I also post the simulation of problem on google chrome forums too.

Now the thing is F5s protection technic is nice. sending a meta refresh then forwards another url then the regular site but problem is its not tested well or this working technics did not tested with latest versions of chrome. cause Chrome somehow as default does not do meta refresh automatically so it stops on a null page..

If i open for example firefox while chrome stays on whitepage. firefox works then chrome starts working cause F5 accepts the client after client visit the page with a accurate browser so it undertands this as working browser.

So problem seems like on chrome but needed to fix on F5 side to send something diffent.

This problem alone cost us already around 100k for 3 days. Cause i am from turkei and in turkei there is religious holiday. So we are near to choose different program then F5 asm.

Even with litespeed&ha_proxy with 10g line via conn rate i can cut heavy 1 gbps 1.5 mil request per second attacks easily. But this problem breaks our hearts cause we are F5 lover. We made company sell all netscalers and get f5s but we are in trouble. We contact F5 but in turkey religious holiday so turkish vendor is in vacation too so as you can see we have alot of problems.

Thanks,
Mike_61719
Cirrus
Aug 30, 2011
Your user base is built around Chrome?
maximillean_953
Nimbostratus
Aug 30, 2011
Yes. Around 45% is google chrome which is registered dating sites and coupon like sites. We have around 30+ sites which runs good as i told before with good income. But this is little emergency problem. My supervisor doesnot want litespeed/haproxy solution as i told before we are heavy F5 supporters we exchange all netscalers to f5 this is the only problem we could not come around. We even implement snmp dynamic ratio load balancing on mysqls. It doesnot send query to slave if snmp dyno ratio detects heavy io on slave machine on F5 slave sql load balancing pool. As i told before this is the only issue we could not overcome while working with this machine. Its awesome. We all need a quick solve to this chrome users thats all.
Mike_61719
Cirrus
Aug 30, 2011
What about Safari?
maximillean_953
Nimbostratus
Aug 30, 2011
Iphones/Ipads/ also mac and windows safari doesnot do meta refresh too.

But its secondary for now. Cause we loose most money from chrome.

As you mention, those devices safari and win/macs safari has same problem.

What i dont understand is even linux console links browsers work but chrome does not.

The all rest works perfectly. Cause what i understand is F5 asm module neccesarity for meta refresh to understand the client is not an attacker. But the thing is we also disable url and source ip integrity check and leave only source ip and url ddos detection on anomaly deteciton/ddos window.

I diagnose this thing too long friend. Believe me i am not wrong. The meta refresh came from F5 is not getting refreshed and forward to the meta mentioned url on chrome. I dont understand that how come chrome doesnot get tested while having this much userbase.

If seamonkey browser doesnot work with f5 asm mechanism i understand that but this is google chrome.
Mike_61719
Cirrus
Aug 30, 2011
I'm not doubting you but it is a Google problem. They are ignoring the meta refresh that is being sent down to the browser.

Create a sample page and test with a browser. Chrome along with Safari is ignoring it.
maximillean_953
Nimbostratus
Aug 30, 2011
I did. But if you selling a security product which is for such thing as ddos attack. Must develop a way to support all browsers. I dont think i am wrong by saying this. F5 develops a way to understand the client is real client not an attacker but this does not work with internets biggest companies browser?

is not it odd for you or i am wrong?
hooleylist
Cirrostratus
Aug 30, 2011
Hi maximillean,

Can you try opening a case with F5 Support directly? You could probably explain that your local partner is not working today and get official feedback from F5.

Aaron
Mike_61719
Cirrus
Aug 30, 2011
The only way I can think of helping the situation is to create an irule to detect the chrome browser and force it on over to a separate location without the meta refresh tagging. Outside of that, I would work with google and F5 Support.