Forum Discussion
maximillean_953
Nimbostratus
NimbostratusF5 Ltm Asm module and google chrome problem
Hello,
Couple days ago we activate the asm module. It works nice with nice features.
But we have only one problem that we could not overcome.
Problem is related with google chrome browser and with it only. The all others works perfectly.
Random times / google chromes gets a null page response from f5 directly without dispatching the request to the pool but sometimes it does dispath and brings me the correct request.
As an example below captured from wireshark from chrome machine.Clean cookie and history first request from and response from F5. F5 added also "?srpclwjccvhocvho" string to meta url
line somehow.
Request
GET / HTTP/1.1
Host: www.test.com
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Response
HTTP/1.1 200 OK
Connection: Close
Pragma: no-cache
Cache-Control: no-cache
Server: HTTP Server 1.0
Content-Type: text/html; charset=UTF-8
Content-Length: 222
html head
meta http-equiv="refresh" content="0;url=http://www.test.com/?srpclwjccvhocvho"
meta http-equiv="pragma" content="no-cache"
meta http-equiv="expires" content="-1"
/head body /body /html
On the other hand another request and response 10 minutes later.
request
GET / HTTP/1.1
Host: www.test.com
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
response
HTTP/1.1 200 OK
Date: Fri, 26 Aug 2011 14:56:32 GMT
X-Powered-By: PHP/5.3.3-1ubuntu9.5
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=800
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
The host i try this is not connected to site all day long can not be blocked did not even connect any other pool on f5 either. It doesnot get recorded in non of the logs nothing.
I try everyting cookies,chunck behaviour on profile anything i mean we try anything. But somehow this only occurs on google chrome and nothing else. 30 people tested site with 6 different browsers for more then 40 hours. Client side all done. clear caches try without clearing tried. This behavior is only seen the pool that asm applied and not on the non asm applied pools.
Also the browser requests from chrome and seamonkey
Chrome
Chrome gets this some of the time and some of the time not.
GET / HTTP/1.1
Host: www.test.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.215 Safari/535.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
HTTP/1.1 200 OK
Connection: Close
Pragma: no-cache
Cache-Control: no-cache
Server: HTTP Server 1.0
Content-Type: text/html; charset=UTF-8
Content-Length: 222
html head
meta http-equiv="refresh" content="0;url=http://www.test.com/?srpclwjccvhocvho"
meta http-equiv="pragma" content="no-cache"
meta http-equiv="expires" content="-1"
/head body /body /html
Seamonkey
Seamonkey gets this all of the time. Which is correct respond from vserver.
GET / HTTP/1.1
Host: www.test.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.18) Gecko/20110412 SeaMonkey/2.0.13
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
HTTP/1.1 200 OK
Date: Fri, 26 Aug 2011 14:56:32 GMT
X-Powered-By: PHP/5.3.3-1ubuntu9.5
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=800
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
tcpdump has no joy. When i get this response on chrome request never reaches the vserver vlan/interface.
Please anyone had same issue as us? Help us.
For tcpdump screenshot. http://i53.tinypic.com/1qhklk.png
hooleylistCirrostratus
Hi maximillean,
I'd try opening a case with F5 Support on this.
Aaron
maximillean_953Hi Aaron,
Now the meta refresh F5 sends is not accepted by chrome and for example curl on console but w3m and links named console browsers works perfectly. This problem only occurs on curl and Google Chrome. I also post the simulation of problem on google chrome forums too.
Now the thing is F5s protection technic is nice. sending a meta refresh then forwards another url then the regular site but problem is its not tested well or this working technics did not tested with latest versions of chrome. cause Chrome somehow as default does not do meta refresh automatically so it stops on a null page..
If i open for example firefox while chrome stays on whitepage. firefox works then chrome starts working cause F5 accepts the client after client visit the page with a accurate browser so it undertands this as working browser.
So problem seems like on chrome but needed to fix on F5 side to send something diffent.
This problem alone cost us already around 100k for 3 days. Cause i am from turkei and in turkei there is religious holiday. So we are near to choose different program then F5 asm.
Even with litespeed&ha_proxy with 10g line via conn rate i can cut heavy 1 gbps 1.5 mil request per second attacks easily. But this problem breaks our hearts cause we are F5 lover. We made company sell all netscalers and get f5s but we are in trouble. We contact F5 but in turkey religious holiday so turkish vendor is in vacation too so as you can see we have alot of problems.
Thanks,
Mike_61719Cirrus
Your user base is built around Chrome?- maximillean_953Yes. Around 45% is google chrome which is registered dating sites and coupon like sites. We have around 30+ sites which runs good as i told before with good income. But this is little emergency problem. My supervisor doesnot want litespeed/haproxy solution as i told before we are heavy F5 supporters we exchange all netscalers to f5 this is the only problem we could not come around. We even implement snmp dynamic ratio load balancing on mysqls. It doesnot send query to slave if snmp dyno ratio detects heavy io on slave machine on F5 slave sql load balancing pool. As i told before this is the only issue we could not overcome while working with this machine. Its awesome. We all need a quick solve to this chrome users thats all.
- Mike_61719What about Safari?
- maximillean_953Iphones/Ipads/ also mac and windows safari doesnot do meta refresh too.
But its secondary for now. Cause we loose most money from chrome.
As you mention, those devices safari and win/macs safari has same problem.
What i dont understand is even linux console links browsers work but chrome does not.
The all rest works perfectly. Cause what i understand is F5 asm module neccesarity for meta refresh to understand the client is not an attacker. But the thing is we also disable url and source ip integrity check and leave only source ip and url ddos detection on anomaly deteciton/ddos window.
I diagnose this thing too long friend. Believe me i am not wrong. The meta refresh came from F5 is not getting refreshed and forward to the meta mentioned url on chrome. I dont understand that how come chrome doesnot get tested while having this much userbase.
If seamonkey browser doesnot work with f5 asm mechanism i understand that but this is google chrome. - Mike_61719I'm not doubting you but it is a Google problem. They are ignoring the meta refresh that is being sent down to the browser.
Create a sample page and test with a browser. Chrome along with Safari is ignoring it. - maximillean_953I did. But if you selling a security product which is for such thing as ddos attack. Must develop a way to support all browsers. I dont think i am wrong by saying this. F5 develops a way to understand the client is real client not an attacker but this does not work with internets biggest companies browser?
is not it odd for you or i am wrong? - hooleylistHi maximillean,
Can you try opening a case with F5 Support directly? You could probably explain that your local partner is not working today and get official feedback from F5.
Aaron - Mike_61719The only way I can think of helping the situation is to create an irule to detect the chrome browser and force it on over to a separate location without the meta refresh tagging. Outside of that, I would work with google and F5 Support.