Forum Discussion
marc_wermund_10
Nimbostratus
NimbostratusTCP reset issue
Hello,
I am in the process of testing a rather simple load balancing pool, but am experiencing TCP resets (as verified with a tcpdump during connection)
the virtual server is setup on 10.14.0.154 and the nodes themselves are on 10.14.0.152 and 10.14.0.153
when i hit the nodes directly (HTTP request on port 8000) it works fine, so we know that the back-end setup is okay. the traffic all staying in the 10.14.0/24 subnet.
health monitors for both nodes to http on 8000 show green (up) and the web pool that serves them shows green.
the virtual server shows green as well and listens on port 8000
config is standard
protocol is tcp
protocol profile client is tcp
protocol profile server is use client profile
oneconnect profile is none
http profile is http
address and port translation are both checked
SNAT pool is auto map
everything else is defaults.
we are on 9.1.1 build 54.6
hoolioCirrostratus
That looks like it should work. Is the clientside TCP connection reset immediately? If not, is any serverside connection attempted? Do you have packet filters enabled which could be preventing this from working?
Can you disable the monitors and run a tcpdump like this to see the packets being sent?
tcpdump -ni 0.0 host CLIENT_IP or host 10.14.0.154 or host 10.14.0.153 or host 10.14.0.152
Aaron
marc_wermund_10i can try that, but before i do. i have a capture from yesterday, but the monitors were still on... this is it:15:17:01.405087 IP (tos 0x0, ttl 64, id 19961, offset 0, flags [DF], proto TCP (6), length 60) 2xbh491-u.local.43567 > searchftc.portal.doitbestcorp.com.8000: S, cksum 0xf2ca (correct), 1776643348:1776643348(0) win 5840 15:17:01.439141 IP (tos 0x0, ttl 253, id 41522, offset 0, flags [DF], proto TCP (6), length 64) searchftc.portal.doitbestcorp.com.8000 > 2xbh491-u.local.43567: S, cksum 0x493f (correct), 2236580769:2236580769(0) ack 1776643349 win 4050 15:17:01.439214 IP (tos 0x0, ttl 64, id 19962, offset 0, flags [DF], proto TCP (6), length 52) 2xbh491-u.local.43567 > searchftc.portal.doitbestcorp.com.8000: ., cksum 0x990a (correct), 1:1(0) ack 1 win 92 15:17:01.439929 IP (tos 0x0, ttl 64, id 19963, offset 0, flags [DF], proto TCP (6), length 191) 2xbh491-u.local.43567 > searchftc.portal.doitbestcorp.com.8000: P 1:140(139) ack 1 win 92 15:17:01.466278 IP (tos 0x0, ttl 253, id 41539, offset 0, flags [DF], proto TCP (6), length 40) searchftc.portal.doitbestcorp.com.8000 > 2xbh491-u.local.43567: R, cksum 0x7c6a (correct), 1:1(0) ack 140 win 4189- marc_wermund_10i disabled the monitors at the node level and pool level and get this:
09:51:46.872726 IP (tos 0x0, ttl 64, id 29076, offset 0, flags [DF], proto TCP (6), length 60) 2xbh491-u.local.45971 > searchftc.portal.doitbestcorp.com.8000: S, cksum 0xc6d2 (correct), 2606839639:2606839639(0) win 5840 09:51:46.893574 IP (tos 0x0, ttl 253, id 25507, offset 0, flags [DF], proto TCP (6), length 64) searchftc.portal.doitbestcorp.com.8000 > 2xbh491-u.local.45971: S, cksum 0x52b4 (correct), 851052900:851052900(0) ack 2606839640 win 4050 09:51:46.893644 IP (tos 0x0, ttl 64, id 29077, offset 0, flags [DF], proto TCP (6), length 52) 2xbh491-u.local.45971 > searchftc.portal.doitbestcorp.com.8000: ., cksum 0xa281 (correct), 1:1(0) ack 1 win 92 09:51:46.895231 IP (tos 0x0, ttl 64, id 29078, offset 0, flags [DF], proto TCP (6), length 191) 2xbh491-u.local.45971 > searchftc.portal.doitbestcorp.com.8000: P 1:140(139) ack 1 win 92 09:51:46.939566 IP (tos 0x0, ttl 253, id 25528, offset 0, flags [DF], proto TCP (6), length 40) searchftc.portal.doitbestcorp.com.8000 > 2xbh491-u.local.45971: R, cksum 0x441a (correct), 1:1(0) ack 140 win 4189
basically the same thing happened as far as i can tell. - hoolioThanks for that. The trace with no monitors enabled is just to remove the non-relevant packets from the trace.
So the client side connection looks like it's being set up correctly but once the client sends data, LTM sends a TCP reset. Does LTM have a self IP address on the 10.14.0/24 subnet that the servers are on?
Aaron - marc_wermund_10it does have a self-ip address on that subnet.
but furthermore, we believe we have the issue being isolated to the health monitor, which needs more than the defaults.
from the application's manual:
Use scripting for load balancers. For example, you can use
http://[host]:[port]/admin?op=ping on the load balancer to check
whether the Dgraph process is running on this port. If it is not running,
the load balancer fails over to another port, and directs queries to the
MDEX Engine that is currently available.
so what we are doing is creating monitors for the individual pool members. no monitors are to be applied at the pool-level or node-level.
the application is endeca.... if anyone's ever played with that running behind an LTM, let me know.
Thanks! - marc_wermund_10quick update. the pool member monitors are half-working now, that is the requests are being received okay by the application, now i just need to figure out what to enter in for my receive string so that the pool members are not marked as down
Mark_CloutierMarc, after I stopped trembling from reading the words dgraph and Endeca, I came to the website to send you a reply..... I first setup a load balanced Endeca environment about 9 years ago I think... This was before SNAT was even an option, so one-armed load balancing wasn't even an option. I haven't worked with it since the division of our company that used Endeca was split off about 4 years ago, but I'm pretty familiar with the whole agraph, dgraph process and sepnt many hours with a sniffer to find out why there were problems using a load balancer. Ultimately it had to do with Endeca having chosen to use their own "improved" tcp routines to generate sockets. Completely violated tcp standards and caused all kinds of problems.
Anyway , feel free to send me an email at clouts85@comcast.net and we can talk further if you'd like- hoolioI thought the pool members were being marked up. If the pool was down it would make sense that the server side connections weren't being opened.
To troubleshoot monitor issues, you can check this recent post for some ideas:
http health monitor help
http://devcentral.f5.com/Default.aspx?tabid=53&forumid=32&tpage=1&view=topic&postid=85213
Ideally, you want to find a page that uses the full functionality of the application to monitor. If there is a page that requires a database call that would be a good test.
Aaron 
AK_5144Marc Wermund,
We're trying to do the same thing
/admin?op=ping HTTP/1.1\r\nHost: www.myhost.com:portnumber\r\nConnection: close\r\n
But we're getting this result
-----------------------------------------------------------
HTTP/1.1 400 Bad Request
Server: MDEX/6.1.3.393516
Expires: Wed, 03 Feb 2010 21:00:58 GMT
Accept-Ranges: none
Cache-Control: no-cache
Connection: close
Date: Wed, 03 Feb 2010 21:00:58 GMT
---
Is endeca port number needed for the health check?- marc_wermund_10HI AK,
Sorry to be the bearer of bad news.... I don't know what the working config is... we ended up selecting a load-balancing method that didn't involve using an F5 product in the end.