I have a web service that is being called through the Big-IP LTM on port 81. This just HTTP, no HTTPS. For some reason the web service won’t load/respond but if you go to each server it works fine so I know it’s a Big-IP issue. The web service calls another web service so that may have something to do with it but everything stays on port 81 with HTTP. Any thoughts or ideas on what the issue could be? Thank you.
Hi Robert,
Does the pool have a monitor enabled? If so, it is marked up? If not, does LTM have a self IP on the same subnet as the servers? If there isn’t one, you’ll need a route.
Do you see the pool stats incrementing?
If you run a tcpdump (tcpdump -ni 0.0 -Xs0 host 1.1.1.1) looking for traffic destined for the server(s) IP do you see a reply coming back when a client connects to the VS? If not, is the server default gateway pointing to a TMM self IP? If not, you’ll need to use source address translation. You can test with SNAT automap enabled on the virtual server.
Aaron
yes it has tcp under the health monitors for the pool. This pool is setup the same way as the pool that handles the HTTP traffic on port 80 on the same two servers with the exception of the session persistence. Port 80 has it and port 81 doesn’t!! The port 80 traffic is working fine but the web service on port 81 isn’t. The HTTP traffic and the port 81 traffic share the same virtual server 10.194.11.111
Can you try a tcpdump then:
tcpdump -ni 0.0 -Xs0 host 1.1.1.1 and port 81
Aaron
Is TCP port 81 added to your list of available virtual services?
I did the TCP dump using the “Support” option and downloaded it but how do I read it? WinDbg opens it but most of it is unreadable.
You can use wireshark:
Basically you should look for a three way TCP handshake to the pool member. If that’s happening, then you can right click on one of the packets and select follow TCP stream to see the payloads in one view.
Aaron
I have the info but can I email to you and let you take a look at it? I don’t know what a TCP handshake looks like; I’m a newbie at this!
If you need help analyzing the tcpdump you can open a support case.
Aaron
Thanks.
Robert,
The obvious things to look for in wireshark is two way traffic. Are you seeing traffic to your servers’ ip addresses and corresponding traffic from the server’s ip addresses? If you’re only seeing traffic in one way, e.g. dest is always x.x.x.x but never source x.x.x.x then that indicates an issue. Also, if you install wireshark on your web servers and run a packet capture, again, you want to see traffic from and to the client ip / or if using snat the LTM ip address. If port 80 is working hopefully you’ll see noticeable differences. Pop “port 80” in the filter bar and then “port 81” to see these differences (without the quotes).
Also, in the GUI statistics, if you select Pool Member stats (I think that’s what it’s called" do you see the connections increment when you try to attach? If not then the request isn’t even been sent to the pool member.
Hope this helps a bit.
N
i agree to open a support case if u r not familiar with packet capturing.
what u should provide them are…
1. tcpdump
tcpdump -nni 0.0:nnn -s0 -w /var/tmp/output.dmp \(host x.x.x.x and port 81\) or \(host y.y.y.y or host z.z.z.z and port abc\)
x.x.x.x is virtual ip
y.y.y.y is pool member ip
z.z.z.z is another pool member ip (if had)
abc is port number which pool member is listening on
2. client ip
3. qkview
hth
Hi Robert,
Can you provide the running configuration of your box. This will help in analyzing the problem you are facing in a much better way.
Try one thing… in the client browser type http://x.x.x.x:81 and see if everything works fine or not…
Techgeeg
nathan, only one way traffic on the port 81. 0 connections!
Techgeeg, it works if I put the server’s IP address but if I put the virtual server’s IP address it doesn’t work. Error is “unable to navigate to this URL” in Internet Explorer.
Below are 4 screenshots of my configuration if it helps…
http://dph.state.al.us/Big-IP\_Robert/virtual\_server\_1.png
http://dph.state.al.us/Big-IP\_Robert/virtual\_server\_2.png
HUGE PROGRESS!!!
I have the web service somewhat responding when it goes through Big-IP but now I may have a persistence issue. Here’s how it works…
A user calls a web service and passes certain parameters and then one of three other web services is called and a PDF file is auto generated for the user. Could this be a persistence issue at this point? It wouldn’t load any of the 3 web services but if I call the main web service and don’t pass anything it tells me a username/password is required. What are your thoughts?
Robert
I notice from the screenshots that the health monitor for the pool is “tcp”. This is only ensuring basic tcp connectivity to the pool member (the three way handshake mentioned previously). I would probably want to monitor both the port and protocol being used. So in this case I’d create a custom monitor based on the default http profile and call it http81, for example. If you then associate this with the pool it will give a better indication of the f5’s connection to the pool members using http on port 81.
As for whether persistence is an issue. It doesn’t look like you’ve got persistence set anyway, again from your screenshots. One example of persistence is that within a certain time period (configurable) a client IP will be sent to the same pool member as before.
As for the traffic being one way it does look like the f5 is attempting connection but not getting any response back. We know routing’s working cos the tcp monitor is up (and you’ve got another VS with Port 80 working). Shot in the dark but is the VLAN setting the same on this VS as the port 80 one? Could you try changing the VLAN and Tunnel Traffic to All Vlans, rather than DMZ2_VS-SNAT?
Have you tried running curl on the f5? Could help. Cmd is:
curl -v http://pool_member_ip:81
N
Nathan, same DMZ as the port 80!
I can open a test web page under the web service THROUGH the Big-IP but the actual web service itself won’t render. It renders when you call it via the server’s IP address. This web service calls another web service that returns a PDF to the client. That’s why I was thinking that I may need to add persistence. The port 80 traffic has persistence and it working great. I haven’t ran curl since the “test.htm” page works going through the Big-IP.
Hi Robert,
I was having the similar type of environment in one of the implementation where the port was not 80 or 8080 but it was different similar is the case with you … can you check the following and reply back… http or https://x.x.x.x:81 where x.x.x.x is the VS IP and see if you get the desired reply through BIG-IP. Initially keep the SNAT to auto map and allow the traffic to all VLANs and check the response.
Faisal
No but I’ve narrowed the problem to the web service. It calls another web service on the same port and that’s where the problem is. It calls this other web service using the fully qualified domain name (http://…)
RESOLVED!!!