Http server node receives partial http request big-ip load balancer

[madhava]

I am trying to setup BIG-IP load balancer first time, here is my setup.

Client send traffic to virtual server and I have a node behind internal server. I am sending a simple get request to virtual server like

http://172.16.0.50 

But my node server not receiving comple http request seems like a partial one just "GET //". This is causing server reponding as "Bad request", what am I missing in settings? any help would be appreciated.

Attaching pcaps for reference, 172.16.0.50  is my VS , 192.168.1.50 is my internalip and 192.168.1.2 is my node ip.

Client

Server

[madhava]

It seems like worked after changing virtual server type from "Standard" to "Performance(HTTP)", Weired but works 🙂

[Paulius]

@madhava In order to figure out why it wasn't working it would be helpful if you could provide the virtual server configuration, pool configuration, and any associated profiles configured on the virtual server when it wasn't working. Switch from Standard to Performance(HTTP) should fix an issue like this and most likely another setting is no longer available that was configured in the standard virtual server and that is why it's working now.

[madhava]

Hi @Paulius Thanks.

Unfortunately my fix of changing type from "Standard" to "Performance(HTTP)" is short lived, now I am again seeing my original issue. As  suggested by @Mohamed_Ahmed_Kansoh earlier I configured SNAT setting as "Auto Map" but no luck. My VS settings are all default except I selected 80 as port and Ip address, I can provide any specific information you want, here is a snap shot of it. I have a single node server in the pool which is basic http server. Server does establish TCP connection but receives only part of HTTP requests (no http version or uri info) causing server responding with bad request error 400 .

Thanks

Madhava

[Paulius]

@madhava Based on your first capture your server is not responding to the F5 since we see the TCP request being sent to the pool member (server) but nothing ever comes back. Now in the second capture we see just the client side of this connection and nothing on the server side. Assuming you have SNAT enabled you should have seen something. Have you tried performing a capture on the server directly to see if it receives any of this communication?

[whisperer]

I would be interested in root cause here as well. You can always allocate another VIP on client side referencing the same pool. This would allow testing at leisure. Performance HTTP loses quite a bit of capabilities when it comes to traffic modification.

[Mohamed_Ahmed_Kansoh]

Hi @madhava , 
First,  I'm reviewing with you your flow and Ips , So Am I correct in the below diagram according your configuration and design : 



Please confirm : Do you use SNAT Automap feature or not in Virtual server properities , you must configure it !

2nd ) 
I need you to execute only one TCPdump command to capture the Client and server sides traffic at the same time and to make it clear to follow the http and tcp stream well. 
use the below command : 

tcpdump -nni 0.0:nnnp -s0 -c 100000 -w /var/tmp/capture.pcap host 172.16.0.2

#where 172.16.0.2 is the IP of your machine or your client ip address.

using this command will capture all needed information in both sides ( Client & server side ) 
Please attach it in your reply to have a look on. 

Looking forward to hearning from you soon. 

[madhava]

Hi @Mohamed_Ahmed_Kansoh 

Thanks! My fix not working anymore, after changing "Standard" to "Performance(HTTP)" it worked for brief period then I am seeing same issue again. This time I tried with SNAT "Auto Map", but no luck still has same issue as described originally, yes diagram you described is correct and I am not using any specific profiles in VS configuration everything I left as default. Here I am attaching TCP dump as you suggested, appreciate your help.,

Thanks

Madhava

[Mohamed_Ahmed_Kansoh · Accepted solution]

Hi @madhava , 

I see in the Pcap that no completed flow between client side and server side , Bigip Resets the connection when seeing the HTTP GET request , I see also this GET request maybe truncated , something corrupted in this Request and not follow the standard of http. 

Please test the below steps : 
1) change the standard http Virtual server to >>> Performance layer 4 , you have to remove http profile prior to changing the virtual server type. 
Performance layer 4 Virtual server  means that >>> Bigip will forward all requests ( somehow acts as a router ) Not http Checks.

2) repeat the Tcpdump command that I sent in the first reply , and upload it. 

Let us see what will be the behavior and decide what is the suitable workaround for your issue. 

Look forward to hearning from you soon 🙂 

[madhava]

Hi @Mohamed_Ahmed_Kansoh,

Thanks again for the response, out of frustration I just started everything a fresh from the scratch (creating new VS etc...). It magically started working, I don't remember doing anything diffrent but worked (though still not clear why it sends truncated http request), but your responses gave me a direction considering my limited experience with BIG-IP. In any case it is now working fine with basic setup, so not spending more time finding root cause until I get the issue again :). BTW mine is a testing environment, means my client and servers get deployed on the fly, so I added some delay to my client so servers get enough time to initialize before client start sending traffic  but not sure if that made any difference. Thanks again for the insight.

Thanks  

Madhava

[JRahm]

what are you using to draw that, @Mohamed_Ahmed_Kansoh ?

[Mohamed_Ahmed_Kansoh]

Hi @JRahm , 
Here you are the Link of Excalidraw : https://excalidraw.com/ 

Also you can download it as application for MACos or Windows. 

[whisperer]

So, you must have self IPs in VLAN used for Virtual Servers on the F5. Does the internal network have a route for this subnet pointing to the floating self IP on the F5? Sounds like the server is not routing the connection back via the F5. Could have async routing in play.

[Mohamed_Salah_]

Hello @madhava ,

Can you please provide us with virtual server configuraiton in both scenarios (standard, and performance)

just type the below command after and before changing the virtual server configuration.

#tmsh list ltm virtual "virtual server name"

So we can start checking both output.

Thanks,

[madhava]

Hi @Mohamed_Salah_ 

I just started everything a fresh from the start (creating new VS etc...). It magically started working, I don't remember doing anything diffrent but worked (still not clear why it sends truncated http request), but some of the responses here helped to give some direction. I am good for now, will use these suggestions in future if I  encounter the issue again.

Thanks

Madhava