LTM networking issue:

sounds like asymmetrical routing issue. Did you enable "automap" for source address translation for the VIP on the F5? If I am understanding your situation correctly, what is happening is the dst host will be the VIP on the F5, which then gets sent on to the target node on the pool. Without automap, the src IP remains the same, and when the target node replies, it will go directly using default route...causing the asymmetrical routing. When you enable "automap", the source IP gets translated to the self-IP of the F5, hence, the reply from the node will be directed back to the F5.

Thank you Vincent. you are correct, the destination is a single address within the range of the wildcard VIP. but unfortunately SNAT:automap on the wildcard VIP, did not help.

you were saying: "...the src IP remains the same, and when the target node replies, it will go directly using default route..." but, why will it go back using default route, while source address (bridged network) is within the mask of a matching self-ip? (bridged vlan-group has a matching self-ip, by instructions) my understanding is that routing table is only used when destination is not in the ranges of self IP netmasks. is that correct? Thanks...

That is correct, but it also depends on whether we have an active listener for that conversation. If you have a conversation that goes back by a different route, we may well use that instead. Bridged networks are notoriously difficult to troubleshoot, and your best bet may be to open a case with Support and have them walk through a packet capture of the problem.

If you're comfortable with packet capture analysis, that may be your best bet moving forward, as well. Remember that you will have to track MAC addresses as well as IP addresses since the IP address will not change across the network.

I would suggest a packet capture on the BigIP on 0.0 to see what the BigIP sees, and another capture on both of the endpoint servers. This will allow you to quickly identify what is happening. If you use 0.0:nnn we will embed the Virtual Server in the packet, as well.

I am not familiar with FWSM, but I do have experience with a variety of other firewalls. Most firewalls will allow you to capture traffic on each individual interface. You just have to build the proper filter to capture the traffic you are looking for. Better yet, do the capture on the source and destination host. You can use wireshark, or microolap's tcpdump. Depending on how many nodes you have in the pool, it may be best to disable all but one, so you don't have to hunt for your session. Your src ip, src port, dst ip and dst port must match for the handshake to establish the session. If the SYN/ACK response does not match, the handshake will not establish.

thank you for your useful comments after some packet capture, it looks like we have SYN-ACK in SRC and a TCP RESET, from SRC, right after that. not sure for what reason, possibly due to unexpected MAC address? any thoughts on further directions?

So I just found the cause of the TCP reset. seems that SRC is getting the response from a modified DST TCP port, and not the addressed one. I still don't know what causes this. I suppose this is the key to the solution. thoughts?

yes, that was what I was suspecting. Commonly, this happens when SNAT automap is not checked. So, I am reading back to your original question. Initially, it looks like you did not have SNAT. Then you enable SNAT automap on the wildcard VIP, and it still did not work. Question, do you have another VIP pointing to a specific service (e.g port 80 or 443)? If you do, F5 matches to the more specific service VIP. As such, you need to check on that VIP to make sure SNAT automap is enabled.

Also, if you have the capture between the src host and dst host, do post it here so I can get a better idea on what's happening.

thanks; Ill try to gather the pcaps. I can confirm that the issue does not occur when addressing a non-wildcard VIP thats a good progress with figuring this out but does this mean that the answer for me is to set up 254 VIPs of type "forwarding" for each network?

Are radical options on the table? I'd break parts of the network down and re-do them. When designing networks, take into account MTTR (Mean Time to Recovery) and ensure network can be effectively troubleshooted and managed at all times.

It's just a recommendation for a future project in your company, not a solution to your question which would give immediate relief

I'm amazed the the issue of random port from response is handled by narrowing the range of forwarding-ip VIP from wildcard to a single address. btw: automap not needed this way. but then again, I can't really set up 254 VIPs per network. it makes it hardly a workaround.