F5 SMTP Fast Template - SNAT Not working as expected

[Franky-frank-reg7]

Hello all,

I'm having issues with getting SMTP to work with the FAST templates. Specifically, I'm trying to configure an SMTP template with no SNAT option checked so the backend pool members received the original clients IP address, but while running debugs we're seeing the server consistently received traffic with the F5's self IP address.

I'm tried to toggle between checking and unchecking the checkbox in the SMTP FAST template for SNAT and automap, but even when completely disabled the feature and creating a new template, the servers are still seeing the F5 self IP's as the source of traffic. Can anyone suggest what might be the fix here? Below is a SNIP of my configuration:

Note: There is no VLAN filtering, TLS termination, or anyting special very plain vanilla configuration.

[Paulius]

@Franky-frank-reg7 If the virtual server configuration does not have SNAT enable then it should not be performing SNAT. Are you positive the traffic that you are receiving from the F5 isn't health monitors to the pool members? I ask this because the health monitor will come from the F5 and most likely appear as legitimate traffic in a tcpdump. I recommend running the following tcpdump to see what's happening with your traffic when you attempt to use the virtual server.

tcpdump -nni 0.0:nnp host <SMTP_VS_IP> -w /shared/tmp/smtp_tshoot.pcap

The tcpdump mentioned will save a file named smtp_tshoot.pcap to directory /shared/tmp/ which you can copy off and then open in wireshark to see what's happening with your traffic.

[Franky-frank-reg7]

The traffic is definitely making it through the VIP, we configured the template with the domain name "mycompanytest.com",  and it's exaclty what appears in the output of the SMTP logs on the server.

[Paulius]

@Franky-frank-reg7 What does the virtual server and any associated configuration look like? Do you see SNAT enabled on the virtual server?

[Franky-frank-reg7]

It's a very straight forward configuration. There is no SNAT configured, no SSL profiles (Client/Server), or VLAN filters. Please see the snip below:

We just re-tested and it appears it may have been correct, the traffic we saw on the servers were the healthy monitors. The thing I can't describe is why traffic isn't making through the VS, exchange and other applications we deployed with FAST templates are working. Just not SMTP at the moment.

[Paulius]

@Franky-frank-reg7 I recommend using tcpdump to see if you can catch anything as it comes in.

tcpdump -nni 0.0:nnp host 10.10.200.200 and port 25 -w /shared/tmp/smtp_tshoot.pcap

With this you should be able to see if traffic is making it to the F5 and if it's being handled by an incorrect virtual server it would show that information as an additional tag labeled F5 I believe.

[Franky-frank-reg7]

@Paulius Traffic is indeed making it through the VS. If you review the post below the server on IP 10.10.221.50 is a regular server sending SMTP traffic to the VS on IP: 10.20.225.228 and it utimately hits the backend pool member on IP 10.20.225.150 shown in the wireshark capture below.

What we're unable to describe is why traffic is not being processed on the exchange server (10.20.225.150) as mail traffic. It's almost as if the F5 is formatting the packet, even though there's no SNAT, TLS termination, or any other settings accept the TCP profiles configured in the template.

[Franky-frank-reg7]

When we send traffic from an app server attempting to relay SMTP traffic to the VIP. On the Exchange server running SMTP we get the output red and black output in wireshark below:

When we directly relay from the app server to the Exchange server. We're getting successful connections as displayed in the blue connections of the wireshark capture above. The only thing I can think of that could potentially be changing connection parameters is the TCP profile, but again these are all default values we're using as part of the template.  The Fast template version we're using is listed below. Some please offer your insights. The connections are currently being reset from the exchange server as we can see the output above:

[Paulius]

@Franky-frank-reg7 Would you mind sharing the entire configuration that relates to this SMTP virtual server so I can take a deeper look? The configuration should be virtual server, tcp profiles, pool, and any associated iRules.

[Franky-frank-reg7]

"class": "ADC", "schemaVersion": "3.0.0", "id": "urn:uuid:a858e55e-bbe6-42ce-a9b9-0f4ab33e3bf7", "Exchange2019": { "class": "Tenant", "Exchang2019_SMTP_Internal": { "class": "Application", "template": "generic", "Exchang2019_SMTP_Internal": { "virtualAddresses": [ "10.20.22.228" ], "virtualPort": 25, "snat": "none", "class": "Service_TCP", "pool": "Exchang2019_SMTP_Internal_pool", "profileTCP": "wan", "iRules": [] }, "Exchang2019_SMTP_Internal_pool": { "class": "Pool", "members": [ { "serverAddresses": [ "10.20.22.150" ], "servicePort": 25, "connectionLimit": 0, "priorityGroup": 0, "shareNodes": true } ], "loadBalancingMode": "least-connections-member", "slowRampTime": 300, "monitors": [ { "use": "Exchang2019_SMTP_Internal_monitor" } ] }, "Exchang2019_SMTP_Internal_monitor": { "class": "Monitor", "monitorType": "smtp", "interval": 30, "timeout": 91, "domain": "smtp.regeneron.com", "username": "", "passphrase": { "ciphertext": "", "protected": "eyJhbGciOiJkaXIiLCJlbmMiOiJub25lIn0" } } } } }

[Paulius]

@Franky-frank-reg7 It seems like this is what you push to the device with the template. Do you have the configuration directly from the F5?

[Franky-frank-reg7]

Hi @Paulius,

We made it step further to get the SNATed client IPs terminated on the Exchange server, but the issue is exchange actually needs to see the real client IP addresses for the allowed relay list function to work. So we've instead decided to go with an inline design. We removed SNAT from the virtual server, configured the exchange server to point to the F5 self ip as it's DGW, and configured an IP forwarding Virtual server to forward server connections as shown below:

However,  because the internal VLAN is a segment on the core switch, egress traffic from the server show above in blue is being routed server -> F5 -> core switch and return traffic (red) is coming in on cores and going directly to the server on VLAN225. Since VLAN 225 is a directly attached segment to the core switch, the traffic reaches the server directly instead of going back through the F5 on return. The question is should the internal VLAN instead be isolated so it’s not accessible on the core or can we instead apply a SNAT for only the server IP as it goes outbound to the external network. Please see the drawn options below:

Option1: Isolated VLAN Inline of F5, routes on Cores point to F5 Self IP of External VLAN to route to Internal Isolated VLAN.

Option2:

[Paulius]

@Franky-frank-reg7 You will want to do option 1.

[Franky-frank-reg7]

@Paulius  Can you explain why? Option2 is actually preferred because we don't want to change our topology.

If we want to got with option 2, where would the NAT be applied on the IP forwarding VS? We want the SNAT to only affect the exchange server.

[Paulius]

@Franky-frank-reg7 The reason for option 1 is because it is the easiest to troubleshoot and it would require no SNAT at all. With option 2 when you troubleshoot it could cause you headaches in the future because you typically don't see that topology with F5s, you either have them in path with no bypass or SNAT unless absolutely necessary or you have them in one arm mode where they are out of path completely and you enable SNAT on everything.

[Franky-frank-reg7]

Just so it's clear, in this case we only need to preserve client IP as it comes in to the load balancer. The outbound traffic in the scenario above, applies when the server either needs to talk to the Internet or some core infrastructure service i.e DNS. Which does not require original server IP. The key thing is the servers need to see the real client IP, this is only th ingress component, for egress (Internet bound) traffic, there is no requirement to preserve original IP's.

So the question is, can I just SNAT on the forwarding IP VS for traffic leaving the server outbound? To my understanding, this would only change the server's IP as it's headed outbound. For traffic initiated from a client  inbound, it will hit the SMTP VS and return through original connection in the connection table on the F5. Is that not the case?

[Franky-frank-reg7]

@Paulius Any ideas on the question above?

[Paulius]

@Franky-frank-reg7 That is true but to simplify the configuration so that you reduce the man hours you might/will spend in the future troubleshoot a non-standard deployment of the F5 you should deploy it as option 1

[Franky-frank-reg7]

I'm going to try both methods and reply back for closing the thread. One last question, for the option 1, where the isolated subnet is only routable through the F5. I'm assuming we need to create another IP forwarding VS for the management traffic i.e RDP, SNMP, WMI, etc to monitor and managed the server correct? So there will be one IP forwarding VS for server initiated traffic and another IP forwarding VS for inbound communications to the server, as shown in the picture below:

Also for option 2,  I tried to configured the SNAT on the internal fowarding VS, where source is restricted to the server IP: 10.50.22x.150 but it didnt work. Is it better to configure global SNAT? Can you give some guidance for the option2? I've added some questions to the picture below:

[Paulius]

@Franky-frank-reg7 You only need 1 forwarding virtual server that has a source and desination of 0.0.0.0 which will allow the F5 to pass traffic from one interface to the other without really changing anything on it. You do not need 1 forwarding VS per direction if the intent is just to route traffic from external to internal and internal to external for any IP you aren't load balancing on the F5.

[Franky-frank-reg7]

Perfect, thanks @Paulius.

Last question before we test this is -- how do we designate which interface is external?

Currently, all traffic default routes to a GW IP that is available out of one of my self IP interfaces. I would imagine this would be my external interface?

[Paulius]

@Franky-frank-reg7 External and Internal are just common names that most people use when configuring F5s in path. Your external VLAN/interface would be which ever interface is in the subnet that you point your gateway to for your default route out typically denoted by a 0.0.0.0/0 route.