static route versus IP forwarding VS

Hi,

When you receive Net Unreachable then of course cause is lack of routing entry matchin received packet. To use Forwarding (IP) you have to have either static or dynamic routes.

In the simplest setup at lest Default Route configured via Routes.

Now situation when you did that is some kind of error, trace is showing ICMP requests send from BIG-IP self IP to your Linux host:

10.21.1.18 > 10.21.1.67: ICMP echo request

That for sure will not work, opposite direction should work - as your first tcpdump where:

10.21.1.67 > mil04s04-in-f3.1e100.net: ICMP echo request

I would suggest using

tcpdump -nni [internal vlan name]:n -e 'icmp and host 10.21.1.67'

and

tcpdump -nni [external vlan name]:n -e 'icmp and host [self ip used as src packet - set by SNAT automap'

Just open two terminal to BIG-IP and run both at the same time, then send ping from your Linux host

Piotr

BTW: What do you mean by "disabled all my virtual servers"? Just setting Disabled in VS config? If so you are not in fact disabling precedence rules used by BIG-IP.

By default setting is that if there is VS that best matches traffic and this VS is disabled, less matching VS will not be used.

Something like that:

• VS1 (disabled) 10.10.10.1:443
• VS2 (enabled) 0.0.0.0:443

VS2 will not be used even if traffic is directed to port 443.

If you wan't to really disable VS use this method:

Choose Enabled On but leave Selected list empty

or

Choose Disabled On and in Selected place VLAN from which you are sending pings

or

Create tunnel object of type tcp-forward, let's say blackhole. Then assign it to Selected panel for Enabled On

Sorry, I was looking at first pings in the trace, latter on there are pings from Linux to google via (probably) BIG-IP self IP.

Assuming that your VS has All Protocols set and BIG-IP receiving ping then some strange situation there is.

Try to run:

tcpdump -nni 0.0:np -s0 -e host 10.21.1.67

If possible open another terminal windows to big IP and launch this commands:

watch -n 0.1 tmsh show sys connection cs-client-addr 10.21.1.67

watch -n 0.1 tmsh show sys connection ss-client-addr [ip used by automap]

Piotr

Some more ideas for test:

Can you ping google from F5? If so which VLAN is used for outgoing requests, and did those request go back via the same VLAN.

Can you check routing using ip route get [google IP - 216.58.198.35]

What about other type of connections from Linux - can you reach any site in the Internet - for example using curl -k https://www.google.fr -v

If not what is in the trace on F5, did F5 send RST packets to Linux? If so you can try:

tmsh reset-stats net rst-cause

then

watch -n 1 tmsh show net rst-cause

Repeat connection using curl and look what reasons for reset are listed by previous command.

To be honest it's very basic setup I did hundreds of times before and never had issue. Very mysterious.

If you can, as last resort reboot F5 - to be honest more than once I have situation when something did not work but should - magically after reboot it started - especially on VE.

Piotr

Hi, still here...digging (with the network team). I promise i'll try to do and answer everything. Reviewing yesterday's post from Piotr :

BTW: What do you mean by "disabled all my virtual servers"? Just setting Disabled in VS config? If so you are not in fact disabling precedence rules used by BIG-IP.

By default setting is that if there is VS that best matches traffic and this VS is disabled, less matching VS will not be used.

Something like that:

VS1 (disabled) 10.10.10.1:443
VS2 (enabled) 0.0.0.0:443
VS2 will not be used even if traffic is directed to port 443.

If you wan't to really disable VS use this method:

Choose Enabled On but leave Selected list empty

or

Choose Disabled On and in Selected place VLAN from which you are sending pings

or

Create tunnel object of type tcp-forward, let's say blackhole. Then assign it to Selected panel for Enabled On

Thanks for the information which i confirm is new to me.

Still from Piotr :

Some more ideas for test:

Can you ping google from F5? If so which VLAN is used for outgoing requests, and did those request go back via the same VLAN.

Can you check routing using ip route get [google IP - 216.58.198.35]

Yes i can ping from the F5 and it is going through eth0 directly. Example while ping 8.8.8.8 (google DNS) :

[admin@f5gm03:Active] ~  tcpdump -quiet -i eth0 host 8.8.8.8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:05:02.717615 IP f5gm03 > google-public-dns-a.google.com: ICMP echo request, id 65338, seq 13, length 64
12:05:02.722784 IP google-public-dns-a.google.com > f5gm03: ICMP echo reply, id 65338, seq 13, length 64
12:05:03.718761 IP f5gm03 > google-public-dns-a.google.com: ICMP echo request, id 65338, seq 14, length 64
12:05:03.725721 IP google-public-dns-a.google.com > f5gm03: ICMP echo reply, id 65338, seq 14, length 64

And with ip route get :

[admin@f5gm03:Active] ip route get 8.8.8.8
8.8.8.8 via 10.21.60.254 dev eth0  src 10.21.60.37
    cache  mtu 1500 advmss 1460 hoplimit 64

Now wondering if this is expected behaviour...10.21.60.254 is the F5 default route and is our core switch...Below the output of "netstat -rn" (after obfuscating few IPs)

[admin@f5gm03:Active] netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
x.x.x.x   0.0.0.0         255.255.255.252 U         0 0          0 HA-G2
x.x.x.x   0.0.0.0         255.255.255.252 U         0 0          0 SYNCHRO-G-2
x.x.x.x 0.0.0.0         255.255.255.224 U         0 0          0 DMZ-Test
10.21.1.0      0.0.0.0         255.255.255.0   U         0 0          0 DMZ-intern
x.x.x.x       0.0.0.0         255.255.255.0   U         0 0          0 tmm
x.x.x.x     0.0.0.0         255.255.255.0   U         0 0          0 eth2
10.21.60.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
x.x.x.x       0.0.0.0         255.255.255.0   U         0 0          0 GVA-netsrv2-Z1
x.x.x.x       0.0.0.0         255.255.255.0   U         0 0          0 GVA-netsrv-Z1
x.x.x.x       0.0.0.0         255.255.254.0   U         0 0          0 GVA-app-Z1
x.x.x.x       0.0.0.0         255.255.254.0   U         0 0          0 GVA-appfe-Z1
x.x.x.x       0.0.0.0         255.255.254.0   U         0 0          0 GVA-srv-Z2
x.x.x.x      0.0.0.0         255.255.248.0   U         0 0          0 GVA-User-DHCP
x.x.x.x       127.1.1.253     255.255.0.0     UG        0 0          0 tmm
x.x.x.x      0.0.0.0         255.255.0.0     U         0 0          0 tmm_bp
0.0.0.0         10.21.60.254    0.0.0.0         UG        0 0          0 eth0

Question of the day :) From my linux host i can ping 8.8.8.8 as long as i have this IP forward virtual server :

ltm virtual IP_forward {
    destination 8.8.8.8:any
    ip-forward
    ip-protocol tcp
    mask 255.255.255.255
    profiles {
        fastL4 { }
    }
    source 0.0.0.0/0
    translate-address disabled
    translate-port disabled
    vs-index 29
}

I insist : if i remove this IP forwarding VS (or disable it the proper way, thanks Piotr!) the ping does not work anymore (i am talking now of "ping 8.8.8.8" from my linux host). I would therefore be sure that this IP forwarding VS is used.... no ? "IP route get 8.8.8.8" from linux host shows it is going to the F5:

root@smt02:~ ip route get 8.8.8.8
8.8.8.8 via 10.21.1.16 dev eth0  src 10.21.1.67

This is again confirmed by tcpdump on my F5 thanks to which i see traffic is coming through the "DMZ-Intern" interface (and nothing on eth0). But looking at the statistics of all the VS (including the IP forwarding VS of course) : all stats remains 0.... Can't understand....

Hi,

If your ping to google goes via eth0 there is something wrong with routing here. eth0 is management interface and will never be used by any data traffic processed by F5.

netstat -rn is showing exactly that - only default gateway defined on your system is on eth0 (management interface)

Could you perform

tmsh show net route static

Next could you perform

tmsh list net route

tmsh list sys management-route

When you set Destination as 8.8.8.8 then F5 is replying to ping not google server. If you will try to establish TCP connection it will fail.

I if it will work the same on every TMOS version as I can recall there were some changes in behavior, but you can try such mod and then test ping again

• Go to Local Traffic ›› Virtual Servers : Virtual Address List
• Click on 8.8.8.8
• Uncheck ARP

You should no longer receive reply.

So my guess is that for some reason your default gateway entry set via Network ›› Routes:

Name    Internet
Partition / Path    Common
Description 
Destination 0.0.0.0
Netmask 0.0.0.0
Resource    Gateway Address   144.144.144.129

either was removed (suggested by output of netstat -rn) or IP is pointing to device that can't route packets to Internet.

Piotr

Thanks again Piotr for your help.

[sp_admin@chgva-f5g-m03:Active:Changes Pending] ~  tmsh show net route static
[sp_admin@chgva-f5g-m03:Active:Changes Pending] ~  tmsh list net route
[sp_admin@chgva-f5g-m03:Active:Changes Pending] ~  tmsh list sys management-route
sys management-route default {
    gateway 10.21.60.254
    network default
}

About the ping you wrote :

When you set Destination as 8.8.8.8 then F5 is replying to ping not google server. If you will try to establish TCP connection it will fail.

Which in some way is good to know because definitively i was not understanding how the ping could be so fast and why doing a traceroute from my linux server to 8.8.8.8 was reaching in one hop! Now having said this, wy would f5 answer to some ping which is not targetting the f5 ? I guess the answer is related to this arp setting you wanted me to uncheck but i didn't find it on the Virtual server interface.

About you writing the following :
So my guess is that for some reason your default gateway entry set via Network  ››  Routes:

Name    Internet
Partition / Path    Common
Description 
Destination 0.0.0.0
Netmask 0.0.0.0
Resource    Gateway Address   144.144.144.129
either was removed

i take this opportunity to ask my 2 "basic questions of the day" : -yes i confirm i removed it because so far is still did not understand whether this is mandatory or not ? Could an "IP forwarding" Virtual server could be enough to answer my need or would i anyway a static route in addition to this "Ip forwarding" virtual server

-without this static route : does the fact that "ping 8.8.8.8" works from the F5 (therefore using eth0) is expected or definitively not ? (in which case i would ask the people who implemented the F5 here to come back). If i understand well your previous input : no it should not work! means this ping traffic should definitively not go through eth0 (which should be dedicated to management). Am i right ?

Many many thanks

Back to something i did not answer yet Piotr :

Assuming that your VS has All Protocols set and BIG-IP receiving ping then some strange situation there is.

Try to run:

tcpdump -nni 0.0:np -s0 -e host 10.21.1.67

If possible open another terminal windows to big IP and launch this commands:

watch -n 0.1 tmsh show sys connection cs-client-addr 10.21.1.67

watch -n 0.1 tmsh show sys connection ss-client-addr [ip used by automap]
`

Here you are : 

Runing simultaneously :
1) ping from my linux box to 8.8.8.8

`root@smt02:~ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=255 time=0.135 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=255 time=0.202 ms
`

2) tcpdump on the F5 :

`[admin@f5gm03:Active] tcpdump -nni 0.0:np -s0 -e host 10.21.1.67
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on 0.0:np, link-type EN10MB (Ethernet), capture size 65535 bytes
15:50:18.275150 00:50:56:9e:d9:b3 > 00:23:e9:ec:8f:84, ethertype 802.1Q (0x8100), length 109: vlan 1000, p 0, ethertype IPv4, 10.21.1.67 > 8.8.8.8: ICMP echo request, id 46957, seq 306, length 64 in slot1/tmm0 lis=
15:50:18.275169 00:23:e9:ec:8f:84 > 00:50:56:9e:d9:b3, ethertype 802.1Q (0x8100), length 109: vlan 1000, p 0, ethertype IPv4, 8.8.8.8 > 10.21.1.67: ICMP echo reply, id 46957, seq 306, length 64 out slot1/tmm0 lis=
15:50:19.275325 00:50:56:9e:d9:b3 > 00:23:e9:ec:8f:84, ethertype 802.1Q (0x8100), length 109: vlan 1000, p 0, ethertype IPv4, 10.21.1.67 > 8.8.8.8: ICMP echo request, id 46957, seq 307, length 64 in slot1/tmm0 lis=
15:50:19.275348 00:23:e9:ec:8f:84 > 00:50:56:9e:d9:b3, ethertype 802.1Q (0x8100), length 109: vlan 1000, p 0, ethertype IPv4, 8.8.8.8 > 10.21.1.67: ICMP echo reply, id 46957, seq 307, length 64 out slot1/tmm0 lis=
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
`

3) watch on the F5 :

[admin@f5gm03:Active] watch -n 0.1 tmsh show sys connection cs-client-addr 10.21.1.67

`Every 0.1s: tmsh show sys connection cs-client-addr 10.21.1.67                                    Fri Jun  2 15:58:31 2017

Sys::Connections
Total records returned: 0


4) second watch on the F5 :
[admin@f5gm03:Active] watch -n 0.1 tmsh show sys connection ss-client-addr 10.21.1.18

    Every 0.1s: tmsh show sys connection ss-client-addr 10.21.56.18                                    Fri Jun  2 16:05:04 2017

Sys::Connections
10.21.1.18:42138  10.21.1.88:8  10.21.1.18:42138  10.21.1.88:8  icmp  10  (tmm: 0)  none
10.21.1.18:42143  10.21.1.84:8  10.21.1.18:42143  10.21.1.84:8  icmp  8   (tmm: 0)  none
10.21.1.18:42131  10.21.1.67:8  10.21.1.18:42131  10.21.1.67:8  icmp  14  (tmm: 0)  none
10.21.1.18:42160  10.21.1.88:8  10.21.1.18:42160  10.21.1.88:8  icmp  0   (tmm: 0)  none
10.21.1.18:42137  10.21.1.22:8  10.21.1.18:42137  10.21.1.22:8  icmp  11  (tmm: 0)  none
10.21.1.18:42136  10.21.1.86:8  10.21.1.18:42136  10.21.1.86:8  icmp  11  (tmm: 0)  none
10.21.1.18:42153  10.21.1.67:8  10.21.1.18:42153  10.21.1.67:8  icmp  4   (tmm: 0)  none
10.21.1.18:42159  10.21.1.22:8  10.21.1.18:42159  10.21.1.22:8  icmp  1   (tmm: 0)  none
10.21.1.18:42158  10.21.1.86:8  10.21.1.18:42158  10.21.1.86:8  icmp  1   (tmm: 0)  none
10.21.1.18:42145  10.21.1.87:8  10.21.1.18:42145  10.21.1.87:8  icmp  7   (tmm: 0)  none
10.21.1.18:42151  10.21.1.85:8  10.21.1.18:42151  10.21.1.85:8  icmp  5   (tmm: 0)  none
10.21.1.18:42149  10.21.1.88:8  10.21.1.18:42149  10.21.1.88:8  icmp  5   (tmm: 0)  none
10.21.1.18:42154  10.21.1.84:8  10.21.1.18:42154  10.21.1.84:8  icmp  3   (tmm: 0)  none
10.21.1.18:42142  10.21.1.67:8  10.21.1.18:42142  10.21.1.67:8  icmp  9   (tmm: 0)  none
10.21.1.18:42148  10.21.1.22:8  10.21.1.18:42148  10.21.1.22:8  icmp  6   (tmm: 0)  none
10.21.1.18:42147  10.21.1.86:8  10.21.1.18:42147  10.21.1.86:8  icmp  6   (tmm: 0)  none
10.21.1.18:42134  10.21.1.87:8  10.21.1.18:42134  10.21.1.87:8  icmp  12  (tmm: 0)  none
10.21.1.18:42140  10.21.1.85:8  10.21.1.18:42140  10.21.1.85:8  icmp  10  (tmm: 0)  none
10.21.1.18:42156  10.21.1.87:8  10.21.1.18:42156  10.21.1.87:8  icmp  2   (tmm: 0)  none
Total records returned: 19

Hi,

Why F5 is answering ping when you have host VS (mean single IP as destination no wildcard) - answer is because that is what F5 decide it to be :-) - so VS with single IP configured will answer pings even if those pings are not delivered to real server behind F5.

And pig is targeting F5 a bit as your VS is set with single IP. ARP setting I mentioned are not in Virtual Server config but in Virtual Address config. Go there, find IP set as destination for VS and enter settings. If you have quite new version of TMOS you will see ICMP and ARP options (in older versions - like 11.2.0 there will be only ARP), if ICMP option is there just disable it and you will not receive ICMP reply at all, if only ARP disable it and outcome will be the same.

If you removed default gateway entry from F5 then Forwarding (IP) server will not work for Internet hosts - this type of VS is based on routing entries - actually it more or less behaves as router. So you have to have entries for all subnets not directly connected to F5 (not ones reachable based on self IP config) for Forwarding (IP) VS to be able to deliver packets to target server.

Now there is a part I can't help you - I don't know what is your network setup between F5 and Internet when using DMZ VLAN on F5 for sending traffic to Internet - maybe you have some Firewall or other devices here that are preventing ICMP request sourced for F5 self IP (SNAT automap set in VS) from reaching Internet.

If only default gateway is set for MGMT port TMM traffic (data traffic) will never use tis route - it's limited to MGMT traffic and traffic sourced from F5 (like DNS queries etc.)

You need routes configured via Network >> Routes tab defined for Forwarding (IP) VS to work.

So yes, ping is not routed anyway, it's VS that is replying, you need to have static routes defined for data traffic (processed by TMM not host subsystem used for MGMT).

From show sys connection it's obvious that VS by itself is handling ping.

So fix routes, check if traffic from DMZ VLAN can reach Internet and everything will be fine.

Piotr