For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

perryzou_317374's avatar
perryzou_317374
Icon for Altocumulus rankAltocumulus
May 25, 2017

VIP is not working

Great ones, Would you help take a look at my problem. Thanks. I have an Openstack instance with F5 lbaasv2. It uses under cloud F5 VE instance with VXLAN overlay. But I won't be able reach vip of created loadbalancer. F5 VE received ARP request about vip, but it won't reply. In addition, there is no any packets captured on tunnel-vxlan-1.

F5 11.6.1 OpenStack: mitaka root@controller1: pip list | grep f5 f5-icontrol-rest (1.3.0) f5-openstack-agent (9.3.0b2) f5-openstack-lbaasv2-driver (9.3.0b2) f5-sdk (2.3.2)

 active loadbalancer

neutron lbaas-loadbalancer-list
+--------------------------------------+---------+--------------+---------------------+------------+
| id                                   | name    | vip_address  | provisioning_status | provider   |
+--------------------------------------+---------+--------------+---------------------+------------+
| 0ad55dfc-6f79-4619-9ed7-78fa566add4b | test-lb | 192.168.0.14 | ACTIVE              | f5networks |
+--------------------------------------+---------+--------------+---------------------+------------+

virtual address in F5 VE
root@(host-192)(cfg-sync Standalone)(Active)(/Project_3b04b644e8a642f4acbc4275f2488d22)(tmos) list ltm virtual-address 
ltm virtual-address Project_0ad55dfc-6f79-4619-9ed7-78fa566add4b {
    address 192.168.0.14
    auto-delete false
    description test-lb:
    mask 255.255.255.255
    partition Project_3b04b644e8a642f4acbc4275f2488d22
    traffic-group /Common/traffic-group-1
}

active members
root@(host-192)(cfg-sync Standalone)(Active)(/Project_3b04b644e8a642f4acbc4275f2488d22)(tmos) list ltm pool  
ltm pool Project_7c5c7b3b-02c7-4fdb-add2-497dc7e4cc05 {
    description Project_7c5c7b3b-02c7-4fdb-add2-497dc7e4cc05:
    members {
        192.168.0.8%0:http {
            address 192.168.0.8
        }
        192.168.0.9%0:http {
            address 192.168.0.9
        }
    }
    partition Project_3b04b644e8a642f4acbc4275f2488d22
}

member works
root@controller1:~ ip netns exec qdhcp-aa9bfd8f-721b-4a42-8bc7-8e6497e861af curl 192.168.0.8
Welcome to 192.168.0.8

won't reach vip
root@controller1:~ ip netns exec qdhcp-aa9bfd8f-721b-4a42-8bc7-8e6497e861af curl 192.168.0.14
curl: (7) Failed to connect to 192.168.0.14 port 80: No route to host
root@controller1:~ 

From management console, no traffic is observed on statistics of virtual server.

there is no fdb entry for vip port on controllers.
root@controller1:/var/log/neutron bridge fdb | grep fa:16:3e:2c:61:8a
root@controller1:/var/log/neutron  neutron port-list |  grep 192.168.0.14
| 85eefe74-1c87-46a9-bb5a-350955bf3d3c | loadbalancer-0ad55dfc-6f79-4619-9ed7-78fa566add4b                            | fa:16:3e:2c:61:8a | {"subnet_id": "511da169-7aa9-45ae-bcd7-fb9044613320", "ip_address": "192.168.0.14"}   |

arp broadcast could be observed on interface of data (used for Overlay). There is no arp reply observed.
[root@host-192:Active:Standalone] Project_3b04b644e8a642f4acbc4275f2488d22  tcpdump -ni lb-data -vvv
tcpdump: listening on lb-data, link-type EN10MB (Ethernet), capture size 96 bytes
08:06:55.838413 IP (tos 0x0, ttl  62, id 17884, offset 0, flags [none], proto: UDP (17), length: 78) 10.153.36.74.27778 > 192.168.250.4.4789: [no cksum] UDP, length 50
08:06:56.838079 IP (tos 0x0, ttl  62, id 17975, offset 0, flags [none], proto: UDP (17), length: 78) 10.153.36.74.27778 > 192.168.250.4.4789: [no cksum] UDP, length 50

open with wireshark
6   7.461238    fa:16:3e:30:c1:3d   Broadcast   ARP 92  Who has 192.168.0.14? Tell 192.168.0.2

no any traffic is observed on tunnel-vxlan-1 interface in F5 VE instance, even when curl vip.
12: tunnel-vxlan-1:  mtu 1500 qdisc noqueue 
    link/ether fa:16:3e:5b:2c:53 peer 00:00:00:00:00:00
    inet 192.168.0.5/22 brd 192.168.3.255 scope global tunnel-vxlan-1
    inet6 fe80::f816:3eff:fe5b:2c53/64 scope link 
       valid_lft forever preferred_lft forever
BIG-IP
cloud
LTM

3 Replies

  • perryzou_317374's avatar
    perryzou_317374
    Icon for Altocumulus rankAltocumulus
    May 25, 2017

    I tried to ping vip on F5 VE instance, and looks it works. But it failed when curl vip. Also listed route fyi.

         I recreated another loadbalancer with vip of 192.168.0.16.
    [root@host-192:Active:Standalone] Project_3b04b644e8a642f4acbc4275f2488d22  ping 192.168.0.16
    PING 192.168.0.16 (192.168.0.16) 56(84) bytes of data.
    64 bytes from 192.168.0.16: icmp_seq=1 ttl=255 time=0.418 ms
    64 bytes from 192.168.0.16: icmp_seq=2 ttl=255 time=0.283 ms

    [root@host-192:Active:Standalone] Project_3b04b644e8a642f4acbc4275f2488d22  curl 192.168.0.16 -v
    shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
    * About to connect() to 192.168.0.16 port 80 (0)
    *   Trying 192.168.0.16... connected
    * Connected to 192.168.0.16 (192.168.0.16) port 80 (0)
    > GET / HTTP/1.1
    > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 OpenSSL/1.0.1l zlib/1.2.3 libidn/0.6.5
    > Host: 192.168.0.16
    > Accept: */*
    > 
    * Closing connection 0
    * Failure when receiving data from the peer
    curl: (56) Failure when receiving data from the peer

[root@host-192:Active:Standalone] tmp  tmsh show /net route

------------------------------------------------------------------------------------
Net::Routes
Name                Destination         Type       NextHop                 Origin
------------------------------------------------------------------------------------
fe80::/64           fe80::/64           interface  tmm0                    connected
ff02::/64           ff02::/64           interface  tmm0                    connected
ff02:ffd::/64       ff02:ffd::/64       interface  /Common/lb-data         connected
ff02:ffe::/64       ff02:ffe::/64       interface  /Common/lb-ha           connected
fe80::%vlan4093/64  fe80::%vlan4093/64  interface  /Common/lb-data         connected
fe80::%vlan4094/64  fe80::%vlan4094/64  interface  /Common/lb-ha           connected
fe80::/64           fe80::/64           interface  /Common/socks-tunnel    connected
ff02::/64           ff02::/64           interface  /Common/tunnel-vxlan-1  connected
fe80::/64           fe80::/64           interface  /Common/tunnel-vxlan-1  connected
fe80::/64           fe80::/64           interface  /Common/http-tunnel     connected
fe80::%vlan4095/64  fe80::%vlan4095/64  interface  tmm_bp                  connected
ff02:fff::/64       ff02:fff::/64       interface  tmm_bp                  connected
fe80::%1/64         fe80::%1/64         interface  tmm0%1                  connected
ff02::%1/64         ff02::%1/64         interface  tmm0%1                  connected
127.1.1.0/24        127.1.1.0/24        interface  tmm0                    connected
192.168.250.0/24    192.168.250.0/24    interface  /Common/lb-data         connected
192.168.253.0/24    192.168.253.0/24    interface  /Common/lb-ha           connected
127.20.0.0/16       127.20.0.0/16       interface  tmm_bp                  connected
192.168.0.0/22      192.168.0.0/22      interface  /Common/tunnel-vxlan-1  connected
127.1.1.0%1/24      127.1.1.0%1/24      interface  tmm0%1                  connected
internal-bm         10.153.36.64/27     gw         192.168.250.1           static


[root@host-192:Active:Standalone] tmp  route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.153.36.64    192.168.250.1   255.255.255.224 UG    0      0        0 lb-data
127.1.1.0       0.0.0.0         255.255.255.0   U     0      0        0 tmm0
127.3.0.0       0.0.0.0         255.255.255.0   U     0      0        0 mgmt_bp
192.168.250.0   0.0.0.0         255.255.255.0   U     0      0        0 lb-data
192.168.253.0   0.0.0.0         255.255.255.0   U     0      0        0 lb-ha
192.168.254.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.0.0     0.0.0.0         255.255.252.0   U     0      0        0 tunnel-vxlan-1
127.7.0.0       127.1.1.254     255.255.0.0     UG    0      0        0 tmm0
0.0.0.0         192.168.254.1   0.0.0.0         UG    9      0        0 eth0
  • perryzou_317374's avatar
    perryzou_317374
    Icon for Altocumulus rankAltocumulus
    May 26, 2017

    I read F5 documentation and looks Overlay needs to enable SND service. I doubt that licence key is the cause.

    root@(host-192)(cfg-sync Standalone)(Active)(/Common)(tmos) show sys license 

Sys::License
Licensed Version    11.6.1
...
Active Modules
  LTM, Lab, VE (FPYNJUP-NVRYDZA)
    IPV6 Gateway
    Rate Shaping
    Ram Cache
    Client Authentication
    Application Acceleration Manager, Core
    SSL, VE
    Max Compression, VE
    Anti-Virus Checks
    Base Endpoint Security Checks
    Firewall Checks
    Network Access
    Secure Virtual Keyboard
    APM, Web Application
    Machine Certificate Checks
    Protected Workspace
    Remote Desktop
    App Tunnel
    
root@(host-192)(cfg-sync Standalone)(Active)(/Common)(tmos)
  • perryzou_317374's avatar
    perryzou_317374
    Icon for Altocumulus rankAltocumulus
    Jun 02, 2017

    The root cause is that licence didn't include SDN service.

     

    In addition, I observed that port 4789 won't be listed with netstat even VIP is back to work.