Forum Discussion

rlvcosta's avatar
rlvcosta
Icon for Nimbostratus rankNimbostratus
Sep 08, 2022

BigIP VE KVM VM network interface mapping into tmsh Interfaces logic

I could not find any other KB, F5 document or ticket with an answer for this very basic question about how BigIP VE maps its OS level to tmsh Interface convention.

As reference, a regular Linux like RedHat 7, uses "Consistent Network Device Naming", which in general create the interface name at OS level as ens<PCI slot>p<port number>.

So, for example, in a HP server with slot1 network card and 2 ports, would have its naming mapped into ens1p0 and ens1p1. It will always map this way and never change its order, unless this card is moved to another slot!

But apparently BigIP VE uses the very old ethX network naming convention, exactly abandon to avoid during boots the interfaces to be mapped into different order.

Giving a quick look into BigIP VE OS level, I see udev scripts under /lib/udev/rules.d/ but I cannot be sure on all its logic.

My problem, and probably from many other people, is to understand how BigIP Ve maps its ethX order of the KVM setup. In my case, I have Guest configured in sequence with hostdev SRIOV devices in sequence, or PCI sot 7, then PCI slot 8 and so on, as well the MAC addresses also in sequence.

But when I see BigIP VE OS, I see it mapping my first hostdev, or PCI slot 7, as eth7, which then maps ok into interface 1.7. But then my interface PCI slot 8, as eth9, which again maps into interface 1.9, and then other 8 interfaces mapped without any reasonable order.

I see here two main problems:

1) In a disaster recover that I have to recover eventually this system, we will never know what interface order the reinstalled BigIP VE will follow, so I cannot recover any backup on this box

2) Any HOST upgrade, like KVM, could also potentially remap interfaces, again, breaking completly BigIp deployment

Besides this is a very ugly solution. I have a pair of the exact same HOST HW, with the same Guest setup, or the PCI hostdev allocations, and each one had the network mappings compltly different on BigIP VE.

I would like to understand how BigIP VE maps the Guest HW interfaces into its ethx so I can try to control and make it follow the order I want, like following PCI ID order.

See my actual configuration below:

[root@localhost:Active:Standalone] config # lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04)
00:03.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller (rev 01)
00:04.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:04.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:04.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:04.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:05.0 Communication controller: Red Hat, Inc Virtio console
00:06.0 SCSI storage controller: Red Hat, Inc Virtio block device
00:07.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 20) ->Expected to be MGMT
00:08.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function->Expected to be eth0
00:09.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function->Expected to be eth1
00:0a.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function->Expected to be eth2
00:0b.0 Ethernet controller: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function->Expected to be eth3
00:0c.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)->Expected to be eth5
00:0d.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)->Expected to be eth6
00:0e.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)->Expected to be eth7
00:0f.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)->Expected to be eth8
00:10.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)->Expected to be eth9
00:11.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)->Expected to be eth10
00:13.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon

2: eth6: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:0a brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:0c brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:0d brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:0f brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:11 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:13 brd ff:ff:ff:ff:ff:ff
8: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master mgmt state UP qlen 1000
link/ether 52:54:00:89:3c:de brd ff:ff:ff:ff:ff:ff
--
9: eth7: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 52:54:00:6d:90:02 brd ff:ff:ff:ff:ff:ff
--
10: eth8: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:04 brd ff:ff:ff:ff:ff:ff
11: eth9: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 52:54:00:6d:90:06 brd ff:ff:ff:ff:ff:ff
--
12: eth10: <BROADCAST,MULTICAST,ALLMULTI> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 52:54:00:6d:90:08 brd ff:ff:ff:ff:ff:ff

See the PCI slot sequence as the MAC 52:54:00:6d:90:02 and incrementing by 2. So for example, the eth10 is my PCI slot c.

My search here is:

1) Understand how BigIP VE maps the physical interface into ethx, or indirectly into tmsh interfaces?

2) If editing files like /etc/ethmap, or another procedure, I can force the ordering of these interface mapping?

I have a support ticket open but so far no proper updates on it.

application delivery
  • CA_Valli's avatar
    CA_Valli
    Icon for MVP rankMVP
    Sep 09, 2022

    Hi, according to K12149 , starting from BIG-IP v15.1.0 the enumeration of multiple network interfaces number the interfaces in an orderly manner. 

    /etc/ethmap should be the file that outputs interfaces mapping on the BIG-IP system.

    K17283 details a known issue with deleting/renumbering the VE virtual NICs. It also states that it's a problem seen up to v15.1.0 as well. 

    Hope this helps

    CA

    • rlvcosta's avatar
      rlvcosta
      Icon for Nimbostratus rankNimbostratus
      Sep 12, 2022

      Hi, I had seen those KB documents but first one id K12149 is related to VMWare, not KVM, and I noticed that apparently this enumerated is related with MAC address ordering, but nowhere there is an explicity BigIP documentation explaining it.

      It could be ordering in terms of PCI slot, what woiuld be the best, but apparently it is not.

      The problem arises since BigIP SW is misbehaving when creating Trunks, or Port Channel aggregates. It does not setup the hypervisor network interfaces with the same MAC, causing the Trunk not to work properly. Then some workarounds is to setup the hypervisor network interfaces with the same MAC address.

      Then problem occurs since if BigIP VE is rebooted, it changes the network ordering, creating then issues in all Trunks and all Load Blance logic. This could also be seen as a Backup restore and a reboot.

      So the problem is to know exactly the network mapping logic, so I can try to create a proper workaround to another problem related to Trunk ports not having the MAC address setup properly by BigIP VE.

      On Linux this is very  simple to see,  or "Consistent Network Device Naming". I hope BigIP VE starts to use much more modern mappings like "Consistent Network Device Naming", But actually it is not clear how it does this mapping.

      If somene can give more hints on this, I would appreciate. Eventuially I could create udev scripts, but not sure if those would be propagated, like on Backups or Upgrades. And I could also violate some logic, creating some issues on tmsh processing.

      But the real goal I have is BigIP VE properly work with Trunks, or setup the same MAC address at Hypervision "physical like" network interface at BigIP VE ethX interfaces, so I would not need to put the same MAC on different interfaces component of each Trunk I have.

      Once I put the same MAC on the interfaces to try to fix Trunks, on the boot it maps interfaces in completly different order. The only way to fix it again, after restore the intial MAC, is like in the 2nd link, execute:

      1. Log in to the command line.
      2. Remove the MCPD database by typing the following command:

        rm /var/db/mcpd*

      3. Restart the system by typing the following command:

        reboot

      Any comments would be great since a basic function like Port Channel(Trunk) is not working properly.