Forum Discussion
speachey
Cirrus
CirrusVE LTM Frequently Reporting Bandwidth Exceeding 75% of Licensed 1000 Mbps
A new production VE is constantly reporting that bandwidth is exceeding 75% of licensed bandwith (1G). Looking at the TMM client-side and server-side throughput graphs, the averages are around 10M (client bits out) with some spikes under 100M. I know the graphs use sampled data to smooth things out, but the averages tell me the 1G license should be good, right? Are the frequent logs just from spikes? No complaints about services, but how can I verify my license is sufficient and what is reaching/exceeding the limit? I'm upgrading our main location to 1G VE, as well, and it will handle more requests (80M client bits out throughput average compared to 10M at this site), which is concerning. Did I chose the wrong VE license or can the alerts be ignored until problems are reported?
Any advice or recommendations are greatly appreciated!
Hi speachey ,
Have you checked if Promiscuous mode had been set to Accept on all vSwitches under ESXi. This effectively means they act as hubs and not switches. They copy any traffic they see to all members of the port group. The F5 could be receiving traffic not only for it but every single server on any VLAN's to which it was connected. Every other server will be seeing the traffic as well. It would have been placing quite a bit of network load on customer machines.
Please check if Promiscuous mode can be set to Reject if possible
If you are hosting your VM on ESXi you can refere the following link as Vmware does not encourage customers to turn on promiscuous mode as per: KB1004099
https://kb.vmware.com/s/article/1004099
The current method of calculating bandwidth is: SOL15831
How the BIG-IP VE system enforces the licensed throughput rate (f5.com)
Determine licensed throughputTo determine the maximum allowed throughput rate for a BIG-IP VE system, perform the following procedure:
Impact of procedure: Performing the following procedure should not have a negative impact on your system.
Log in to the TMOS Shell (tmsh) by entering the following command:
tmshTo display the maximum allowed throughput rate, enter the following command:
show /sys license detail | grep perf_VE_throughput_Mbps
View dropped ingress/egress packetsTo view the number of ingress or egress packets that have been dropped, perform the following procedure:
Impact of procedure: Performing the following procedure should not have a negative impact on your system.
Log in to the BIG-IP command line.
To list the number of ingress and egress packets dropped by each TMM, enter the following command:
tmctl -d blade tmm/if_shaperThe output of the command appears similar to the following example:
Note: The following output is from an idle system that has not experienced any ingress or egress packet drops.
# tmctl -i -d blade tmm/if_shaper -w 180
https://support.f5.com/csp/article/K15831
Recommended Actions
1. If your BIG-IP system continually logs messages indicating that the system is exceeding the maximum licensed throughput rate, you may want to consider increasing the licensed throughput rate to avoid traffic drop by the rate shaper.2. If you are not using MAC masquerading on your BIG-IP Virtual Edition (VE) system that is hosted on a VMWare ESX/ESXi hypervisor, you may want to consider turning off promiscuous mode on the hypervisor.
3. For instructions on turning off promiscuous mode on the hypervisor, refer to the documentation from your hypervisor vendor.
HTH
Hi speachey ,
Have you checked if Promiscuous mode had been set to Accept on all vSwitches under ESXi. This effectively means they act as hubs and not switches. They copy any traffic they see to all members of the port group. The F5 could be receiving traffic not only for it but every single server on any VLAN's to which it was connected. Every other server will be seeing the traffic as well. It would have been placing quite a bit of network load on customer machines.
Please check if Promiscuous mode can be set to Reject if possible
If you are hosting your VM on ESXi you can refere the following link as Vmware does not encourage customers to turn on promiscuous mode as per: KB1004099
https://kb.vmware.com/s/article/1004099
The current method of calculating bandwidth is: SOL15831
How the BIG-IP VE system enforces the licensed throughput rate (f5.com)
Determine licensed throughputTo determine the maximum allowed throughput rate for a BIG-IP VE system, perform the following procedure:
Impact of procedure: Performing the following procedure should not have a negative impact on your system.
Log in to the TMOS Shell (tmsh) by entering the following command:
tmshTo display the maximum allowed throughput rate, enter the following command:
show /sys license detail | grep perf_VE_throughput_Mbps
View dropped ingress/egress packetsTo view the number of ingress or egress packets that have been dropped, perform the following procedure:
Impact of procedure: Performing the following procedure should not have a negative impact on your system.
Log in to the BIG-IP command line.
To list the number of ingress and egress packets dropped by each TMM, enter the following command:
tmctl -d blade tmm/if_shaperThe output of the command appears similar to the following example:
Note: The following output is from an idle system that has not experienced any ingress or egress packet drops.
# tmctl -i -d blade tmm/if_shaper -w 180
https://support.f5.com/csp/article/K15831
Recommended Actions
1. If your BIG-IP system continually logs messages indicating that the system is exceeding the maximum licensed throughput rate, you may want to consider increasing the licensed throughput rate to avoid traffic drop by the rate shaper.2. If you are not using MAC masquerading on your BIG-IP Virtual Edition (VE) system that is hosted on a VMWare ESX/ESXi hypervisor, you may want to consider turning off promiscuous mode on the hypervisor.
3. For instructions on turning off promiscuous mode on the hypervisor, refer to the documentation from your hypervisor vendor.
HTH
speacheyThanks Cirrus!
I'll check if packets are being dropped and try to figure out what they are intended for.
We recently replaced legacy LTM platforms with the VEs reporting the bandwidth alerts. Our legacy MAC Masquerade settings were also applied to the VEs. When the VEs were activated, pools were green but no traffic in or out. I found the vmware article you referenced during the maintenance to enable promiscuous mode. I was not sure what would happen in HA failovers if MAC Masquerade was disabled and asked our vmware admin if they could try enabling it on the interface configured with portgroup/trunk VLAN 4095 (to allow all vlans). As soon as they did, traffic started flowing and we left it in place.
I do not know much about vmware; is promiscuous mode required to enable a portgroup on a VE interface?Do you recommend disabling MAC Masquerade on our VEs (ESXi) and turning off promiscuous mode in vmware? Our (platform) failovers (with MAC Masquerade) have been seamless in the past; what difference could be expected in a failover event with MAC Masquerade disabled in vmware? I’m sure that depends on the network and other factors, but I’m wondering if dropped connections should be expected on a typical network if we do disable.
Incidentally, because of the vmware warnings about promiscuous mode, I posted another question to DevCentral about this very topic titled, "VE MAC Masquerade in VMware - Good or Bad?"
Hi speachey ,
When promiscuous mode is enabled at the virtual switch level, all portgroups within the vSwitch will default to allowing promiscuous mode. However, promiscuous mode can be explicitly disabled at one or more portgroups within the vSwitch, which override the vSwitch defined default.
For MAC masquerading
To optimize traffic flow during failover events, you can configure MAC masquerade addresses for any defined traffic groups on the BIG-IP system. A MAC masquerade address is a unique, floating MAC address that you create. You can assign one MAC masquerade address to each traffic group on a BIG-IP device. By assigning a MAC masquerade address to a traffic group, you associate that address with any floating IP addresses associated with the traffic group.
The BIG-IP system uses the MAC masquerade MAC address when sending a gratuitous ARP during a failover event.
Gratuitous ARP announcements for masqueraded MAC addresses are not limited to the specific VLANs that virtual address instances reside. The virtual addresses gratuitous ARP announcements are sent out on all configured VLANs.
MAC masquerade does not affect health monitor traffic. The BIG-IP systems continue to use the original MAC address when performing health checks.
When configuring traffic-group MAC masquerading for BIG-IP Virtual Edition (VE) on VMware ESX or ESXi servers, you must set the virtual switch's Forged Transmits and Promiscuous Mode settings to Accept. (These settings are disabled by default).
For information about enabling Promiscuous Mode and Forged Transmits on the virtual switch, refer to the VMware knowledge base article listed in the Supplemental section or in the VMware documentation for your specific VMware version.
F5 Recommendations
F5 recommends that hypervisor administrators be very conservative with regard to interface usage after you enable promiscuous mode.All packets are mirrored to all interfaces in the same portgroup or vSwitch on which promiscuous mode is enabled. For each interface in the vSwitch or portgroup, an additional hypervisor CPU is required to copy these packets.
This can lead to CPU exhaustion for the hypervisor, even if an interface is uninitialized on the BIG-IP system. F5 recommends that you use only one interface in a portgroup or vSwitch on which promiscuous mode is enabled.
Additionally, you should never use the standby unit on the same hypervisor as the active unit (which is normally a best practice for BIG-IP VEs) because, in promiscuous mode, the system copies all traffic to both the active and standby devices when MAC masquerade is in use on VMware.
Starting from VMware ESXI 6.7, Promiscuous Mode can be replaced by MAC Learning in a supported environment, that is, Promiscuous Mode can be set to Reject when MAC Learning is enabled on the vSwitch on which BIGIP's VM is part of that network. The MAC Learning feature is supported only on Distributed Virtual (DV) Port groups.
To optimize the flow of traffic during failover events, you can configure MAC masquerade addresses for any defined traffic groups on the BIG-IP system. A MAC masquerade address is a unique, floating MAC address that you create. You can assign one MAC masquerade address to each traffic group on a BIG-IP device. By assigning a MAC masquerade address to a traffic group, you associate that address with any floating IP addresses associated with the traffic group. By configuring a MAC masquerade address for each traffic group, a single VLAN can potentially carry traffic and services for multiple traffic groups, with each service having its own MAC masquerade address.K3523: Choosing a unique MAC address for MAC masquerade
https://support.f5.com/csp/article/K3523Please let me know for more details and i will be glad to assist you further.
HTH
