For more information regarding the security incident at F5, the actions we are taking to address it, and our ongoing efforts to protect our customers, click here.

Forum Discussion

TMcGov_92811's avatar
TMcGov_92811
Icon for Nimbostratus rankNimbostratus
May 15, 2014

IP Forwarding VS 0/0 causing LTM to respond for disabled Standard VS

Problem Description

 

I have two LTMs. One is a production device (3600) that is currently passing traffic with many virtual servers. I have a staged pre-production device (4200, v11.4 ) that is also connected to the same VLANs. The Self-IP addresses for the staged 4200 device are unique so as not to conflict with the in-service 3600. The plan is to migrate Virtual Servers one by one according to a scheduled deployment schedule.

 

Initially I had many problems whereby the staged LTM was improperly responding to traffic for the production Virtual Servers despite the fact that they were disabled. I quickly learned that you must also disable the Virtual Address and uncheck the ARP and ICMP Echo settings in order to 100% ensure that the staged LTM would not respond to traffic for those Virtual Servers.

 

On my staged 4200, I was playing around with a 0/0 IP Forwarding Virtual Server to do some testing and had it enabled. Immediately I started receiving reports of connectivity issues to several Virtual Servers. I did a tcpdump and was shocked to find the staged 4200 LTM was once again responding to requests, even though the Virtual Servers and Virtual Addresses remained disabled. For example, the VS 10.88.225.109:9851 was responding and the dump clearly references my 0/0 VS '/IP-Forwarding-VS_web'

 

[admin@ma-np-ltm-4200a:Active:Changes Pending] ~ tcpdump -s0 -ni any host 10.88.225.109 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type EN10MB (Ethernet), capture size 65535 bytes 13:50:29.247717 IP 10.88.225.109.9851 > 172.20.110.177.51132: S 1382498159:1382498159(0) ack 3068292258 win 4380 in slot1/tmm0 lis=/Common/IP-Forwarding-VS_web 13:50:29.247727 IP 10.88.225.109.9851 > 172.20.110.177.51132: S 1382498159:1382498159(0) ack 3068292258 win 4380 out slot1/tmm0 lis=/Common/IP-Forwarding-VS_web 13:50:29.254044 IP 172.20.110.177.51132 > 10.88.225.109.9851: S 3068292257:3068292257(0) win 8192 in slot1/tmm0 lis=/Common/IP-Forwarding-VS_web 13:50:29.254068 IP 172.20.110.177.51132 > 10.88.225.109.9851: S 3068292257:3068292257(0) win 8192 out slot1/tmm0 lis=/Common/IP-Forwarding-VS_web 13:50:41.247943 IP 10.88.225.109.9851 > 172.20.110.177.51132: S 1382498159:1382498159(0) ack 3068292258 win 4380 in slot1/tmm0 lis= 13:50:41.247987 IP 172.20.110.177.51132 > 10.88.225.109.9851: R 1:1(0) ack 1 win 0 out slot1/tmm0 lis=

 

  1. Can someone explain to me why enabling the 0/0 IP Forwarding Virtual Server would cause the system to start responding for a VS that is completely 'shutdown' ? If this is standard 'feature' and cannot be overcome, my entire migration/cutover strategy with having both devices on-line is flawed and I will have to go back to the drawing board.
  2. I have a strange issue when enabling/disabling Virtual Servers and Virtual Addresses on the 4200 v11.4. Sometimes the Virtual Address List will completely disappear from the GUI on the Active node - nothing will be listed. Sometimes the same will happen on the Standby. After a few refreshes or logging out, the list will come back. Perhaps this is just a cosmetic bug.

Regards, Tom

 

availability
BIG-IP Access Policy Manager (APM)
design
management
security
TMOS

4 Replies

  • Patrik_Jonsson's avatar
    Patrik_Jonsson
    Icon for MVP rankMVP
    May 16, 2014

    Hi!

     

    Not sure, but I believe it's because the 0/0 is supposed to listen for everything and if the VS is disabled it just moves down the list and finds that one matching. Have you tried disabled arp for it? We recently had some production disturbances because a VS forwarding server "stole" traffic not destined for it and that solved the issue.

     

    Can't you just add them in a v11 active-active cluster and use traffic groups for the migration? Ie. having one traffic group for prod and one for the staging?

     

    /Patrik

     

  • Domai's avatar
    Domai
    Icon for Altostratus rankAltostratus
    May 16, 2014

    We are currently doing the exact same VIP migrations from old to new. We created nodes and pools first and made sure all the fw rules were in order and then during the actual cut over deleted the old vips - sync and created the vips on the new boxes. I know this is not what you are doing or asking but thought I would get it out there.

     

    Regarding why the virtuals would respond when you enable ip_fwd even when your apr is disabled ...do you think during your initial setup (before you figured out to disable arp and icmp) the arp requests are cached?

     

  • Mohamed_Lrhazi's avatar
    Mohamed_Lrhazi
    Icon for Altocumulus rankAltocumulus
    May 16, 2014

    My 2 cents.... I never trust the "disable" for virtual servers... I put unrelated IPs on them, if I have to... like 1.2.3.4 (I know, that's valid subnet).... but you could use some private subnet you are not using else where if you prefer....

     

    BIGIP also, when you make some config change, tends to remember the previous one for a while... and apply it to traffic (so as not to break existing services)....

     

  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    May 17, 2014

    I did a tcpdump and was shocked to find the staged 4200 LTM was once again responding to requests, even though the Virtual Servers and Virtual Addresses remained disabled.

     

    is virtual address any (0.0.0.0/0) and its arp disabled?

     

    if not, i think it is expected behavior because it is wildcard virtual server which will accept all traffic.