Forum Discussion

Nimbostratus

Nov 06, 2015

Upgrade from v11.6.0 HF4 to v11.6.0 HF6 breaks some monitors

Hi all, Is there any problem known with v11.6.0 HF6 upgrade with http monitors?

The customer was using the following http monitors in v11.6.0 HF4:

`GET /_layouts/15/success.aspx HTTP/1.1\r\nHost: mysitestage.domain.com`

`GET / HTTP/1.1\r\nHost: mentoring.domain.com\r\nConnection: Close\r\n\r\n`

After upgrade to v11.6.0 HF6 the http monitors looks like the following:

`GET /_layouts/15/success.aspx HTTP/1.1\\r\\nHost: mysitestage.domain.com`

`GET / HTTP/1.1\\r\\nHost: mentoring.domain.com\\r\\nConnection: Close\\r\\n\\r\\n`

So the upgrade procedure changed all backslashes to double backslashes, why? Also very strange is that the new "HF6" monitor with the string:

`GET / HTTP/1.1\\r\\nHost: mentoring.domain.com\\r\\nConnection: Close\\r\\n\\r\\n`

is STILL working in HF6.

But the monitor with the string:

`GET /_layouts/15/success.aspx HTTP/1.1\\r\\nHost: mysitestage.domain.com`

is NOT working anymore, it is turning red. After we changed all the monitors back to a single backslash every monitor was working again.

It's bad that again a hotfix upgrade leads to this bug. We can not recommend to upgrade to HF6 then.

Any ideas what kind of bug this is again?

Thanks for answering!

Best regards, Peter

Chris_103447
Nov 16, 2015
I got bit by this last night, which resulted in extended downtime until we found the cause. On our LTM-VE (standalone), it only doubled the backslashes, which allowed it to work on some pools still. Our 2000s HA pair got triple backslashes, though, and marked everything down as a result. Not good.

https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-5-1.htmlrn_ki

ID 453256 seems to point to an issues with double quotes are backslashes in monitors. Seems pretty hard to avoid getting hit by it (I'd probably over/under escape).

https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-6-0.htmlrn_ki

Different ID, 291584, points to another backslash issue where the workaround is to delete the extras.

Brad_Parker
Cirrus
Nov 06, 2015
Your GET strings don't look any different?
Peter_Baumann_5
Nimbostratus
Nov 07, 2015
Hi Brad, Sorry very strange editor here, it removed all my double backshlashes. I edited the text, now you should see what I mean...
Brad_Parker
Cirrus
Nov 13, 2015
I just experienced this very same issue! Have you opened a case yet?
- Peter_Baumann
  Cirrostratus
  Nov 16, 2015
  Hi Brad, Not yet since we have a workaround in editing the wrong monitor after HF6 and remove the backslashes.
Brad_Parker_139
Nacreous
Nov 13, 2015
I just experienced this very same issue! Have you opened a case yet?
- Peter_Baumann
  Cirrostratus
  Nov 16, 2015
  Hi Brad, Not yet since we have a workaround in editing the wrong monitor after HF6 and remove the backslashes.
SynACk_128568
Cirrostratus
Nov 16, 2015
This issue is in other v11 versions , i saw this in v11.5.2 version , only workaround is to remove the back slashes
Chris_103447
Altocumulus
Nov 16, 2015
I got bit by this last night, which resulted in extended downtime until we found the cause. On our LTM-VE (standalone), it only doubled the backslashes, which allowed it to work on some pools still. Our 2000s HA pair got triple backslashes, though, and marked everything down as a result. Not good.

https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-5-1.htmlrn_ki

ID 453256 seems to point to an issues with double quotes are backslashes in monitors. Seems pretty hard to avoid getting hit by it (I'd probably over/under escape).

https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-11-6-0.htmlrn_ki

Different ID, 291584, points to another backslash issue where the workaround is to delete the extras.
Matvey_182654
Nimbostratus
Nov 23, 2015
Hello all, we are experience fully the same issue.
Brad_Parker_139
Nacreous
Nov 23, 2015
What we have done to work around this to limit downtime is the following below. This should result in no pool being marked erroneously down on the Active device. I have also notified support.

Disable auto sync(if its enabled.)
upgrade Standby and reboot.
after the standby reboots, overwrite config on Standby from Active with a manual sync
force Active to Standby once all monitors are verified to be green(if that's expected)
repeat 2-4
re-enabled autosync
- Peter_Baumann
  Cirrostratus
  Nov 23, 2015
  Very good idea for a workaround. Thanks for sharing...
Brad_Parker
Cirrus
Nov 23, 2015
What we have done to work around this to limit downtime is the following below. This should result in no pool being marked erroneously down on the Active device. I have also notified support.

Disable auto sync(if its enabled.)
upgrade Standby and reboot.
after the standby reboots, overwrite config on Standby from Active with a manual sync
force Active to Standby once all monitors are verified to be green(if that's expected)
repeat 2-4
re-enabled autosync
- Peter_Baumann
  Cirrostratus
  Nov 23, 2015
  Very good idea for a workaround. Thanks for sharing...
Matvey_182654
Nimbostratus
Nov 27, 2015
We've got an answer from F5: "It appears that you are hitting a known issue "Bug alias 558517 TMSH does not process backslashes and embedded double-quotes correctly" which manifests itself on upgrade to 11.6.0 HF6.

The workaround would be to remove the unneeded backslashes. "