Forum Discussion
EastCoast_16835
Altostratus
AltostratusSSL error (ssl_error_bad_mac_read) between LTM and Firefox
We have noticed that recent versions of Firefox 36+ are frequently giving SSL errors [ssl_error_bad_mac_read] when talking to our LTM. The LTM is used as a reverse proxy for a website and does SSL bridging.
The error happens sporadically on some web pages but some other web pages are giving it pretty constantly.
The error happens with all tested flavors of SSL/TLS: SSLv3, TLS 1.0, TLS 1.2.
The error does not happen with IE, Chrome and previous versions of Firefox (before 36).
The error does not happen if we bypass LTM and connect directly to the website with any version of TLS.
Has anybody already seen this issue? What could be a problem?
Any help will be appreciated
UPDATE 1 If I disable in Firefox all ciphers except 3DES+SHA, everything works well.
UPDATE 2 I have three different VIPs on our LTM that use different SSL certificates. I tested all of them with Firefox. In all cases TLS 1.2 with the cipher suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f) was negotiated. In two cases the SSL connections fail with a "bad mac" error. In the third case, I have been unable to reproduce the issue.
UPDATE 3 According to Wireshark captures the SSL connection fails sometimes right after the handshake. But sometimes it fails later after have transferring some amount of HTTP data. Looks like a bug in crypto libraries.
UPDATE 4 Tested LTM with an OpenSSL client using TLS 1.2 and the AES128-SHA cipher. Got a similar behavior with an intermittent decryption error.
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record
Jonathan_MansfiNimbostratus
We were seeing similar behavior on one of our 11050's, though we are offloading SSL at the F5. It was only affecting traffic coming from our private IP space which was going through a u-turn nat to reach publicly addressed services in our DMZ. The interesting thing is that we were able to remediate the problem temporarily by changing the snat IP on our firewall which was being used by affected clients. Ultimately the behavior returned after about three days using the new snat IP. We also saw similar behavior from Chrome. We failed traffic to another HA member, and the problems subsided. After restarting the problematic member, we failed traffic back to it and it appears to be running clean. We haven't determined the root cause yet. I'm wondering if anyone else has run into this before. (These boxes are currently running 11.5.1hf7)- Hi EastCoast, what is the type of LTM (something like B2250 Viprion or Virtual Edition or 6900F) and TMOS version (please be on a recent hotfix)?
EastCoast_16835At the time it was 11.4.1 HF6 and I was told by support that the SSL engine in this version had a regression bug fixed in HF7+.- EastCoast_16835Update. We upgraded the LTM version to 11.4.1 HF9 and still having the same issue. There are sporadic BAD MAC errors when using AES SSL ciphers. Could it be a hardware problem? This is a B2250 blade with a hardware SSL Accelerator.
EcesureshkumarDisable and enable respective virtual servers/Pools to reset all the memory binding to the VS
- Jonathan_MansfiJust wanted to give an update on the situation we're seeing as well. We have seen our issue return while running on the other member in our HA pair. It is most prevalent on a VIP where we are decrypting and then re-encrypting the traffic. We still have not found the root cause, but the problem acts like a resource leak in that it doesn't manifest immediately but becomes increasingly worse once it does. So far, we've found three things that alleviate the problem temporarily 1) failing over to another HA group member, 2) restarting the problem member, 3) changing the SNAT IP for the clients experiencing the issue. In all three cases, our issue seems to recur within 3 - 5 days.
- EastCoast_16835Jonathan, in your case does it depend on the SSL ciphers used? We have found that 3DES does not produce errors, while AES does.
- Jonathan_MansfiEastCoast, do you have an open case for your issue? Our SR for our issue is 1-1838670408.
- Jonathan_MansfiEastCoast, I'll have to do some testing--it may.
- EastCoast_16835Yes, we have. I do not have the SR though. Thank you for the reference.