F5 CIS, TLS Extensions, and troubleshooting
Summary
Recently, I learned that the iRule that is deployed by default with HTTPS VIPs created by Container Ingress Services (CIS) can be removed natively with a Policy CRD. And I learned a little about TLS versions and TLS extensions, and testing with openssl.
Background
If you are unfamiliar with CIS, it is a controller that runs within Kubernetes (K8s) and monitors the K8s API server. CIS is able to modify the BIG-IP system configuration based on changes made to containerized applications.
By default, when you use a VirtualServer CRD to create a HTTPS VIP with CIS, an iRule is created and attached to the VIP on the BIG-IP. This iRule is about 292 lines long, and from what I can tell, is used for deciding which clientSSL profile should be used if there are mutliple profiles to choose from. (I'm not an expert with this iRule, but it looks like it would be required if you used SNI to distinguish between multiple TLS websites hosted at the same IP address).
However, the iRule isn't strictly required if you have a very basic (but common) set up: a simple HTTP/S VIP that performs TLS decryption, only has 1 possible clientSSL profile (no shared VIP with other TLS sites), and then simply proxies the traffic to pool members.
Problem Definition
A customer called me and said:
"CIS is creating an unnecessary iRule that's attached to my VIP. Half of my TLS clients cannot connect to the VIP, but when I remove this iRule manually in the GUI, they connect fine. These TLS clients are old medical devices. Can we just have CIS not create this?"
With the default set up, my TLS connections were failing from medical devices.
More details
After some packet captures, I saw that when the TLS client is a browser, the TCP handshake, TLS handshake, and conversation look successful. But when the medical devices try to connect, the BIG-IP is sending a RST packet immediately after receiving the Client Hello - so it's likely that there is something in this Client Hello message that the BIG-IP doesn't like.
Focusing in on the troublesome Client Hello, we can see a few things that jump out immediately:
- the TLS version is 1.0. That's pretty old.
- the cipher suites offered by the client are old.
- there is no SNI extension.
- In fact, there are no TLS extensions at all (spoiler alert, this fact is important)
Closing in on our problem
I used openssl to test, trying to emulate these medical devices. I ran this command to use an old TLS version, cipher suite and remove the SNI header.
openssl s_client -connect demo.my-f5.com:443 -noservername -tls1 -cipher 'DHE-RSA-AES128-SHA'
But my connections still worked! I showed the problem was not the TLS version, the cipher suite, or the lack of SNI extension. Perhaps the problem is that there are no extensions at all in the Client Hello from the medical devices?
I couldn't remove every TLS extension from my Client Hello, and I wondered why. After researching, it looks like I cannot use openssl s_client to make a connection without any TLS extensions. For that I'd need to use python or some other language, but frankly I gave up this pursuit and checked out the iRule in more detail.
Sure enough, the iRule contains this line, which was dropping any connection where the Client Hello does not contain any TLS extensions at all. After this line, the iRule goes on to identify the extensions and use the SNI extension.
if { ! [ expr { [info exists tls_extensions_len] && [string is integer -strict $tls_extensions_len] } ] } { reject ; event disable all; return; }
What does all this mean? It means that by default, really old TLS clients that don't have any TLS extensions at all will fail to connect to a VIP created by CIS. This is certainly an edge case, but let's solve it!
Solution ideas
Here are the ideas I had to solve for this.
- I could use a ConfigMap and AS3! I know from previous experience that there's multiple ways to configure CIS. CRD's are the best way, but once upon a time we had to use Ingresses or ConfigMaps. This is probably the most "cloud-native" approach because it keeps all configuration inside Kubernetes.
- I could use a TransportServer to create a Layer4 VIP and not terminate SSL on the BIG-IP. I could send traffic into K8s encrypted, and then use NGINX Ingress Controller to decrypt. This is another "cloud-native" approach but requires an additional component in NGINX.
- VIP targeting. I could manually create a VIP with a clientSSL profile. I'd have medical devices hit my manually-created VIP, perform SSL decryption, and forward traffic to a HTTP-only VIP that CIS creates. This is a little sloppy: there's manually created configuration outside of the CI/CD pipeline.
Lastly, I was embarrased when the customer found the answer by themselves. It turns out I'm not special and this was requested once before.
There is a Policy CRD that can remove the iRules that are created by default. I simply created and attached this Policy CRD.
Conclusion
Well, this was a long story for a short answer in the end! What I learned: I knew that old TLS versions, old cipher suites, and non-SNI-enabled clients often cause problems, but I didn't know much about TLS extensions generally before this experience. It's rare to see no TLS extensions from the client at all, and probably a warning sign that you should ask why and proceed carefully.
In the end, we dealt with our problem by removing an iRule that required the existence of TLS extensions and fortunately was not serving any other purpose for our application. If it was, we may have had to resort to another option, but hey, that's what we as developers/architects/devops folks love! Thanks for reading!