Heartbleed: Network Scanning, iRule Countermeasures

Get the latest updates on how F5 mitigates HeartbleedGet the latest updates on how F5 mitigates Heartbleed

I just spent the last two days writing “business-friendly” copy about Heartbleed. I think the result was pretty good and hey, it even got on the front page of f5.com so I guess I got that going for me. Then I wrote up some media stuff and sales stuff and blah, blah blah.

But what I really wanted was to do talk tech. So let’s go!

What’s Wrong with OpenSSL?

The Heartbleed vulnerability only affects systems derived from, or using, the OpenSSL library. I have experience with the OpenSSL library. As one of F5’s SSL developers I spent a lot of time with OpenSSL before we basically abandoned it in our data plane path. Here are some examples of what’s wrong with OpenSSL from someone who spent hundreds of hours mired in its gore.

• OpenSSL does weird things for weird reasons. As Theo De Raadt points out in his small rant, the team’s decision to use their own malloc and free instead of trusting the OS by default was a tactical fuckup with major future impacts (Heartbleed).
• OpenSSL has dozens of different APIs with a lack of consistency between them. For some APIs, the caller frees the returned memory. In others, the API frees the memory itself. It’s not documented which is which and this results in lots of memory leaks or worse, double-frees.
• There are pointers opaque everywhere. A higher-level interface or language (even C++ instead of C) could provide lots of protection, but instead we’re stuck with pointers and a lack of good memory protection.

Those are just three off the top of my head. The real meta-issue for OpenSSL is this:

There is no real, commercial organization has any stake in maintaining the OpenSSL library.

Other open source code bases typically have some very close commercial firm that has a significant interest in maintaining and improving the library.

• BSD has Apple.
• Linux has Ubuntu.
• Gecko has Mozilla.

OpenSSL has nobody. The premier security firm, RSA, long ago forked off their own copy of the library (even hiring away its original development team) and thus they don’t put changes back upstream for OpenSSL. Why would they? Maybe you could say that OpenSSL has Adam Langley (from Google) on its side, so at least it’s got that going for it.

In Dan Kaminsky’s analysis about the aftermath of Heartbleed, he says that Professor Matthew Green of Johns Hopkins University makes a great point: OpenSSL should be considered Critical Infrastructure. Therefore, it should be receiving critical infrastructure funding, which could enable code audits, static code analysis, fuzz-testing, and all the other processes and improvements it deserves. I agree: in spite of all its flaws, OpenSSL has become the most significant cryptographic library in the history of computing.

Speaking of Kaminsky, if you are looking for the last time there was a vulnerability of this severity, you’d probably have to go back to CVE-2008-1447 - the DNS Cache Poisoning Issue (aka the "Kaminsky bug"). I wonder if he wishes he’d made a logo for it.

 

Detecting Heartbleed

Is it worth trying to detect Heartbleed moving forward? Heartbleed is so severe that there’s a possibility that nearly all vulnerable systems will get patched quickly and we’ll stop talking about it except in a historical sense. But until that happens, it is worthwhile to detect both vulnerable servers and Heartbleed probes.

There’s been some debate about whether probing for Heartbleed is legal, since a successful probe will return up to 64K of server memory (and therefore qualifies as a hack). Regardless of the legality, there’s a Metasploit module for Heartbleed. For your browsers there’s also a Google Chrome extension and a Mozilla Firefox extension. In fact, here’s a whole list of Heartbleed testing tools.

Scanning a network for Heartbleed using nmap

I used the nmap engine script to scan my home network from an Ubuntu 13.04 client desktop. The script requires a relatively recent version (6.25) of nmap so I had to upgrade. Is it me or is it always more of a pain to upgrade on Ubuntu than it should be?

Install Heartbleed nmap script

1. Upgrade to nmap 6.45
2. Install Subversion (if libsnv_client load error)
3. Install tls.lua NSE script and ssl-heartbleed.nse script.

Run against your network like this:

nmap -oN ./sslhb.out --script ssl-heartbleed --script-args vulns.showall -sV 192.168.5.0/24

On my home network I found one vulnerable server which I knew about and one which I didn’t!

Nmap scan report for 192.168.5.23
Host is up, received syn-ack (0.00078s latency).
PORT     STATE SERVICE  REASON  VERSION
22/tcp   open  ssh      syn-ack OpenSSH 6.1p1 Debian 4 (protocol 2.0)
80/tcp   open  http     syn-ack Apache httpd 2.2.22 ((Ubuntu))
443/tcp  open  ssl/http syn-ack Apache httpd 2.2.22 ((Ubuntu))
| ssl-heartbleed:
|   VULNERABLE:
|   The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. It allows for stealing information intended to be protected by SSL/TLS encryption.
|     State: VULNERABLE
|     Risk factor: High

Scanning your network (home or otherwise) should hopefully be just as easy.

I love nmap, so cool.

Heartbleed iRule Countermeasures

You probably don’t need an iRule. If you are using BIG-IP in front of HTTP servers, your applications are already protected because the native SSL code path for all versions isn’t vulnerable. Over 90% of the HTTPS applications fronted by BIG-IP are protected by its SSL code.

However, there may be reasons that you might have BIG-IP doing simply layer 4 load balancing to a pool of HTTPS servers.

• A pool of servers with a security policy restricting private keys to origin server hosts.
• An ingress point for multi-tenant SSL sites (using SNI and different keys).
• A virtual ADC fronting a pool of virtualized SSL sites.

If any of these cases apply to your organization, we have an iRule to help.

While I was busy writing all the copy for Heartbleed, my long-time colleague and friend Jeff Costlow was on a team of heroes developing a se tof iRules to counter Heartbleed.

Here’s the iRule to reject client connections that attempt to send a TLS heartbeat. No browsers or tools (such as curl) that we know of actually send TLS heartbeats at this time so this should be relatively safe.

Client-side Heartbleed iRule: https://devcentral.f5.com/s/articles/ssl-heartbleed-irule-update

And here’s a server-side iRule. It watches for HTTPS heartbeat responses from your server and discards any that look suspicious (over 128 bytes). Use this one if you have a client app that sends heartbeats.

Server-side Heartbleed iRule: https://devcentral.f5.com/s/articles/ssl-heartbleed-server-side-irule

Like I said, you probably don’t need to deploy them. But if you do I suggest making the slight tweaks you need to make them compatible with High-Speed logging.

Notes on Patching Systems

See F5 Solution 15159

All F5 users on 11.5.0 or 11.5.1 should patch their systems with F5 Heartbleed hotfix. Users on 11.4 or earlier are safe with the version they have. This includes the virtual edition, which is based on 11.3.0 HF1.

See the Ask F5 solution for more information on patching.

And don’t ignore internal systems

A colleague of mine recently reminded me that after you get your externally-facing systems patched (and their certificates redeployed) you need to take care of any internal systems that might be affected as well. Why? Let’s say you have an internal server running a recent version of Linux. And it uses a recent version of OpenSSL. Someone inside your organization (anyone) could send it Heartbleed probes and perhaps pick out passwords and browsing history (or bank records) or of course, the private keys in the system itself. It’s a privilege escalation problem.

Conclusion

If you need some less technical collateral to send to up your management chain about why your F5 systems are protecting your application from Heartbleed, here’s a link to that business-friendly copy:

https://f5.com/solutions/mitigation/mitigating-openssl-heartbleed

American computer scientist Gerald Weinberg once said, “If builders built buildings the way programmers wrote programs, the first woodpecker to come along would destroy civilization.” Heartbleed is one of those woodpeckers.

We got lucky and had very limited exposure to it, so I guess we had that going for us. Hopefully you had limited exposure to it as well.