Mat Lee on Heartbleed: How a Keep-Alive Bug Shook Down the Web

aNewDomain.net — If you’ve been on the Internet lately — and changing your passwords and wondering why everyone else is changing theirs with the hugest of urgency — you’ve heard about the Heartbleed Bug.

Calling it a bug makes it sound like a little problem easily quashed.

It isn’t.

Heartbleed is a heartbreaking vulnerability in some of the web’s most-trusted cryptographic implementations, OpenSSL. The bug, unnoticed for two years, effects the heartbeat extension in OpenSSL that keeps a line open between what were thought to be secure servers.

Click here for my colleague Brian Burgess’ explainer on Heartbleed and quick ways to check if your server or others you routinely connect with are vulnerable — before you change passwords.

Backing up, it’s turned out there was a problem with OpenSSL’s implementation of the TLS / DTLS heartbeat extension. This extension allows so-called “keep alive functionality” without requiring constant and continual server-to-server renegotiations.

Sort of like a heartbeat, get it?

The Heartbleed bug feasibly has for two years been an open door to let in attackers. Through it, they could siphon off a 64K chunk of data in raw RAM with each heartbeat, which could have all sorts of interesting bits of information inside once parsed. Passwords, account numbers, you name it.

The keys to the kingdom might be in there — or nothing might be in there. The net effect is like playing the slots over and over and over. Hackers could use this bug — we don’t yet know how many did or if any did — to do this as many times as they wanted. And with zero footprint.

Now remember, this is only a serious problem if the server in question did not have perfect forward secrecy implemented properly. And it’s a big pain in the butt at best if they did.

Perfect forward secrecy generates a new random public key for each session, discarding the old key. This means that even if attackers were able to siphon off a key in that 64K chunk, it would only be good for the session that generated it, thus protecting all previous sessions. For more information on perfect forward secrecy, check out the Wiki.

The people who discovered the bug were Riku, Antti and Matti at Codenomicon and Neel Mehta of Google Security. The former are now running the Heartbeat site, which you should check out.

But in doing a little research, I came across this Heartbleed Subreddit post written by Ryan Walsh of The Walsh Foundations saying he inadvertently discovered it on March 25, 2014 while changing webservers at his day job.

He says:

I only found the bug because we’re changing webservers at my ‘day job’ and as the systems analyst, I was asked to ensure that we were compliant with federal data processing regulations. I went to Qualys.com and used their SSL tester to have a quick idea what was going on before I got into the servers and started monkeying around.”

Walsh then posted the reports he generated with their tools. You can check out that and his side of the story here. Find the whole of the Subreddit post here. Did Ryan Walsh really discover it first? I’m of the opinion that he did. I would love to hear your thoughts in the comments below.

I should also mention that Ryan makes it sound like snagging 64K sections of RAM and sniffing it for keys is the least of the server admin problems. That’s because, at least according to his site, it’s also possible to display the contents of an SQL database by exploiting OpenSSL.

Something about the flaw helping you drop into a root shell while monitoring the uplink. From his site:

Pair all of that with a single uncaught, unsanitized user-facing database input, and this type of attack can display the contents of an otherwise-secure SQL database … I haven’t found a DB input on our form that allows this specifically, but anything is possible programmatically once you’re in …”

He continued:

I want to state that I don’t believe this scenario has actually happened yet, my concern is that moving forward some hacker is going to identify this vulnerability and exploit it to steal data from the servers in question.”

Pretty scary stuff.

Whether Ryan Walsh first discovered it or someone else, the important thing is that it was discovered before it was widely exploited. Or was it?

Seems to me that, if this was something known in the community, I would go to great lengths to keep it on the down low.

According to an advisory from the Carnegie Mellon Vulnerability Notes Database, OpenSSL versions 1.0.1 through 1.0.1f contain the heartbeat flaw. It has been fixed in OpenSSL 1.0.1g.

According to Netcraft, this roughly translates to about 17.5 percent of SSL sites accounting for about half a million certificates issued by trusted certificate authorities. You can check out Netcraft’s graph and writeup here.

So what does this all mean for everyone not running their own web server?

It’s a great reminder, for sure, to be sure you always use top notch password rules. Don’t use the same password on multiple sites. Enable two-factor authentication if at all possible. Change your passwords every now and then — and do so especially if you get an email from a site you use, telling you it’s been affected by Heartbleed.

I’ve also heard that if you are super-worried about your privacy online, you should probably just quit the Internet for a couple of weeks. This sounds extreme to me. But here’s what analyst Bruce Schneier said about it when word got out about Heartbleed:

 On (a) scale of one to 10, this is an 11.”

Then he goes on to say:

The bug has been patched. After you patch your systems, you have to get a new public/private key pair, update your SSL certificate, and then change every password that could potentially be affected. I’m hearing that the CAs are completely clogged, trying to reissue so many new certificates. And I’m not sure we have anything close to the infrastructure necessary to revoke half a million certificates.”

This means that if you change your password before the server has been patched, you just wasted another password.

You need to wait until the server has been patched to a non-vulnerable OpenSSL version, and the bad certificates have been revoked and new ones installed. Then it should be safe to change your password. Click here for Brian Burgess’ advice on how to check on that before you change yet more passwords or any at all.

So, like a user named Arma wrote in the Tor blog post:

 If you need strong anonymity or privacy on the Internet, you might want to stay away from the Internet entirely for the next few days while things settle.”

Of course we at aNewDomain.net can’t just stop using the Internet for a couple of weeks. We have a site to run and we’ve assured ourselves that aNewDomain is uneffected.

But I do like what was recommended in a blog post by Sean Cassidy. He says we should “pay money for security audits of critical security infrastructure like OpenSSL.”

I couldn’t agree more.

For a really cool deep look at the code behind this little bug, Sean Cassidy did a great blog post about it over on his programming blog.

There’s also a really great video on Vimeo here, posted by Elastica, Inc., explaining the flaw and its high-level mechanics. It’s well worth watching if you are so inclined.

An article in Bloomberg Businessweek reveals Heartbleed also affects a bunch of networking hardware. It says Cisco Systems, Inc. and Juniper Networks, Inc. reported some of their networking products are vulnerable. Check if other servers you use were affected at any time since the 2012 bug appeared by looking at Brian Burgess’ collection of links here. 

Brian Krebs also has a great blog post about what measures you can take to protect yourself here.

Also check out what Steve Gibson of TWiT’s Security Now podcast had to say about it. Find it on episode 450. We also talk about it on episode 91 of Yet Another Tech Show (YATs).

(UPDATE 4/11/14) According to a post on the Sidney Morning Herald, German software developer Robin Seggelmann has taken responsibility for the bug, saying, “I was working on improving OpenSSL and submitted numerous bug fixes and added new features. In one of the new features, unfortunately, I missed validating a variable containing a length.” This will of course add fuel to the conspiracy fire. Is he a CIA spook scapegoat? Or did he actually just make a mistake, as humans tend to? I’m leaning towards the latter.

For aNewDomain.net, I’m Mat Lee.

Based in Kalispell, Montana, Mat Lee is a senior editor and podcaster at aNewDomain.net. He is yet another teamBYTE vet on our aNewDomain.net edit team and he hosts the popular Yet Another Tech Show (YATS). He is an Android man. Email Mat at Mat@aNewDomain.net and follow him on Google+, where he is +Mat Lee.