Single Point of Failure Is Not Just a Hardware Issue

As those of you close to me know, I have an interest in historical military firearms. As a veteran, I of course have an interest in the M-16, since I used it, but otherwise my tastes run more to World War II era weapons of all countries. Interestingly, in the United States a law change in 1986 made it rather difficult to own a fully automatic weapon, and a change created by a special form of license allows collectors like myself to buy and own them under certain conditions, but the limitation that after 1986 no automatic weapon can be produced in the US or imported except to be sold to military and law-enforcement agencies, which makes the cost of pre-1986 automatic weapons prohibitive. The sense or nonsense of this series of laws is not the point of this blog, and you can guess, as a collector, where I fall on the issue. The real point is that this whole vast body of law stopped the production of M-16s for civilian use in the US… But it did not stop the creation of automatic M-16 style weapons for civilian use. Yes, you read that right.

The civilian version of the M-16 is the AR-15. The primary difference between the two weapons is that the AR-15 is semi-automatic only. You can fire it as fast as you can pull the trigger, but you can’t empty a clip with a single pull. The AR-15 is still produced in some quantity in the US, and retails for $600 to $2000 USD depending upon brand and model. Shortly after the M-16 and other full-auto weapons were banned in the US, a simple workaround was created. Buy an AR-15, replace select parts with non-banned M-16 parts, and drop a thing called an auto-sear in. Nearly instant full-auto AR-15.  The laws evolved, and at this point owning a drop-in auto sear AND an AR-15 is considered felony possession of an automatic weapon, and legal drop-in auto sears sell for many times the cost of an AR-15 because the auto-sear itself is now classified as a machine gun. Weird solution, but it solved the “problem” of people buying them and converting AR-15s.

Typical drop-in auto sear - quarter-bore.com

And the moral of the story? It’s the part you weren’t thinking about when you designed a solution that will get you every time.

When you’re going through your “No single point of failure” routine, make sure you account for every bit of software that the system relies upon. Most of the time “single point of failure” is seen as the hardware that provides ingress to the application, the core application itself, and the database. But most applications these days have external links – we’ll call them auto-sears – to web services, SOA enabled legacy apps, external file systems, ADS… The list goes on and on.

This post is just a friendly reminder to diagram all dependencies and make certain that your “no single point of failure” architecture doesn’t have an auto-sear loophole in the software architecture side. The law had the time to evolve when the auto-sear loophole became obvious, you won’t have that luxury, because a single point of failure you don’t cover in your architectural planning will come back to bite you in emergency mode. There are tools to help you figure out what your dependencies are, even a simple link-checker will give you a list of URLs that your site includes, though going through them to find critical paths that could bring your site down is a manual job… It’s better than not figuring this stuff out ahead of time.

Some of you are going “that’s silly, everyone knows that”. To which I calmly respond “Intellectually, yes. But most organizations have at-risk applications that have not had their entire software architecture reviewed. Either there is a belief at the time the application was deployed that it shouldn’t operate without critical piece X (which may or may not be a correct belief) or there wasn’t time to go through everything because we’re constantly doing more with less.” Either way, don’t wait until it is an emergency, go find those auto-sears and close the loopholes now.

As virtualization and cloud computing continue to make inroads in the enterprise, you’ll have the perfect opportunity to identify some of these issues when you move the applications. If they fail due to failed access to some required resource, check and make certain the resource is not a single point of failure, then fix the original problem.

And no, I don’t own either an AR-15 or an M-16. I’ve looked at AR-15s, but my preferred style is one that’s not common (the A1 variant for those in-the-know), so I’ve not yet found one I was willing to purchase.

I have been bitten by the software single-point-of-failure issue though. I wish I could say “just once”.