Don’t Confuse A Rubber Stamp With Validation

There are many instances in the world where third-party verification of thoughts and ideas are just a useful thing to have. Cases where the vested interest of one party makes their opinion suspect, even if it is unbiased. For those cases we have a whole collection of organizations and corporations that will research and verify, test and certify, validate and verify, whatever, depending upon the issue and the needs of the target audience. A good example of this that I know of for the obvious reasons is gluten-free foods. There is a “certified gluten free” program in the US, but some of the testers are more thorough than others, and the label is sometimes misleading if not outright wrong. Other foods are marked (not certified) as gluten free, but are actually produced on lines that process foods containing gluten, which means that the ingredients are technically gluten free, but the final product may be contaminated with gluten molecules. It gets worse if you have an actual Gluten allergy (as opposed to going on a GF fad diet because US labeling laws only require you report use of wheat in your product, ignoring other gluten sources) You see the same type of thing in nearly every market. “Independent” test labs are popular in the high-tech business, but the definition of “independent” is stretched when you have to pay the company to do the testing. Of course those who pay are paying for results they can use, and while some organizations are good at taking negative feedback and using it to improve, since the funds for independent test labs generally come out of the marketing budget, most aren’t. They’re paying for marketing collateral, not feedback on improving the product.

And the same is often true of your use of outside professionals. In my day I have seen some uses of outside IT consultants that to this day leave me shaking my head. One place I worked was paying a suit-coat international consulting firm for an array of security services, including penetration testing. As the security staff walked out with one of the on-site consultants one day, he said “Oh, I forgot my briefcase”, walked back through the doors – where the guard let him pass because he had just come out – and quickly went from desk to desk in a user pod just inside the doors. It didn’t take him long to find usernames and passwords on sticky notes, and, jotting a few down, he left. Upon returning to his hotel, he called the penetration team and passed them actual user credentials.

Image Courtesy of LiveGlutenFreely.com

I was not on the security team, but I was in their department. This seemed like completely fair game to me. Those ne’er-do-wells who want into your organization will not play by a single rule in the book, and it only takes an unmanaged second for a visitor to your building to come away with useful information – from subnet IDs to usernames and passwords. It didn’t seem so to the security team, and it certainly didn’t seem so to the organization’s VP, who called the consulting firm and threatened to cancel their high-dollar contract over it. These results were mandatorily reported over said VP’s head, and he was not thrilled that there were multiple penetrations. My thought was that we should have taken the incident to heart and changed policy – either enforced the “don’t write down your credentials” rule, or enforce the “no visitors in the building unaccompanied” rule. But my thoughts were definitely in the minority. Needless to say, any security audit done after this affair was a rubber stamp, not a validation, and the outrageous fees the organization paid to said firm were wasted – not on their account, but on ours.

At another employer, we were seeking a replacement for a mission-critical system that was twenty years old. Not only was it twenty years old, but we had bought the source, and the language it was written in was twenty years out of date. I’m pretty certain that, had they been able to find programmers for the old system, it would still be running today. But they couldn’t, so they were looking at replacing it with something updated. There were several vendors who provided the specific vertical market solution required, and there were definite “camps” concerning which solution was best. So IT hired an outside business systems consulting firm for a hefty fee to help navigate the requirements and selection waters. This firm came in, reviewed all of the data, went and interviewed managers, and then decided that… Surprise! The system preferred by those who write the consulting checks was the system to choose. It was very much not a fit for the environment it was being put into, the company was too small, the source too immature, and the design to burdened. But the price was… Outrageously cheap. So by kicking a percentage of the cost over to a consulting firm, a rubber stamp of the selection was purchased, but the organization was not served. Indeed, in the end, the product by a top-tier vendor, whose sticker price (with install and consulting to get up and running) would have been cheaper and would have cut a year or more of over-runs off of the project.

And the third, which I’ve mentioned on this blog more than once, was the time that one of the top three analyst firms in the world gave me three different answers to “who holds the largest database market share” based upon whom I talked to and how I worded my questions. For the amount we were paying them every year, giving the answer they thought I wanted was not providing service. Wonder if they still rubber stamp rather than answer direct questions.

Sometimes, the politics of the situation dictate that even though you know you are right, you have to go get third-party help to validate your decisions. As long as there are reasonable people disagreeing on the best solution, a third party is a good solution if it is an either-or choice. And third party validation that your data is secure is certainly a worthwhile proposition. But make sure you’re getting impartial third parties to validate choices, and make sure they’re given the background and leeway to help you come to the best solution. In all of the cases above, the money was completely wasted, and that’s just not good for the organization. Bring in specialists, get a champion for each solution to tell them why it should be selected, let them research, and take their advice to heart. Don’t pay them to tell you what you’d already decided, that’s not helping anyone, if you want to run IT systems by fiat, then do so and save the money.

And good IT management knows that. Most organizations navigate these waters most of the time with no problems and manage to get the most out of their consulting and analyst help. Just be aware when you’re leaning away from unbiased, and don’t jade a project with it. It is also good to remember when you read claims by vendors touting third party validation – including us – to look the gift horse in the mouth. Of course they paid for that validation in one way or another, what you have to decide is whether the validating organization is an authority or a rubber stamp factory. About a year ago we took to testing our own stuff and publishing the test data on DevCentral so you and others could pick it up and validate it yourselves. But third party references are also important, so we still have those too, and they’re useful, as long as you scrutinize the source.

Just keep doing what’s best for the organization, and even though less-than-optimal choices will be made along the way, you’ll recover from them. When you’re making 500 choices a year about technology, apps, storage, networking gear, staffing, etc etc etc. there will be road bumps. Just as long as they were made in good faith and not with fiat by rubber stamp, it’s a sign of institutional health.

Related Blogs:

Published Feb 17, 2011

Version 1.0