Bare Metal Blog 2013 Performance Report

#BareMetalBlog #F5 releases a new iteration of the ADC industry’s repeatable, transparent performance report.

For those of you new to the Bare Metal Blog series, the entire list of posts can be found here.

This week F5 released the 2013 Performance report. This is an important report for the industry, because it is not marketing slideware. It is actual tests, performed on actual gear, with not only results, but configurations and test environment information also. By design, it is open – you can, assuming you have the proper gear, reproduce these tests exactly in your own environment, tweak the settings to reflect your local network, and run tests that have meaning explicitly for your organization.

The testing (and much of the writing) for this report was performed primarily by one of our very smart Product Management Engineers, Jeff Apple. His indefatigable efforts brought the test results forward in a way that we were able to craft it into the report, and his experience with previous performance reports added volumes of quality information to the written portion. While there were others involved, he is no doubt the lynchpin of this operation, and deserves credit here.

Or you can trust our amazingly smart engineers to have done a fair job at testing. I’ve been through the numbers a zillion times, asked questions where things seemed odd or out of place, put them together a bunch of different ways, and honestly, the results are solid. It would be easy for competitors to claim we weighted the testing in our own favor, except for two things. First off is the openness. You can look at the configuration files for the test equipment and for any piece of gear tested to see (assuming you know the device you’re looking at configs for) that our team really did go out of their way to show these ADCs in their best light. In one test, we even accepted less-than-optimal performance out of F5 gear in order to keep the test fair to all devices tested.

The test environment was actually customized slightly for each device to show it in the best possible light – if you’re familiar with Ixia gear, the SimUsers was tweaked to the best setting for the device in question. About the only bit you could nit-pick these test results on is that some devices were not licensed at their maximum (we do have to pay for them), but F5 gear was (we didn’t have to pay for them). But where it appeared software license limiting was the bottleneck factor in a test, we pointed it out.

I did help write the report, and honestly, it was very reminiscent of Network Computing Lab testing back when we were comparing products. Jeff did a great job of providing information, not spin. Fairness to the vendors, but with an interest in helping you determine what is best for your environment. Toward that end, some of our engineers even came up with a new way to look at the data that offers a far better picture than just a single test. More on that next week.

Some have asked me “Speeds and feeds, why should we care at this point?” and my answer is simple. It is not just speeds and feeds, it is network performance, particularly at the very intensive layer seven. Want to know if your chosen vendor has enough CPU cycles left to handle offloading encryption? This report will use test results to help give you an idea. Of course a report done elsewhere cannot tell you about your explicit environment, but with the large array of things we are asking ADCs to do, their ability to perform is increasingly critical to the performance of your overall web presence.

So take a peek, check out what your vendor and the competition are capable of. Figure out if you are getting what you need if you are just considering a purchase. Some of these limitations are pretty extreme compared to how a product is sold, so it’s worth knowing what you have or will be getting. This is a good place to point out too that some products actually outperformed their marketing numbers, which they were, and why we suppose they did so is discussed in the report, but it is always nice to see vendors promising the moon and delivering the stars.

It really isn’t all about speeds and feeds in the ADC market these days, but given this report, a list of add-on functionality available from your vendor, and a serious look at the impact adding required functionality may have on these baselines can help you understand your needs now and going forward.