Any questions? Post'em

I recently facilitated an off-line exchange between Joerg Nalik, Director of Partner Solution Management at SAP Labs, and F5's Mike Lowell. I thought the discussion was worth sharing here on DevCentral, so without further ado....

 

 

(Note: Mike's in-line replies are indicated with a leading '>')

 

 

From: Nalik, Joerg [mailto:joerg.nalik@sap.com]

 

Sent: Monday, September 24, 2007 6:39 PM

 

To: Ron Carovano, Jr.

 

Subject: RE: Comparative performance testing

 

 

Hi Ron,

 

 

first of all I think this is a very nice paper,

 

 

> Thanks!

 

 

perhaps you can tell me in a few weeks what its impact is: to your customers and your competitors.

 

 

Now as a trained scientist and with some experience in this area I could start nagging about the paper until you feel really bad :) Not too much point to that because not many scientist will care about the paper and anybody about them. So here some ideas only:

 

 

Many measurements show higher F5 throughput. So why not buying 2 competitor boxes for one F5 box and getting the same?

 

This points to my price/performance question (not answered on dev central).

 

 

> Keeping the report as short as possible was a key goal for us. To make this report useful/digestible for the largest

 

> target audience we had to do a lot of trimming. I would have liked to include price/performance information because

 

> it *heavily* favors F5.

 

 

On some measurements it says Cisco doesn't offer some feature in their ACE. Do they offer it in another product line?

 

Not everybody needs to slice and dice features and products like F5 does.

 

 

> Agreed. Part of the limitation here was one of scope (i.e. keep the document short, focused on the largest target

 

> audience). From a customers point of view having multiple boxes is often less desirable because it typically

 

> increases total cost of ownership. TCO increases because the box count increases (at least 2x of each box to provide

 

> redundancy), and in the case of Cisco the devices providing each function were from different acquisitions so

 

> managing the devices is more complex (i.e. you have to learn how to manage each device in it's own way, they are not

 

> similar). TCO also increases because overhead increases: with all the features in a single box there are many

 

> efficiencies to take advantage of. Having separate devices requires that multiple devices implement many of the same

 

> technologies (TCP/IP stacks, HTTP parsing, etc) which increases overhead, latency, and risk (more points of failure).

 

> Multiple boxes also means more rack space, more heat, more power, etc.

 

>

 

> It would be an oversimplification to say that F5's way of slicing features is always better, though -- I do agree with

 

> you.

 

 

As an application guy this doesn't tell me too much because your methodology is too deep down in network terms. As business guy it's even worse. So cool thing that F5 also publishes results for SAP apps which are better to relate to a customers' business, which in the end also provides the budget for network gear.

 

 

> Agreed. This was again the result of our limitations of scope and target audience. Having said that, the relative

 

> performance of the devices as you saw in the report is very likely to be similar to most application environments.

 

> i.e. if BIG-IP was twice as fast as vendor X in the report, it's likely this 2x advantage will also apply to an SAP

 

> deployment. It would of course be an oversimplification to say that's always true, though.

 

 

The database and hardware server folks do benchmarks like this all the time. In order to make them valuable, they all agreed on a set of standardize benchmark scenarios and methodologies, which they defined and acknowledged together. Therefore their benchmark results have high credibility to customers and are apple to apple comparable. The benchmark F5 publishes tries to demonstrate transparency but in the end it is a paper ordered and paid for by F5, only done with methods and scenarios only F5 (or your contractor) uses. Therefore, F5 competition can just ignore those results and claim they are somehow manufactured and not credible. I actually asked a few network vendors last year why there is not network benchmark standard? The whole concept seemed to be alien to the network industry. I credit it to your work with SAP and maybe other app folks that the benchmark idea is making it slowly into the network world.

 

 

> I've a huge fan of standardized testing. At F5 we've been looking at this more and more in recent months. I hope to

 

> see standardized testing for the ADC space within the next 2 years, and I think F5 will play an important role in

 

> driving this. BTW, did you see the "how to create performance testing methodologies" doc linked in the report? I

 

> think it may be interesting to you.

 

>

 

> Lastly, I wanted to give you my feelings regarding the report. First, I can say without hesitation I'm much more of a

 

> scientist than anything else, I agree with you that there are very few scientists in this business. If I had my way

 

> the report would have been 100-200 pages long as a result of including significantly more detail (about everything),

 

> as well as by the inclusion of many more related angles (app testing, pricing, and much more). Sadly, this would have

 

> taken ~1 year to complete, and would have had a significantly smaller target audience (difficult to justify given

 

> business priorities). In the end, however, I'm glad we were able to produce something useful and be open and honest

 

> about it, i.e. more than just ridiculous vendor claims :) I hope this raises the bar for other vendors in our space,

 

> as that would significantly help our efforts to reach standardization.

 

>

 

>Thanks for your comments!

 

 

- Joerg