The hot story in the news this week is Volkswagen's reported brazen cheating in diesel engine emissions testing. This brought to mind a host of similar occurrences, from Samsung/HTC cheating at benchmarks to alleged cheating in SPC enterprise storage performance testing. Cynics say we should just assume we're being cheated, but is this a world in which we want to … [Read more...] about Your Mileage May Vary: Lies, Damn Lies, and Benchmarks
Everyone wants to be the best, so outrageous claims of supremacy are as old as time. In IT, these claims often revolve around synthetic benchmarks chosen to highlight a system's performance. Buyers have grown wary of these claims, smartly asking to try before they buy. But predictability is even more important than real-world testing, and this is particularly difficult for … [Read more...] about Predictable Performance Beats Maximum Speed Any Day
I've long been critical of poorly-executed performance comparisons and the "fastest is always best" mentality behind them. Who really cares if a Honda minivan accelerates quicker than a Toyota when no real-world owner will ever keep the accelerator floored from stop to highway speed? The same goes for enterprise gear: Is an over-subscribed backplane really a problem when most … [Read more...] about In Praise of Performance Comparisons
Ever since Microsoft and Intel declared that the combination of Windows and Nehalem could deliver over a million iSCSI IOPS, I've been curious about just how they did it. What black magic could push that many I/Os over a single Ethernet connection? And what was on the other end? Now Intel has revealed all in a whitepaper, and the results are surprising! What iSCSI Did Let's … [Read more...] about How Did Microsoft and Intel Get 1 Million iSCSI IOPS?
HP recently commissioned Tolly Group to benchmark their BladeSystem c7000 against the Cisco UCS 5100. The short report focuses on two results, and reads like so many competitive benchmarks in the IT industry: Tolly focuses on metrics that highlight the strength of HP's solution and the weaknesses of Cisco's. I do not dispute the accuracy of these results, and HP and Tolly are … [Read more...] about Innocence, Fairness, and Technology Benchmarks