FCoE vs. iSCSI – Making the Choice

"FCoE vs. iSCSI" isn't a battle or cage match. Your choice depends on many factors, and is more a reflection of convergence than a religious conviction. (photos by Peter Tsai, @SuperTsai)

My presentation at Interop in Las Vegas on May 11, 2011, focused on the protocols that will underpin converged storage networking in the future. My topic, assigned by network computing editor Mike Fratto, was “FCoE vs. iSCSI – Making the Choice.” Although this sounds like a grand competition between the two protocols, my take on the subject is very far from that idea. Rather than a battle, the rise of FCoE and iSCSI is part of the ascendance of convergence of storage and data networking on Ethernet.

“The notion that Fibre Channel is for data centers and iSCSI is for SMB’s and workgroups is outdated. Increases in LAN speeds and the coming of lossless Ethernet position iSCSI as a good fit for the data center. Whether your organization adopts FC or iSCSI depends on many factors like current product set, future application demands, organizational skill-set and budget. In this session we will discuss the different conditions where FC or IsCSI are the right fit, why you should use one and when to kick either to the curb.”

I began my session by pointing out that I am neither a vendor nor protocol cheerleader and don’t really have a horse in the race in terms of a transition to FCoE, iSCSI, InfiniBand, SAS, or any other protocol.  Frankly, I don’t see this as a race, and I don’t care who wins if it is one as long as IT infrastructure progresses to a more flexible state.

Converging on Convergence

The important aspect of any discussion of FCoE is not the protocol itself but the underlying shift away from specialized storage networks converging on Ethernet. ISCSI began this trend almost a decade ago, and the Ethernet roadmap leaves Fibre Channel in the dust.

I see three key elements converging to bring, if you pardon the pun, convergence of data and storage networking:

1. The wholesale adoption of Intel compatible processing architectures
2. A shift toward open systems (Windows and UNIX) for application processing
3. And the widespread adoption of IP as an internetworking protocol.

None of these “trends” is surprising or even questionable: Intel compatible open systems servers using IP dominate modern data centers.

Given this dominant processing architecture, Ethernet is a logical choice as an interconnect. No other network protocol even comes close to the market share, compatibility, and support for Ethernet.

Then we must consider the factors that drive convergence of networking protocols. After all, we have long seen a variety of different protocols in niches such as storage, voice, video, WAN, clustering, and other areas. But virtualization of servers, the need for consolidation to reduce port count and cabling, and a continuing thirst for better performance makes convergence on a single protocol a logical step for these and other areas of IT infrastructure.

If we converge on Ethernet, much will change both inside and outside the data center. Server managers will see greater flexibility and mobility of virtualized servers and blades, as well as increased performance overall. Storage managers will shift from managing esoteric networking protocols to a focus on data management and array performance. But network managers will bear the brunt of the shift, with a wider sphere of influence and new headaches from workloads that do not behave like conventional LAN applications.

The Performance Picture

Turning back to the face of storage networking, we see that one major driver for convergence is pure performance. Although Fibre Channel has an impressive roadmap, with performance doubling again and again, it can’t hold a candle to Ethernet. With historical leaps of an order of magnitude and performance, Ethernet will soon leave Fibre Channel well behind.

When iSCSI first appeared, it was hitched to fairly unimpressive Gigabit Ethernet even as Fibre Channel networks made a transition from 2 to 4 Gb. But iSCSI made a quantum leap in performance this year, transitioning to 10 Gb Ethernet even as Fibre Channel networks moved to 8 Gb. ISCSI FCoE will continue benefiting from Ethernet performance improvements in the coming years, transitioning to 40 Gb and 100 Gb. This will make 16 Gb and 32 Gb Fibre Channel look slow by comparison.

One area that is often overlooked in terms of performance is latency of I/O operations. Although iSCSI over 10 Gb Ethernet can carry 50% more data than 8 Gb Fibre Channel (thanks to more efficient encoding), it also benefits from drastically lower latency. It can handle 50% more packets than 8 Gb Fibre Channel or 10 times as many as Gigabit Ethernet. In other words, in a shared virtual environment, 10 Gb Ethernet allows more systems to get more work done in the same amount of time.

Ethernet Enhancing Data Centers

But performance is only half the story of converged Ethernet. It also supplies server connectivity, reducing the all too frequent situation where configuration and location of servers is dictated by cable availability rather than application need. This will change the face of the data center, encouraging the use of blade servers, virtualization, and flexible (dare I say “cloud”?) infrastructure. It will encourage mobility of machines, especially virtual ones, and demand new networking protocols like OpenFlow.

Ethernet required a serious upgrade to handle this workload, however. Although iSCSI works fine over just about any network, thanks to TCP/IP, FCoE and similar protocols require flow control and guaranteed lossless data delivery. This led to the development of data center bridging protocols (DCB), including priority flow control, bandwidth management, and congestion management. With the first two of these now widely available and the third following shortly, Ethernet is ready to take center stage.

FCoE vs. iSCSI

With discussion of convergence out of the way, we can finally talk about making the choice between iSCSI and FCoE. There are four main reasons to choose one protocol or the other:

1. Data center strategy
2. Performance needs
3. Desire for compatibility
4. Cost concerns

Each of these is a valid reason to pick FCoE or iSCSI in any given situation, and none is a drop–dead decision-maker. There are cases where FCoE will be cheaper than iSCSI and vice versa, for example.

Regardless of the choice between these two protocols, one element remains the same: SCSI. Nearly every enterprise block storage protocol is based on SCSI, and it is one of the seminal technologies that enabled the development of enterprise storage as an industry. Every enterprise block storage protocol, including FCoE, iSCSI, SAS, and plain old Fibre Channel, is really a transport for SCSI.

This makes the selection of protocol less relevant to operating systems and applications, since all will “see” storage the same way. There are major differences between the three SAN protocol choices, in terms of routability, availability of host and initiator hardware and software, maturity, and the availability and selection of management tools.

iSCSI has a more robust support matrix than Fibre Channel over Ethernet, with hardware and software drivers available for nearly every operating system. It is widely supported with mature storage systems available from nearly every vendor. Green field SAN designs with no existing Fibre Channel infrastructure should look no further: iSCSI is a great choice for new storage networks.

The selection of FCoE, on the other hand, is more about evolution from Fibre Channel in enterprise storage networks. There is a threefold path for Fibre Channel architects: They can continue with end-to-end Fibre Channel, and Ethernet and FCoE at the edge, or attempt to build out an end-to-end FCoE SAN. This last option is only recently possible, and is by far the least popular model for Fibre Channel architecture at the present time but will become dominant eventually.

Making the Choice

There are good reasons and bad to pick one protocol over the other, and none rises to the level of religious conviction one might see perusing blogs and tweets on the subject.

FCoE is an evolutionary transition for organizations that already have a large installed base of Fibre Channel equipment, tools, and skills. These environments can incrementally adopt Ethernet as an edge protocol while they continue to leverage the enterprise Fibre Channel storage arrays they already own. Strategically, FCoE makes perfect sense for users of “blocks” or “stacks” from vendors like Cisco, EMC, HP, and NetApp. But FCoE remains somewhat unproven, and some supporting protocols, like congestion notification and so-called Ethernet fabric technology, are immature at best when it comes to interoperability.

One common refrain when comparing FCoE and iSCSI is the efficiency of the protocols. Packaging SCSI in TCP and IP can’t be efficient, can it? But an analysis of the protocols reveals that absolute bit efficiency is very similar between Fibre Channel, FCoE, and iSCSI. Tests by Dell’s Tech Center and others show that iSCSI is fairly efficient in terms of data throughput and CPU utilization as well.

Stephen’s Stance

iSCSI is an excellent choice in situations where Fibre Channel investment is nonexistent or badly in need of wholesale upgrade, while FCoE is likely to take over in high-end enterprise shops

iSCSI is an excellent choice in situations where Fibre Channel investment is nonexistent or badly in need of wholesale upgrade. It will continue to grow based on ease of use, low cost and high performance, and widespread support, in the transition to 10 Gb Ethernet could not be simpler. FCoE, on the other hand, is likely to take over in high-end enterprise shops. It is relentlessly promoted by major vendors, and it seems that they will force the upgrade eventually. But some areas are still not ready for prime time, and buyers should beware of grandiose promises at this point.

In counterpoint, one may ask the question of why we chose Ethernet at all. It required much work, and unnatural acts like DCB, to prepare Ethernet to become the dominant protocol for convergence. Why not use InfiniBand instead, since it already works, has widespread implementation, excellent performance and scalability, as well as interoperability and hardware availability? Price is one concern, but the major factor is far more basic: No one doubts that Ethernet will eventually ascend and overcome its obstacles. It is a foregone conclusion.

In retrospect, many alternative protocols might have been better suited to convergence, including ATM and even Token Ring. Although the topic of Fibre Channel over Token Ring (FCoTR) brings a smile to the faces of network and storage nerds everywhere, we all expect that fiber Channel over Ethernet (FCoE) and iSCSI will rule the day.

Photos by Peter Tsai

Watch the presentation from Interop (apologies for the poor camera angle and sound!)