It's not going to be this easy to bridge Fibre Channel and Ethernet!

Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.

Setting the Stage

FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.

Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?

The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.

My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.

Technical Considerations

Link Aggregation on CNA’s

Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.

How Do You Handle Virtual Machine Mobility?

As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?

How Should FCoE Be Implemented Over Longer Distances?

Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.

Implementation Considerations

Is TRILL Required for FCoE Networks?

This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.

Should All Switches Be Full FC Forwarders?

There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.

How Will OpenFCoE and LoM Be Used?

OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.

How Will Technologies like Zoning Interoperate?

Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.

The Big Question: When Will We See the “Killer App” For FCoE

Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.

This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.

Read Scott Lowe’s response: What Does FCoE Have To Do With VM Mobility?

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Eight Unresolved Questions About FCoE
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Converged Networking Adapters like this QLogic 8242 are all the rage, but which are supported in VMware ESX and which have the broadest coverage of DCB features?

VMware has one awesome hardware compatibility list. It’s not the breadth of it (in fact, ESX’s range of hardware support is astonishingly small) but the thorough, public way in which VMware shares this information. It’s really worth a look, and I regularly visit the list to see what’s changed.

But the thoroughness and detail of the VMware HCL can be daunting. It’s fairly easy to search for a specific piece of hardware, but it’s difficult to tell what’s supported in a general sense. This causes trouble for people who are shopping for hardware, since it’s hard to know what’s supported without an exhaustive search.

As part of the preparation for my Storage for Virtual Servers seminar series this year, I decided to do some research. I’ve boiled down certain key hardware categories into a general plain-english list of what’s in and out of the ESX HCL. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

Minimum Requirements

No one should run unsupported hardware in a production environment. So the very minimum requirement for every buyer should be a list in the VMware ESX HCL with the proper firmware, driver, and software.

FCoE also requires a flow control mechanism, preferably 802.1Qbb, and the DCBX protocol to enable communication between ports. I strongly suggest running 802.1Qaz bandwidth management as well, since it will allow you to get more performance from your CNAs than plain PFC.

Congestion management (802.1Qau) isn’t required, and I’m not sure about the state of affairs for interoperability. So that’s a “nice to have” at this point, though it will certainly become more desirable in the future.

You might also look for a CNA that supports other protocols over DCB, since iSCSI is an attractive alternative to FCoE in the data center, as is NFS. But that’s just my opinion!

FCoE CNAs for VMware ESX

Let’s kick things off with FCoE CNAs. I’m not a huge fan of FCoE in general, but I do see it as an increasingly-viable protocol for large-scale enterprise virtualization products.

It’s critical to have supported hardware, and I suggest researching specific models on the HCL before buying. But here’s the general state of affairs with regard to FCoE CNAs:

I was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such a spec!

It’s also puzzling to see that QLogic does not list 802.1Qaz Bandwidth Management (ETS) as supported for the (admittedly older) QLE8042 card. Perhaps that’s a error? But then again, Emulex doesn’t list 802.1Qbb Priority Flow Control (PFC) for the LP21000, and although FCoE doesn’t require Qbb, it’s a very good idea…

Update: Emulex tells me the LP21000 does indeed support PFC (Qbb).

Stephen’s Stance

If you’re shopping for networking gear and want to move to FCoE eventually, I hope this list will help you get started. Most OEMs supply the latest Emulex, QLogic, and Brocade CNAs with their own part number, and the long ESX HCL is testament to this fact. But, generally speaking, if you’re using a latest-generation CNA like the QLogic 8200 or Emulex OneConnect series, you ought to be fine. The Brocade CNAs look good, too, but are a little harder to find.

Emulex tells me they will add QCN support once it’s ratified, and I look forward to learning more about the Brocade and QLogic implementations. I’d also love some feedback on how well these features work, and if they’re all they’re cracked up to be. Is one vendor’s PFC or ETS as good as another’s? And does DCBX really work in practice? I believe most folks aren’t mixing HBAs widely and are instead sticking to one manufacturer. That’s probably a good idea out here on the cutting edge!

I was surprised that no Intel or Broadcom cards are listed as supporting FCoE in the HCL. The Intel X520 is impressive, and I’d taken their “open FCoE” claims to include VMware. But I guess they’re not there yet, and this seems to be a glaring omission for a major player like Intel! Similarly, Broadcom is only talking about iSCSI with VMware. Where is their CNA with VMware support?

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX FCoE CNA Compatibility in Plain English
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The podcast is now available on iTunes!

Although the topic is an open roundtable, I thought I might give my readers a peek into the areas I plan to bring up:

• Why storage is so critical to VMware admins: It’s all about the IOPS!
• EMC’s new Symmetrix V-Max and why it’s important to VMware shops
• The great FCoE/CNA debate and Cisco’s UCS home run
• Super low-end storage for the small business or lab

Of course, what will actually get discussed is anyone’s guess, but it’s good to go in with a list of topics to avoid dead air!

So dial in or listen online, and let the discussion start!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Interested in VMware and Storage? Tune In to the VMware Communities Podcast!
This post was categorized as Enterprise storage, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Sure, prices remain high, but they’re falling fast and many folks are already implementing 10 GbE in their data centers. Intel CNAs have dropped to around $900, Cisco Nexus switches are running around $1,500 per 10 GbE port and Arista has hit $400, and 10 GbE iSCSI storage arrays are available. Intel has declared that 2009 is the year of iSCSI, and server vendors are moving to make that happen, embedding 10 GbE ports on motherboards. If you’re looking to go cheap, check out D-Link’s 10 GbE iSCSI xStack array, which sells for under $13,000 and can be populated with 15 of your favorite drives.

So is 10 GbE worth the upgrade? Absolutely! Microsoft reported wire speed iSCSI to a NetApp array at WinHEC (see my writeup on wire-speed 10 GbE iSCSI on Gestalt IT). How would you like to push over a gigabyte per second with non-crazy-or-high-end hardware? iSCSI over 10 GbE works great, and Microsoft and NetApp, at least, have demonstrated that they can make use of all that bandwidth.

Then there is Fibre Channel over Ethernet. Cisco has been pushing that technology hard as a unification of datacenter interconnectivity, and Microsoft announced that they are putting together a logo test for Fibre Channel over Ethernet FCoE hardware. Storage array makers are hopping onboard, too, and angling for position in this new market. It’s not just the year of 10 GbE; 10-gig is the future of connectivity, and the future is here.

I’m moving lots of my enterprise storage and networking content to Gestalt IT so I can focus on lower-end topics here. If you want to catch my latest work on those topics, it’s worth subscribing!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | 10 GbE, iSCSI, FCoE, Microsoft, and the Future
This post was categorized as Enterprise storage, Gestalt IT. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This is part of an ongoing series of longer articles I will be posting every Sunday as part of an experiment in offering more in-depth content.

There has been a lot of discussion in the storage industry about Fibre Channel over Ethernet (FCoE), making it the toast of Storage Networking World, but this technology remains relatively unknown to end users. Like so many storage protocols before it, the $10,000 question is whether FCoE will take off like iSCSI or fizzle as a niche product like FCIP, DAFS, and so many others.

If it does succeed, another critical question is what this means for iSCSI, Fibre Channel, InfiniBand, and to a lesser extent AoE, expanded SAS, and other options for SAN storage. The enterprise data center is poised for a complete change in server connectivity, with 10 Gb Ethernet converged network adapters (CNAs) and new core switches carrying both network and storage traffic, and this holds promise, especially in virtualized environments. But CNAs do not equal FCoE, and iSCSI, conventional Fibre Channel, and other protocols are roaring ahead. What impact will FCoE really have?

Why FCoE Matters

With 8 Gb Fibre Channel and alternative storage solutions leveraging InfiniBand now available (and with 10 Gb iSCSI imminent), many would ask why we need another enterprise block storage medium. In real-world applications, FCoE at 10 Gb will likely deliver roughly the same performance as 8 Gb native FC. But FCoE will be one or two years late and (initially) more expensive. Although active standards participation and “plug fests” mean FCoE will likely be more interoperable than Fibre Channel was in its early years, the lack of support from operating system manufacturers is troubling. Plus, users will soon be able to build a very similar infrastructure by mixing iSCSI and 10 Gb Ethernet, and this will include all the advantages of IP and solid support.

So why pay more for the same performance from an untested protocol? It’s all about the future, and enterprise users will go where the market goes, just as wide availability of VHS tapes buried Betamax. Storage, network, and SAN vendors alike are lining up solidly behind FCoE as the next-generation enterprise interconnect. Although InfiniBand plays Betamax in this home video metaphor, with superior technology and availability, FCoE’s VHS camp has all the market ammunition. To paraphrase the (alleged) words of Bob Metcalfe, no matter what the technology looks like, the future of networking will be called Ethernet.

The biggest storage vendors are behind FCoE simply because they see that converging and leveraging I/O technology makes sense for them. They can swap out the physical and data link layers from Fibre Channel to Ethernet relatively easily, so the FCoE switch is an easier change than iSCSI. It is likely that they will be able to leverage commodity Ethernet hardware to reduce (their, not your) cost and increase profit margins once this switch is made. Plus, FCoE will potentially increase SAN attachment rates (and thus enterprise storage market penetration) thanks to the potential availability of converged network adapters (CNAs) on the server side, and the cost-effectiveness that sharing a CNA between network and storage implies. From the storage side, FCoE is all good.

Also see my posts on FCoE versus iSCSI and Cisco VFrame

The drive is similar on the network side. The era of differentiated SAN and LAN producers is over – all of the major networking and SAN vendors are repositioning themselves as next-generation I/O providers, setting up a battle in the network space to rival the mainframe shakeout of the 1980s and the PC wars of the 1990s. Converged I/O is the business model for connectivity vendors, and most are taking up the “data center Ethernet” (DCE) charge (also called “converged enhanced or enterprise Ethernet” or CEE) which includes FCoE as the storage protocol for virtual I/O. iSCSI is still there in a DCE world, but FCoE will take center stage for the enterprise market.

Counting the Benefits of FCoE

It may seem strange to declare an upstart like FCoE the winner when established options like InfiniBand, conventional Fibre Channel, and iSCSI are already out in the market, but this examination of the vendors indicates that it is indeed the case. Is this a case of the tail (vendors) wagging the dog (consumers)? Perhaps, but they will come along willingly given the strong case presented by converged and virtualized I/O.

Enterprise buyers are ready for a next-generation SAN technology, and some are beginning to look at 8 Gb Fibre Channel. The few that really need performance will certainly buy 8 Gb FC today, but this has little bearing on the overall prospect for FCoE. When an application requires performance and money is available, purchases will be made regardless of future strategy.

Enterprise storage and network architects are beginning to consider the implications of server consolidation and virtualization. As they see footprint shrink thanks to compact or blade servers and server virtualization, they will begin to question the proliferation of interconnects on the back end required to keep up with the I/O demands of these super servers. Already, virtual I/O purveyors like Xsigo are making hay in this market, and, as mentioned above, their SAN and LAN vendors are spreading the message, too. It won’t be long before they are convinced.

Also see my posts on VMware VDC-OS and VDC-OS vStorage

Many people mistakenly assume that DCE means pushing all protocols through a single LAN, but this is not the case. These networks will be engineered like SANs from the start, with redundant connections and transparent failover. Although storage and network connectivity will share the same physical “pipe”, they will certainly be segregated on separate VLANs and protected with quality of service technologies. They have to be separated – FCoE (lacking IP) will require a totally different network topology than LAN connections.

So Who Buys FCoE?

Note that, throughout this discussion, I am referring only to the large-scale enterprise data center storage market. Smaller corporate environments have already embraced iSCSI en masse, expanding the penetration of consolidated storage concepts beyond anything Fibre Channel could ever accomplish. And small office and home networks are beginning to embrace these concepts as well, but are relying on protocols like CIFS and AFP for file servers and may begin to look at ATA over Ethernet (AoE) and proprietary protocols like the one pushed by Zetera/NetGear instead of iSCSI.

This leaves us with a layer cake of appropriate protocols from the smallest to largest networks. But all have one thing in common: They are all converged and they are all carried in Ethernet packets. Bob Metcalfe was right!

Update: Storagebod points out that FCoE might see its first application in inter-switch links and other similar storage network infrastructure connections. And Chris Mellor at The Register points out that SAS is already displacing FC as an internal drive interconnect.

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Reality Check: The FCoE Forecast
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The pistol shot heard this week was the starting gun for FCoE, not the execution of iSCSI

To hear this week’s storage industry news reports, one might think that Wagner’s fat lady came to Storage Networking World (SNW), singing her song as the iSCSI world collapses. Storagebod wonders what iSCSI’s death will look like. Chris Mellor at The Register says “Game Over” as NetApp, QLogic, Emulex and VMware join EMC and Cisco in singing the praises of Fibre Channel over Ethernet (FCoE). Mellor suggests that the protocol will devalue Dell’s EqualLogic investment, as if HP’s acquisition of LeftHand wasn’t enough, even as fellow Register-ite, Bryan Betts disagrees.

But The Register didn’t invent the “FCoE kills iSCSI” meme – it’s just natural to imagine that these two protocols would be in a fight to the death. And if it’s a duel, then this year’s SNW conference would seem to be the first volley, as EMC introduced a FCoE Connectrix switch (based on Cisco), NetApp announced the first native FCoE array, and everyone qualified Emulex and QLogic adapters. However, despite these announcements, it’s way too early to bury iSCSI!

FCoE and iSCSI are similar in concept:

• Both rely on Ethernet physical connectivity
• Both transmit SCSI packets
• Both are aimed at date center users

But there are major differences as well:

• iSCSI is routable in an IP network
• iSCSI can use IP services like IPsec
• Software initiators can give iSCSI connectivity to any server, regardless of hardware
• FCoE will require converged network adapters (CNAs), while iSCSI can run on any Ethernet adapter
• FCoE will start at 10 Gb, while iSCSI can operate at just about any speed

Looking at this list, one might realize that FCoE is really a competitor for faster-than-4 Gb Fibre Channel. It’s not just a data center product, it’s an enterprise (read high-end and expensive) product, and that’s exactly where it will flourish. I have no doubt that Cisco and Brocade will successfully transition their Fibre Channel product lines to FCoE, and that QLogic and Emulex will sell a gazillion CNAs. But what about the rest of the market?

VMware’s adoption has shown that there is a taste for shared, networked storage outside the rarified budgets of the enterprise world. So far, no storage protocol has won the midrange and virtual server market, with Fibre Channel, iSCSI, and NFS duking it out along side internal SAS and SATA and the odd InfiniBand and external SAS solution. Although CNAs and FCoE ought to work fine in the virtual data center, not everyone will have a taste for them. There will always be plenty of folks who just want inexpensive external networked storage arrays, and iSCSI is the best thing they’re likely to see any time soon!

And iSCSI isn’t done evolving. Folks like Mellor, Chuck Hollis, and Storagebod are lauding FCoE at 10 gigabit speeds, but seem to forget that iSCSI can run at that speed, too. It can also run on the same CNAs and enterprise switches. And although wide(r)-area SANs aren’t common, I’m beginning to see some interest in leveraging the routability and other advanced features of IP in the storage world. iSCSI still has some cards to play! And the non-enterprise space isn’t nearly as awful as some make it sound – it is and will remain a bigger, more diverse market than the high end, and there are some serious buyers that will never get into FCoE.

Right now, the SAN world is expanding like it hasn’t done for years. iSCSI deployments are accelerating, growing the whole market. Sure, FCoE will probably completely replace old-school Fibre Channel over the next five years. But it will have to share the market with the now well-established iSCSI. It looks to me like Dell and HP made smart investments.

Update: More coverage on the topic:

• Doug Rainbolt from Alacritech is skeptical of the drivers for FCoE
• David Dale from NetApp feels that FCoE is unlikely to intrude on the iSCSI “sweet spot”

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Is the FCoE Starting Pistol Aimed at iSCSI?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.