Multi-Hop FCoE Is Not Ready For Prime Time (Yet)

Apparently my “bias” against FCoE is showing. I asked a question, Will 16 Gb Fibre Channel Derail FCoE?, and stirred up controversy and a series of responses: Metz, ReamsOnisick and lots of Twitter talk. Although FCoE wasn’t really the topic of that little post, some readers criticized my statement that FCoE isn’t “really ready for prime time at this point.” So let’s talk about that.

Ready For Prime Time?

Now, “ready for prime time” isn’t really a technical term with a defined meaning, and perhaps this is the root of our issue. “Prime time” refers to the weekday night hours that have  traditionally been popular with television viewers and from which network ratings are derived. A program in prime time must have broad appeal and be developed well enough for a good long run if it becomes popular. It doesn’t need to be popular yet, but it must be ready for mass market acceptance.

J Metz goes out of his way to argue that FCoE really is “ready for prime time”, refuting four “statements” attributed to critics (including me). But he appears to be using a different definition of that term, suggesting that Cisco’s product GA is sufficient. This is his opinion, but I don’t share it. I think it takes much more than a few initial products and deployments for any technology to be ready for prime time, especially in storage!

Multi-Hop Standards

My article poses the question, “why use a 10 Gb Ethernet standard that remains in flux when 16 Gb FC is shaping up nicely?” Looking back, I agree that the standards aren’t what’s in flux so much as the interpretation and implementation of them. Although the standards will evolve (and are already evolving), they are fixed and functional today.

The standards for DCB are done, and implementation and interoperability is looking good (with the exception of QCN, which is of questionable value). But multi-hop FCoE needs way more than DCB. FC-BB5 is the real standard for placing Fibre Channel over an Ethernet backbone, and that’s been accepted for a long time. Real scalable FCoE networks will also probably need a datacenter fabric, and Cisco’s TRILL-esque FabricPath technology is closest to some kind of standard for that.

But real, functioning end-to-end multi-hop FCoE networks need more than standards, they need consistent and predictable implementation, and that picture is a lot less clear. For every confident Metz, Miniman, or Onisick there’s a Ferro, Fratto, or Bourke who continue to question the implementation of, and need for, these standards.

Multi-Hop Implementation

I later call Fibre Channel Forwarding and Ethernet fabric technology “decidedly experimental.” Metz counters that Cisco has a functional implementation, and I do not doubt that. But one company’s recent GA status doesn’t make Multi-Hop FCoE “ready for prime time” by my standards. By his own admission, “multihop FCoE for Director-Class systems (the most common for Aggregation and Core deployments) has only been available for two months.”

I trust that Cisco and Metz have a working implementation, but not enough to go out telling enterprise storage administrators to take the plunge on multi-hop FCoE in general or even Cisco’s product in particular. Give it a little more time to mature, and give me a reference customer or two. And it would be nice to have more than one vendor to buy from.

FCoE Interoperability

I also state that Multi-Hop FCoE interoperability “is a serious question” and Metz points out that HP and Cisco have an interoperable solution. But his example is an FEX module for HP blade servers, not an FCoE-capable switch that interacts correctly with Cisco Nexus using standards-based Multi-Hop FCoE technology. It’s not even using FabricPath, let alone TRILL or some other FCoE standard!

This whole interoperability conversation reminds me of the “Fool the Guesser” scene in The Jerk: Your interoperability is right here, between the ashtrays and the thimbles. As long as you want to connect this very specific thing with that very specific thing, ignoring the rest of the world of products, you’re interoperable. And don’t ask for multi-vendor FC forwarding yet.

Metz also makes a non-sequitur suggestion that someone wants the industry to wait for Brocade, but I’ve never said anything of the sort. We don’t need everyone to proceed, but we do need more than one company. Standards only matter if they help us do something productive and positive, and single-vendor standards might as well not exist at all.

The bottom line is that we do not have interoperability of FCoE switches today, and we won’t have it for a long, long time. This is not Cisco’s fault, since they’re closest to standards-compliant, but it’s the truth. Eventually someone (Juniper? HP? Brocade? Dell?) will come out with a standards-compliant FCF/TRILL FCoE switch and they will issue a joint press release with Cisco and the industry will rejoice. But most customers probably won’t mix switch vendors anyway…

FCoE Adoption

This big V8 is a "mild hybrid", delivering all of the "warm fuzzies" of alternative fuels without changing the world...

Metz’ final point is to refute something I’ve often said: That “no one” is using FCoE today. Truly, I say this more for laughs and to provoke thoughtful questions than as a statement of fact. I know that lots of customers are using edge-only FCoE in critical production environments with Cisco UCS today. In fact, I consider edge-only FCoE to be a sound practice and do recommend it to buyers of high-end enterprise IT gear.

But edge-only FCoE adoption is a double-edged sword (if you pardon the pun) for proponents of convergence. It benefits customers with simplified client connectivity, delivering much of the benefit of convergence in an easy-to-adopt package. And it gets the protocol out there in production, offering a path to an Ethernet-based SAN future. But it might just short-circuit the value proposition for full end-to-end FCoE, blunting its impact and slowing the urgency for exactly the kind of customers who might adopt FCF switches.

Metz and I have often talked about real customer adoption, and he assures me that there are customers of “Full Monty FCoE” out there, but they’re not talking yet. After all, it’s only been available for two months.

Stephen’s Stance

I leave it to the reader (and the buyer) to decide if FCoE is ready for prime time.

  • First, they have to define FCoE: Does edge-only count?
  • Then they have to decide if they have a use case that some flavor of FCoE fits.
  • Then there’s the real question of risk: Are you ready to take the plunge?

I know that a number of FCoE-related standards are settled, and I know that there are products in the market and even some limited multi-vendor compatibility. I even accept that some customers are deploying real “Full Monty FCoE” in production. But I just can’t recommend that technology yet: It’s not prudent, widespread, and low-risk, so I say it’s not ready for prime time.

  • Anonymous

    OK, so I am a networking guy, so don’t beat up on me too badly, but, at the end of the day, we are not talking about replacing FC with something else (I’m looking at you Greg), but talking about changes at the bottom of the stack around the underlying transport for FC?

    I think there is a tendency to view FCoE as a special case and I am not sure that is fair or warranted.  I would argue that its no different than any other change in transport, say from 4G FC to 8G FC to 16G FC:

    – It will the time for implementations to mature, even when the standards are baked
    – Interoperability is not a given – look at all the interoperability modes we had to create for the MDS to work with Brocade, McData, etc for plain old FC
    – You don’t need to deploy it everywhere–just where it makes sense (and is capable)

    I don’t think the intent of FCoE has ever been to displace FC as much as to give a path fwd for folks that want to preserve their FC environment but also want to take advantage of convergence, simplification, etc.  In the access layer, its a no-brainer; moving towards the core, I don’t think its a binary ready/not-ready discussion as much as understanding the performance and feature envelope and seeing how it matches up against a particular set of requirements.

    OK, I’ll go back into my wiring closet now…. :)


    Omar Sultan (@omarsultan:disqus)

  • tonybourke

    Great writeup on this surprisingly controversial subject :)

    I’ve long said that edge FCoE is a very different animal than multi-hop FCoE (hop defined as either an FC hop or an Ethernet hop). They’re at two very different levels of adoption, two very different levels of interopability, and two very different levels of deployment impact. 

    Edge FCoE is used widely, and it works fantastically well. It’s great for blade systems, and great for Cisco UCS (and now HP).  

    Edge FCoE can interoperate with any FC switch that runs NPIV, not just a Cisco MDS. And deploying edge FCoE doesn’t require your network to run DCB switches with FC stacks, PFC, etc. So the impact is minimal. 

    It’s all FCoE, but the impact is very different. 

    Edge FCoE is prudent, widespred, and low-risk. 

    Multi-hop FCoE is not widespread by any means, and because it isn’t wide-spread we can’t really definitively say it’s low-risk, and therefore it’s tough to say it’s prudent in most situations (where iSCSI and native FC would be).

  • sfoskett

    Thank you! We agree! Edge FCoE is great. Full-Monty FCoE is just too new to get behind. I’m a slow dinosaur!

  • sfoskett

    This is a very smart response, for a networking guy! Convergence on Ethernet makes too much sense long term to not happen. The only question is when. I say “when” isn’t now for the majority of buyers looking to replace FC switches and won’t be for a while. In fact, I say maybe 16 Gb FC will give a second wind and hold off Ethernet for a good long while. Others say “let’s jump in with both feet! The products are fine!” I’m not them.

  • Jon Hudson

    Totally agree!  

    What we all call FC is really FCP over FC. FCoE is basically FCP over DCB.

    While one shouldn’t undercut the value of the data-link layer, in many ways we are just replacing data-link-like layers and changing timing on physical so FCP can run on Ethernet. DCB is FCP’s passport to strange new worlds. 

    So long term I see FCoE as the Fibre Channel Protocol expanding into new areas. (Secret: you still get plogi’s and flogi’s)

    And I also REALLY agree with the “right tool for the right job” idea. Edge FCoE is getting very cool, and has some great applications. So does NFS over 10GbE. And iSCSI with DCB. And 16G FC. And yes, even Infiniband. (CIF is intentionally NOT mentioned)

    This is not Highlander! No one needs to loose their head. 

    Think of FCoE as a very cool shiny new tool in your box. Not a tool that replaces the whole tool box and everything in it. 

  • tonybourke

    “I am Ethernet McCloud of the clan McCloud.”

  • Brook Reams


    Many jobs, many tools. 
     – Right tool, good job. 
      – Wrong tool, not so good job.

    New tool, then you need to learn the best way to use it and what jobs are easier to do well with the new tool.  Takes time, opportunity and patience.

    Onward through the fog …

  • The_Hiker

    You talk about FCoE interoperability however, FC has been out for decades now and switch to switch interoperability pretty much doesn’t exist.  I’m not talking about NPV mode edge switches, but real honest to goodness switch – ISL – switch (e_port) interoperability.

    LSANs/VSANs/VirtualFabrics/InterVSAN routing, FC-GS-4 “enhanced” zoning.  Just a few years ago brocade got rid of their ‘interop mode’, which for years was used to connect brocades to mcdatas or cisco switches.  In order for Cisco to be interoperable with brcd/mcdata they had to create 4 different interop modes depending on what model of brocade (or mcdata) it was attachining to.  With all of the weird domainID restrictions that mcdata had, and the non-standard domain,port based zoning that brocade has.  It’s a miracle that anybody could get a brocade and a cisco to communicate with.  Don’t forget about the obfuscating of the zoneset that brocade did in 5.3.

    So please Steve, don’t go about ranting how FCoE isn’t interoperable while 16Gb FC is, b/c 1,2,4 and 8Gb FC isn’t interoperable. And until this basic problem is solved, interoperability at the FC layer in FCoE is a pipe dream.

  • Pingback: Technology Short Take #16 - - The weblog of an IT pro specializing in virtualization, storage, and servers()

  • Pingback: A Tale of Two FCoEs « The Data Center Overlords()