Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.
Setting the Stage
FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.
Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?
The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.
My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.
Technical Considerations
Link Aggregation on CNA’s
Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.
How Do You Handle Virtual Machine Mobility?
As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?
How Should FCoE Be Implemented Over Longer Distances?
Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.
Implementation Considerations
Is TRILL Required for FCoE Networks?
This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.
Should All Switches Be Full FC Forwarders?
There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.
How Will OpenFCoE and LoM Be Used?
OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.
How Will Technologies like Zoning Interoperate?
Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.
The Big Question: When Will We See the “Killer App” For FCoE
Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.
This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.
Read Scott Lowe’s response: What Does FCoE Have To Do With VM Mobility?
Hi Stephen,
Users don’t want ANY new protocol – change is challenging, especially for the highly risk-adverse people in the storage industry. What I believe the discussion is missing is that the requirements of the solutions being built by vendors will require adoption of convergence (with FCoE being one of the options) more and more – this is your “Killer App”. This does not mean that FC goes away overnight or that there aren’t some niggling details to work out on the technology, but this can’t be ignored. I wrote more about this topic here: http://wikibon.org/wiki/v/2012_Prediction%3A_10Gb_Ethernet_growth_and_the_fate_of_FC
You’ve raised a few good technical points. On one of the points – distance, I don’t see why that’s an FCoE issue – replication over distance between arrays is typically a dedicated port, so it could run IP (like SRDF) or even iSCSI. Host over distance is rarely used and iSCSI would be a better fit in that scenario.
“Users don’t want ANY new protocol” Great point! Protocols are nothing – it’s all about solutions.
Cisco UCS/vBlock is an amazing solution and showcase for FCoE. This is an example of when it will take off.
Thanks for the comment and read, Stu!
To answer the question about TRILL, it depends on if the switch is a full FCF or not. If it is, TRILL isn’t needed since it’s a regular FC fabric that just happens to run atop Ethernet. If the switch isn’t a FCF, then TRILL or something like it most certainly should be there. Or anything but spanning-tree, that is.
> On one of the points – distance
I think Stephen may have had in mind the “stretched cluster” situation, with storage shared between hosts in different locations.
My gut feel is that somebody is bound to try it, given how attractive this idea may look to WAN connectivity service providers: they can switch on all kinds of fancy QoS features available on their transmission kit (not for free, of course), and push high margin high bandwidth services as “an alternative for customer deploying a second storage system”.
An issue is that your switches need to support FCoE and given prices for upgrading many people wont have that option. UCS also doesnt necessarly mean switching to FCoE right? still can use FCoFC? and i think there is still something to be said about having storage not on the same physical switches? Although there are always pros/cons. What I would like to see however is more storage solutions that allow you to move away from big expensive storage sans similar to vmware’s storage solution that creates shared storage from your local disk. issue is that only supports 3 nodes currently.
I’ve been selling and deploying FCoE networks since their inception and hopefully I can shed some light, or at least bias, as to where things are or where they should go.
CNAs Link Aggregation or MPIO:
Data Networks use Link Aggregation and probably should. They have no intelligence as to how the upper layer protocols will handle thing. Link Aggregation requires configuration on the switch or virtual switch or virtual device context to be effective. It has difficulty working between switches or independent data networks. MPIO is a zero configuration items (with driver installation), the upper layer protocol supports it (meaning SCSI-3 and not IP-FC or other things) and works between switches and independent fabrics.
MPIO is really easy to use and rock solid. Link Aggregation still gives network admins trouble at more than half the shops I visit, and when they do it it going spanning tree hamgstrings it. I’d love to see link aggregation take a page out of the MPIO rulebook, but TCP/IP and it’s upper layer protocols have routing issues introduced layering between networks.
Virtual Machine Mobility:
Let the networks do what they do best. vNICs with vMACs move with the VM. vHBAs with vWWPNs (aka NPIV) handles this as well. When the VM moves, the data network RARPs to update the data network stack on the switch, the FCoE side logs in with it’s NPIV address and life should be good. Am I missing the question here?
How Should FCoE Be Implemented Over Longer Distances:
It’s easy to forget that both networks need routers introduced. Ethernet takes neither long distance. TCP/IP has routing functions that can be enabled in many switches to allow this. In the native-FC world we employ FC routers. We need to have WWPNs from one fabric route to another, while preserving resiliency if the link in the middle goes away.
In the old days, we bridged FC fabrics together, because that’s all we could do. As long as links stayed up (no backhoes, no bridges falling, no infrastructure updates) life was OK. We had some latency and buffer credit issues to overcome. What that link went down, we had FSPF convergence issues. When routing was introduced, we could isolate the fabrics. We can send over dark fiber, or TCP/IP via iFCP and later FCIP. (Please keep the religion out of this.)
Now converged switch vendors could implement FCoE to FCIP routing and translation capabilities to a switch, but I know of no one doing it today. Should it, the answer is likely yes — to every type of FCoE switch — no. This stuff costs more, there’s extra hardware (memory for buffer credits, FC-NAT tables, etc) and extra software. It costs money to design, make and implement. Today, we extend FCoE with FC-FC routers still (using FCIP or dark fiber).
TRILL Required for FCoE Networks:
Trill seems to borrow a lot from the FC world. In fact, Data Center Ethernet, or Convereged Enhanced Ethernet or whatever non-blocking Ethernet is called with FCoE, it borrows a lot from the FC world. I’m still an MPIO fan. If that is accomplished over TRILL, fine by me.
Should All Switches Be Full FC Forwarders?
All switches are rarely switches in the native-FC world today. We have NPV or access gateway mode, which provides pass-through. The question becomes where do you do the switching? In a storage world, decisions can be made easier, because most traffic is going from a host (server) to a device (storage) at a large fan-in ratio. We can switch higher up and closer to the storage because traffic isn’t going between hosts like in data networks.
How many layers (access, distribution, core, storage) do we have? It depends.
How Will OpenFCoE and LoM Be Used?
Probably the same way iSCSI is today. CNAs provide DMA (direct memory access) within a server and provide a short path through the OS with minimal copying of buffers. LoM introduces software stacks and memory-to-memory copies that must take place. CPU overheads at 10 GbE are typically 10% for iSCSI and 15% for NFS over straight native-FC or FCoE. The DMA and amount of copies through the software stack make a difference. If you’re virtualizing, an extra 10% adds up.
So CNAs offer better performance. Does most storage run at 10 Gb/s, or 4 Gb/s: no. Most run closer to 1 Gb/s in virtualized environments. We’ll like see LoM used like iSCSI, on less important machines and as budgets dictate.
How Will Technologies like Zoning Interoperate?
I think there’s a lot less hetrogeneousness in the data center than you think among data networks. Most are Cisco, HP, Extreme, Brocade/Foundry or Juniper. In storage it’s Cisco, Brocade or Qlogic. Interoperability exists in data networks pretty good. Interoperability exists in storage networks pretty poorly, unless you use NPV/gateway modes, which just forward traffic up to your upper layer switches. Clearly ground is being staked here.
I love VSANs and wish everyone had them. I see this as continuing to be a pain for some time to come. Will people ask for better integration? Sure. Will they get it, not likely — the industry is moving too fast to wait for that level of interoperability.
So these networks do work. We’ve been deploying them for about 3 years now, as just converged, then UCS then vBlock/Flepod. They work very, very well.
Why will people switch? Show me the money. The protocols not really different — there are some things added on for Ethernet, but not much. Where it makes sense is when it saves money. We recently collapsed a whole DC into 3 racks in a colo with a converged solution. We could have done it in one, but we want room for growth.
When people save money, they will switch. They are saving money today. Like it or not, here it comes.
Hi Stephen, I won’t argue how controversial FCoE has been, at least not with the chief instigator of most of this controversy… 🙂
Ethernet is not ideal for block I/O but I think DCB Ethernet is pretty close. If you disagree, why specifically?
In regards to Link Aggregation on CNAs while FCoE is in use, both EMC and Cisco have demonstrated how to accomplish this using vPC. This has been supported for over a year and I know of at least one customer who is doing this in a production environment. For more detail, see the latest EMC FCoE Techbook or look at the joint EMC / Cisco presentation from SNW fall.
Virtual machine mobility and FCoE is something that still needs to be addressed. If don’t actually expect it to be handled by VXLAN, NVGRE or SDN in general although I would really like to see someone start an FC-OF (FC OpenFlow) project, if for no other reason than to have some kind of response to FCoTR.
Native FCoE and very long distance doesn’t make sense. Look at the buffering requirements alone and you’ll see why. I’ve heard that up to 20-30 km could be done without being cost prohibitive but anything longer than this and you are going to want to look at native FC or possibly FC-IP.
TRILL is not a requirement. Look at the topologies that are being recommended and deployed today and not one of them requires TRILL. I can see a couple of potential solutions on the horizon that make use of a TRILL like functionality but nothing today..
Not all FCoE switches need to be forwarders. There are compelling use cases for FIP Snooping Bridges in blade servers that would argue against this. The bottom line is that customers need to be able to select the components that make the most sense for them. Since there are no data integrity / security related concerns with FCoE switches that do not contain forwarders, there is no need to exclude them from consideration.
I would personally like to see FCoE supported on LOMs and am sure that we would evaluate and, if appropriate, support such a solution if one were to become available.
In regards to FC interop, I seriously doubt we will ever see multi-vendor interoperable FC solutions unless Cisco buys Brocade. If the interop problems surrounding blade server FC switch modules didn’t force them to solve their interop issues, I don’t think anything will. It seems like this type of problem has been solved by NPV/AG mode (NPIV) and although it isn’t perfect, there is no better alternative if you have to connect two vendors together. This coming from a person who tested multi-vendor switch interop for years..
In regards to the killer app, I like what Stuart had to say on the topic..
Great comment, except for the “Cisco buys Brocade” bit which made my head explode!
I don’t think it’s fair to punt everything to “whatever works for the customer” though. My questions revolve around the potential for best practices, not just whatever works. Yes, you can do FCoE in a number of ways, but what’s best? And what will emerge as the mainstream way to do things like LACP/MPIO, TRILL, FCF, and such? It’s an honest question, and no one will be able to answer it for a few years yet.
Finally, I’m definitely not the chief instigator of an FCoE controversy. You should talk to some networking guys like Greg Ferro if you want to hear real anti-FCoE ranting!
A really excellent, lengthy, and worthwhile comment on FCoE!
Thank you!
A few points:
– Many of these unresolved questions lie in the gulf between data networking and storage networking. Ethernetworkers love LACP because they’ve never experienced MPIO as in a FC world. The same can be said of true multi path FC (emerging as TRILL et al) – if they saw what we storage people see all the time they might be less inclined to tar storage as a backwater!
– I’m not sold on the “overhead is bad” argument against LoM, IP protocols, etc that many others raise. You’re not going that far, but there’s got to be a point when LoM and OpenFCoE become so attractive cost-wise that folks accept the overhead. Just ask Intel.
– “Show me the money” is exactly right. So far, I’m not convinced that a switch to FCoE is a financial win. I’d say it’s in the win column for greenfield, but maybe not if training is involved. Some day it will probably become a serious cost saver, though (2x to 3x savings) and it might take off.
Thanks again!
Wow. I’m not usually one for a “me too” type post, but this was a fantastic summary.
The other challenge I see in some hardcore shops is logical separation vs physical separation.
A lot of good work has been done to make the logical separation option as pleasant as possible. Cisco for example has done (my opinion) a good job offering separate zoning tables options for example.
For some % of customers logical separation, no matter how clever will never be enough. If an electric signal can pass over a wire, it’s too much risk for them.
Mostly I find this to be people who have spent the 24+ hours restoring a corrupted filesystem from tape, and then applying the oracle redo logs while drinking a case of Mountain Dew.
However, having truly physical separated dual path from initiator to target is not in a happy place today. For FCoE it works fine with good CNA MPIO-tech.
The challenge is all the “other” services running on the server normally have no idea how to handle multiple interfaces on separate networks.
Now there are many interesting ways to solve that last issue, but I haven’t see anything really gain traction yet. (who’s for BGP in the hypervisor?)
Now, the above may not matter. It may end up being that the type of customers I’m talking about have already invested so much into FC/FC gear/knowledge that they aren’t likely to switch anyway. That it’s with shops that don’t have any FC/FC, or are building a new facility where FCoE will shine. Time will tell. Lets face it in some shops FCoE vs FC/FC will be a different conversation then FCoE vs iSCSI or FCoE vs NFS or (shutter) FCoE vs CIFS.
Personally I see FCoE as a new blade in a “swiss army” knife. There are always going to be servers that for whatever reason don’t want or can’t have dedicated storage adapters. For a long time they just had NFS, then added iSCSI and now they have FCoE. Options are a good thing. FC/FC+FCoE+iSCSI+NFS, now that’s a multi-protocol array 😉
And I don’t think new technology always has to be “There can be only one!” and “off with his head” and all that. This isn’t Highlander.
and try not to say things like “oh if FCoE doesn’t have X market share by Y date it’s dead, or it’s a failure”. Change comes at different speeds for different people. No big deal, no one needs to loose their head =)
I’ll also echo the comments of others, really great clean info Urban
and finally on the topic of distance, Cisco put out a great paper discussing the challenges a while a ago that I’ve always liked. http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-542809.pdf
(it’s 4am, so please excuse any typos)
hahahaha, don’t worry Steven. If that happened you’d be so busy avoiding the raining frogs, earthquakes, plagues and locusts it wouldn’t matter =)
Intel 82599 fcoe does not support windows MPIO？How can it deploy in the data center?