It's not going to be this easy to bridge Fibre Channel and Ethernet!

Before the holidays, I posed a question on Google+ that generated quite a bit of interest and feedback. Now that it has settled down a bit I’d like to summarize the unresolved elements to make FCoE truly a world-class storage interconnect.

Setting the Stage

FCoE has been a controversial topic in both storage and networking, and for good reason. No one would deny that Ethernet is not an ideal transport mechanism for block storage I/O. “Porting” Fibre Channel to run on Ethernet networks has been a supreme technical challenge, and many companies and individuals have labored long and hard to make FCoE a reality.

Now that FCoE is specified in the standard and has been deployed in production environments, the question turns to its future. Will it take off and seize the mantle of dominance currently held by what I like retroactively to call “Fibre Channel over Fibre Channel?” Will they coexist for the next decade, with FCoE mainly deployed in “block” environments such as Cisco UCS? Or will FCoE ultimately fail to catch on, displaced by some other storage protocol like plain FC, iSCSI, NFS, or something entirely different?

The data center needs a flexible new protocol to meet the needs of virtual environments, and convergence of storage and data networking makes a great deal of sense in these environments. This was the root of my question, and I ask it in all earnestness.

My question: What elements remain unresolved to make FCoE truly world-class? What should the vendors be prioritizing? Here are the answers I received.

Technical Considerations

Link Aggregation on CNA’s

Converged network adapters (CNA’s) allow multiple protocols to access a single Ethernet connection, but some also include multiple ports that can be aggregated. In traditional Ethernet networks, link aggregation is a respectable approach for performance and availability. But storage networks have traditionally relied on host-based MPIO software, and these features are mutually exclusive. The zeitgeist seems to be a recommendation to avoid link aggregation on CNA’s that are used for storage networks.

How Do You Handle Virtual Machine Mobility?

As I described recently, virtual machine mobility is a major technical challenge for existing networks. The VMware proposal, the VXLAN, seems to be gaining traction right now. But this is only a solution for data networking. How will FCoE SANs handle virtual machine mobility? This remains unresolved as far as I can tell, though Ethernet switch vendors have come up with their own answers. Brocade demonstrated just such a solution at Networking Field Day 2, and I know that others have answers as well. But will there be an interoperable industry solution?

How Should FCoE Be Implemented Over Longer Distances?

Fibre Channel has traditionally relied on routers and other protocols (FCIP and iFCP) to span distances, but FCoE raises the possibility of native traversal. While it is certainly possible to span distances with FCoE, this is definitely not a recommended or supported idea. Without TCP/IP, or any routing mechanism, it’s just a bad idea. But I imagine that it won’t be long before vendors decide to give it a go anyway.

Implementation Considerations

Is TRILL Required for FCoE Networks?

This has been one of my own questions since the very beginning. Clearly, edge only FCoE works just fine without TRILL. But as networks become more complicated, and virtual machines move, it seems an awfully good idea to have some protocol to alleviate East-West routing concerns. I feel much better with TRILL (or some similar Ethernet fabric technology) in a complicated FCoE network.

Should All Switches Be Full FC Forwarders?

There are number of ways to implement FCoE on Ethernet network, and not all involve building a full Fibre Channel stack in each switch. While many (including myself) assumed that FCoE implied Fibre Channel forwarding in all switches, this is clearly not the direction taken by vendors, at least initially. Perhaps the current “Ethernet forwarding” approach is only a stepping stone, or perhaps it will emerge as the dominant FCoE standard.

How Will OpenFCoE and LoM Be Used?

OpenFCoE is a software solution allowing FCoE to be run without a CNA. If this became popular, it wouldn’t be long before data center architects began looking at LAN on Motherboard (LoM) and even 10GBASE-T as a potential SAN alternative. Will this be used in the long run? It could happen, but it’s certainly not something that’s here at the moment. But OpenFCoE is a real player, especially with Intel’s backing.

How Will Technologies like Zoning Interoperate?

Many networkers are just now beginning to see the true complexity of Fibre Channel SANs. Although interoperability of higher-level Fibre Channel functions between vendors has never been a priority in “FC over FC” SANs, Ethernet could change things. I would not be at all surprised to see a groundswell of customer support demanding greater levels of interoperability from FCoE than from FC, and zoning and VSAN is the likely first beachhead.

The Big Question: When Will We See the “Killer App” For FCoE

Just about everyone agreed that the real challenge for FCoE is market acceptance. Customers aren’t yet demanding FCoE, and vendors are finding it hard to articulate a compelling case to move from “tried-and-true” FC. Convergence, cost savings, and performance have all been put forth, but customers aren’t biting. Perhaps they just need a little time and a little more proof.

This post relies extensively on feedback from a number of people, including Ivan Pepelnjak, Tony Bourke, J Metz, Dmitri Kalintsev, Derick Winkworth, David Hardaker, Juan Lage, and Corey Hines.

Read Scott Lowe’s response: What Does FCoE Have To Do With VM Mobility?

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Eight Unresolved Questions About FCoE
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Ready For Prime Time?

Now, “ready for prime time” isn’t really a technical term with a defined meaning, and perhaps this is the root of our issue. “Prime time” refers to the weekday night hours that have  traditionally been popular with television viewers and from which network ratings are derived. A program in prime time must have broad appeal and be developed well enough for a good long run if it becomes popular. It doesn’t need to be popular yet, but it must be ready for mass market acceptance.

J Metz goes out of his way to argue that FCoE really is “ready for prime time”, refuting four “statements” attributed to critics (including me). But he appears to be using a different definition of that term, suggesting that Cisco’s product GA is sufficient. This is his opinion, but I don’t share it. I think it takes much more than a few initial products and deployments for any technology to be ready for prime time, especially in storage!

Multi-Hop Standards

My article poses the question, “why use a 10 Gb Ethernet standard that remains in flux when 16 Gb FC is shaping up nicely?” Looking back, I agree that the standards aren’t what’s in flux so much as the interpretation and implementation of them. Although the standards will evolve (and are already evolving), they are fixed and functional today.

The standards for DCB are done, and implementation and interoperability is looking good (with the exception of QCN, which is of questionable value). But multi-hop FCoE needs way more than DCB. FC-BB5 is the real standard for placing Fibre Channel over an Ethernet backbone, and that’s been accepted for a long time. Real scalable FCoE networks will also probably need a datacenter fabric, and Cisco’s TRILL-esque FabricPath technology is closest to some kind of standard for that.

But real, functioning end-to-end multi-hop FCoE networks need more than standards, they need consistent and predictable implementation, and that picture is a lot less clear. For every confident Metz, Miniman, or Onisick there’s a Ferro, Fratto, or Bourke who continue to question the implementation of, and need for, these standards.

Multi-Hop Implementation

I later call Fibre Channel Forwarding and Ethernet fabric technology “decidedly experimental.” Metz counters that Cisco has a functional implementation, and I do not doubt that. But one company’s recent GA status doesn’t make Multi-Hop FCoE “ready for prime time” by my standards. By his own admission, “multihop FCoE for Director-Class systems (the most common for Aggregation and Core deployments) has only been available for two months.”

I trust that Cisco and Metz have a working implementation, but not enough to go out telling enterprise storage administrators to take the plunge on multi-hop FCoE in general or even Cisco’s product in particular. Give it a little more time to mature, and give me a reference customer or two. And it would be nice to have more than one vendor to buy from.

FCoE Interoperability

I also state that Multi-Hop FCoE interoperability “is a serious question” and Metz points out that HP and Cisco have an interoperable solution. But his example is an FEX module for HP blade servers, not an FCoE-capable switch that interacts correctly with Cisco Nexus using standards-based Multi-Hop FCoE technology. It’s not even using FabricPath, let alone TRILL or some other FCoE standard!

This whole interoperability conversation reminds me of the “Fool the Guesser” scene in The Jerk: Your interoperability is right here, between the ashtrays and the thimbles. As long as you want to connect this very specific thing with that very specific thing, ignoring the rest of the world of products, you’re interoperable. And don’t ask for multi-vendor FC forwarding yet.

Metz also makes a non-sequitur suggestion that someone wants the industry to wait for Brocade, but I’ve never said anything of the sort. We don’t need everyone to proceed, but we do need more than one company. Standards only matter if they help us do something productive and positive, and single-vendor standards might as well not exist at all.

The bottom line is that we do not have interoperability of FCoE switches today, and we won’t have it for a long, long time. This is not Cisco’s fault, since they’re closest to standards-compliant, but it’s the truth. Eventually someone (Juniper? HP? Brocade? Dell?) will come out with a standards-compliant FCF/TRILL FCoE switch and they will issue a joint press release with Cisco and the industry will rejoice. But most customers probably won’t mix switch vendors anyway…

FCoE Adoption

This big V8 is a "mild hybrid", delivering all of the "warm fuzzies" of alternative fuels without changing the world...

Metz’ final point is to refute something I’ve often said: That “no one” is using FCoE today. Truly, I say this more for laughs and to provoke thoughtful questions than as a statement of fact. I know that lots of customers are using edge-only FCoE in critical production environments with Cisco UCS today. In fact, I consider edge-only FCoE to be a sound practice and do recommend it to buyers of high-end enterprise IT gear.

But edge-only FCoE adoption is a double-edged sword (if you pardon the pun) for proponents of convergence. It benefits customers with simplified client connectivity, delivering much of the benefit of convergence in an easy-to-adopt package. And it gets the protocol out there in production, offering a path to an Ethernet-based SAN future. But it might just short-circuit the value proposition for full end-to-end FCoE, blunting its impact and slowing the urgency for exactly the kind of customers who might adopt FCF switches.

Metz and I have often talked about real customer adoption, and he assures me that there are customers of “Full Monty FCoE” out there, but they’re not talking yet. After all, it’s only been available for two months.

Stephen’s Stance

I leave it to the reader (and the buyer) to decide if FCoE is ready for prime time.

• First, they have to define FCoE: Does edge-only count?
• Then they have to decide if they have a use case that some flavor of FCoE fits.
• Then there’s the real question of risk: Are you ready to take the plunge?

I know that a number of FCoE-related standards are settled, and I know that there are products in the market and even some limited multi-vendor compatibility. I even accept that some customers are deploying real “Full Monty FCoE” in production. But I just can’t recommend that technology yet: It’s not prudent, widespread, and low-risk, so I say it’s not ready for prime time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Multi-Hop FCoE Is Not Ready For Prime Time (Yet)
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

I stirred up a lot of controversy last week after posting what I thought was a fairly innocuous question: Will 16 Gb Fibre Channel Derail FCoE? That short post focused on the prospects for 16 Gb FC and was the result of questions I received from the audience at Interop New York. Yet the resulting controversy is all about the fitness of FCoE and my personal motivations. So I suppose it’s time to clarify my position more fully. This will be a multi-part series, since it’s getting kind of long, but let me spoil the ending for you: I believe that FCoE will displace traditional Fibre Channel (“FCoFC”) in about a decade.

This Is Storage!

The enterprise storage industry and market is very, very different from other sectors of IT. It can seem illogical and even foolish to outsiders, but there’s a method to the madness. It isn’t easy to “do storage right” and failures are catastrophic.

This is why storage architects are in love with “best practices” and hardware compatibility lists. And it’s also why I never recommend any solution that isn’t prudent, low-risk, and in widespread usage. Read that again. As much as I love startups and cool new technologies, I never recommend their products for enterprise production use. Storage people have always been cautious, and I’m a storage guy.

There’s also an extremely long tail for storage protocols since minor compatibility and interoperability issues can have major consequences, too. It’s crazy, really, that we still use SCSI as our primary protocol, but we do. Rather than replacing it with something more suited to virtualized environments we extend it with nips and tucks to make it keep working. As much as I’d like to just ditch SCSI and use a protocol that can handle unreliable networks, I know that won’t happen for a long time.

One must also consider the useful lifetime of enterprise storage devices and architectures. Today’s buyers will continue to use and grow their SAN for years to come. If history is a guide, it will take many years for anything to replace 8 Gb Fibre Channel as the majority datacenter SAN interconnect regardless of how awesome the replacement product is.

Just An Ignorant Old Man

So this is my axe, and this is how I grind it:

1. I don’t care if anyone buys anything from anyone, let alone what they buy. I work for no man and I don’t need any vendor’s love. I take money from most companies from time to time, but none owns my loyalty.
2. Although I love cutting-edge tech, I’m professionally very conservative and base all of my recommendations on my three-part definition of “Best Practice”. That includes the part about “widespread usage” – let someone else risk his job on cool new technology.
3. I’ve watched all this happen before, and know it takes a long, long time for serious adoption of any storage technology. Real value is about way more than technical elegance, but it does eventually show through.
4. I don’t care about protocols per se, I care about what customers do with them. I’m not an idiot or a luddite but I’m not a willfully-ignorant cheerleader, either.

I am biased against FCoE because it’s too new to be blithely and broadly recommended for production enterprise use. That’s all. Yes, the standards are standardized and there are products extant. But that’s not enough for me. Next, I’ll talk more about why FCoE is not ready for prime time.

Note: The original post, Will 16 Gb Fibre Channel Derail FCoE? was written as part of an ongoing paid contract with IBM Storage Community, a site funded by IBM. But that post was entirely my own conception and creation with no input from the site editors or IBM and should not be construed to reflect their strategy or opinion.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Why I Am Biased Against FCoE
This post was categorized as Computer History, Enterprise storage, Features, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3
• Storage Changes in VMware ESX 3.5 Update 4
  
• Storage Changes in the VMware vSphere 4 Family

One first step is VMware’s whitepaper, “What’s New in VMware vSphere 5.0 – Storage“.

Licensing and Availability of Features

VMware has once again changed the licensing and pricing model, throwing the Internet into a tizzy:

• “Advanced” has been eliminated, moving up to “Enterprise”
• Pooled vRAM entitlements work across the entire vCenter environment
• New features like Policy-Driven Storage and Storage DRS (along with SIOC) are exclusive to “Enterprise Plus” licenses
• VAAI, PSP multipathing, and Storage vMotion are only found in “Enterprise”
• Thin Provisioning and VADP are available regardless of edition

VMware mucked with vSphere licensing again...

Major New vSphere 5 Storage Features

Storage DRS

Storage DRS is the world’s worst-kept secret, with everyone and his brother talking about it for over a year. Like the existing VM DRS capability, Storage DRS creates resource clusters and automatically moves VMs between them. Storage DRS uses utilization and performance metrics to make the call, and has three modes of operation. It sounds awesome, but it’s an Enterprise Plus-only feature.

Storage APIs – Storage Awareness (VASA)

There’s not much information presently, but a VAAI companion is introduced in vSphere 5: The vSphere Storage APIs for Storage Awareness (VASA) is a communication mechanism for vCenter to detect array capabilities like RAID Level, Thin Provisioning State, Replication State, etc. This will come in handy for all the other features in vSphere 5, especially policy-driven storage!

Policy-Driven Storage

Another new Enterprise Plus feature is Policy-Driven Storage. This allows storage tiers to be defined in vCenter based on SLA, performance, and other metrics which are used during provisioning, cloning, Storage vMotion, and Storage DRS. It leverages VASA for metrics and characterization and supports all arrays in the HCL, regardless of whether they’re NFS, iSCSI, or FC. It includes easy compliance status reporting in vCenter as well.

FCoE Software Initiator

Those of us “in the know” about storage expected VMware to add software FCoE support, so it’s no surprise that they did. This dramatically expands the potential FCoE footprint from just the few CNAs already supported in vSphere 4. It appears to be based on Intel’s OpenFCoE, since it shows up as “Intel Corporation FCoE Adapter” in the GUI!

vSphere Storage Appliance (VSA)

VMware enters the virtual storage appliance (VSA) market with their own offering, the vSphere Storage Appliance (also called VSA). Aimed primarily at the SMB market, it’s actually fairly clever, replicating storage between two or three nodes in a cluster for high availability and using NFS for access rather than iSCSI. And unlike the Celerra UBER that so many techies loved, the VMware VSA is ready for production use!

Existing Storage Features Enhanced in vSphere 5

VMFS 5

VMFS has been improved for scalability and efficiency, but the 2 TB limit on VMDKs remains (except for physical RDM). Only storage geeks like me need to worry about the specifics, but suffice to say that VMFS 5 requires less tuning and worrying and ought to scale and perform better thanks to increased maximums and leveraging the Atomic Test and Set (ATS) technology also used in VAAI. For newly-created volumes, there’s no more block size tuning, and alignment issues are addressed.

Storage APIs – Array Integration (VAAI 2)

See also VMware VAAI Storage Array Support in Plain English

VAAI has been revved, bringing back Thin Provisioning Stun (the AWOL “fourth primitive”) and adding NFS support.

There are now five block primitives for VAAI, depending on if you count thin space reclaim. This is really more of a bug fix than anything, since most folks assumed that the existing thin support already reclaimed deleted VMs and vMotioned VMDKs. I’m more interested in the addition of SCSI UNMAP in addition to WRITE_SAME! There’s also some additional T10 support, though I’m not clear on what it is or where it works.

We’ve also got VAAI for NFS environments now. NAS had sweet thin provisioning support even before block datastores, but the new Reserve Space command adds thick provisioning if that’s what you like. We’ve also got Full File Clone, which is like Full Copy for NFS but doesn’t work with Storage vMotion. And there’s some Extended Stats API to bring in more detail on file status. I also hear there’s an API for Native Snapshot Support, but it’s not widely discussed. Finally, note that NFS plugins come from vendors, not VMware as is the case for block VAAI.

Storage I/O Control

SIOC is enhanced for use in Storage DRS environments, becoming aware of the new datastore clusters. It also gets NFS support, and presumably uses VASA for metrics. But it’s still only available with Enterprise Plus licenses.

iSCSI Initiator GUI

The solid vSphere iSCSI initiator gets a friendly GUI for configuration. I like friendly iSCSI GUIs – just ask Microsoft!

Storage vMotion

The mechanism behind Storage vMotion has changed for a third time in as many releases, this time using “Mirror Mode” to mirror writes to in-progress vMotions. It also now supports migration of vSphere snapshots and Linked Clones. This can be offloaded for VAAI block, but not NFS.

vSphere Replication

New in vCenter Site Recovery Manager 5.0 is software-based replication. Although not technically a vSphere 5 feature, this is a major new storage feature in the VMware world. It allows any-to-any software-based storage replication for disaster recovery.

Stephen’s Stance

Once again, VMware added a ton of new storage enhancements to vSphere. With storage rapidly becoming the limiting factor in scalability and performance of virtual machine environments, this is no surprise. Also not surprising is the fact that major features like Policy-Driven Storage and Storage DRS (along with SIOC) are exclusive to “Enterprise Plus” licenses. I can’t blame VMware for making a buck, but it would be nice if more capabilities were available to the small shops!

I’ll be writing on all these features in detail shortly. Watch this space!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Storage Changes in VMware vSphere 5
This post was categorized as Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

"FCoE vs. iSCSI" isn't a battle or cage match. Your choice depends on many factors, and is more a reflection of convergence than a religious conviction. (photos by Peter Tsai, @SuperTsai)

My presentation at Interop in Las Vegas on May 11, 2011, focused on the protocols that will underpin converged storage networking in the future. My topic, assigned by network computing editor Mike Fratto, was “FCoE vs. iSCSI – Making the Choice.” Although this sounds like a grand competition between the two protocols, my take on the subject is very far from that idea. Rather than a battle, the rise of FCoE and iSCSI is part of the ascendance of convergence of storage and data networking on Ethernet.

“The notion that Fibre Channel is for data centers and iSCSI is for SMB’s and workgroups is outdated. Increases in LAN speeds and the coming of lossless Ethernet position iSCSI as a good fit for the data center. Whether your organization adopts FC or iSCSI depends on many factors like current product set, future application demands, organizational skill-set and budget. In this session we will discuss the different conditions where FC or IsCSI are the right fit, why you should use one and when to kick either to the curb.”

I began my session by pointing out that I am neither a vendor nor protocol cheerleader and don’t really have a horse in the race in terms of a transition to FCoE, iSCSI, InfiniBand, SAS, or any other protocol.  Frankly, I don’t see this as a race, and I don’t care who wins if it is one as long as IT infrastructure progresses to a more flexible state.

Converging on Convergence

The important aspect of any discussion of FCoE is not the protocol itself but the underlying shift away from specialized storage networks converging on Ethernet. ISCSI began this trend almost a decade ago, and the Ethernet roadmap leaves Fibre Channel in the dust.

I see three key elements converging to bring, if you pardon the pun, convergence of data and storage networking:

1. The wholesale adoption of Intel compatible processing architectures
2. A shift toward open systems (Windows and UNIX) for application processing
3. And the widespread adoption of IP as an internetworking protocol.

None of these “trends” is surprising or even questionable: Intel compatible open systems servers using IP dominate modern data centers.

Given this dominant processing architecture, Ethernet is a logical choice as an interconnect. No other network protocol even comes close to the market share, compatibility, and support for Ethernet.

Then we must consider the factors that drive convergence of networking protocols. After all, we have long seen a variety of different protocols in niches such as storage, voice, video, WAN, clustering, and other areas. But virtualization of servers, the need for consolidation to reduce port count and cabling, and a continuing thirst for better performance makes convergence on a single protocol a logical step for these and other areas of IT infrastructure.

If we converge on Ethernet, much will change both inside and outside the data center. Server managers will see greater flexibility and mobility of virtualized servers and blades, as well as increased performance overall. Storage managers will shift from managing esoteric networking protocols to a focus on data management and array performance. But network managers will bear the brunt of the shift, with a wider sphere of influence and new headaches from workloads that do not behave like conventional LAN applications.

The Performance Picture

Turning back to the face of storage networking, we see that one major driver for convergence is pure performance. Although Fibre Channel has an impressive roadmap, with performance doubling again and again, it can’t hold a candle to Ethernet. With historical leaps of an order of magnitude and performance, Ethernet will soon leave Fibre Channel well behind.

When iSCSI first appeared, it was hitched to fairly unimpressive Gigabit Ethernet even as Fibre Channel networks made a transition from 2 to 4 Gb. But iSCSI made a quantum leap in performance this year, transitioning to 10 Gb Ethernet even as Fibre Channel networks moved to 8 Gb. ISCSI FCoE will continue benefiting from Ethernet performance improvements in the coming years, transitioning to 40 Gb and 100 Gb. This will make 16 Gb and 32 Gb Fibre Channel look slow by comparison.

One area that is often overlooked in terms of performance is latency of I/O operations. Although iSCSI over 10 Gb Ethernet can carry 50% more data than 8 Gb Fibre Channel (thanks to more efficient encoding), it also benefits from drastically lower latency. It can handle 50% more packets than 8 Gb Fibre Channel or 10 times as many as Gigabit Ethernet. In other words, in a shared virtual environment, 10 Gb Ethernet allows more systems to get more work done in the same amount of time.

Ethernet Enhancing Data Centers

But performance is only half the story of converged Ethernet. It also supplies server connectivity, reducing the all too frequent situation where configuration and location of servers is dictated by cable availability rather than application need. This will change the face of the data center, encouraging the use of blade servers, virtualization, and flexible (dare I say “cloud”?) infrastructure. It will encourage mobility of machines, especially virtual ones, and demand new networking protocols like OpenFlow.

Ethernet required a serious upgrade to handle this workload, however. Although iSCSI works fine over just about any network, thanks to TCP/IP, FCoE and similar protocols require flow control and guaranteed lossless data delivery. This led to the development of data center bridging protocols (DCB), including priority flow control, bandwidth management, and congestion management. With the first two of these now widely available and the third following shortly, Ethernet is ready to take center stage.

FCoE vs. iSCSI

With discussion of convergence out of the way, we can finally talk about making the choice between iSCSI and FCoE. There are four main reasons to choose one protocol or the other:

1. Data center strategy
2. Performance needs
3. Desire for compatibility
4. Cost concerns

Each of these is a valid reason to pick FCoE or iSCSI in any given situation, and none is a drop–dead decision-maker. There are cases where FCoE will be cheaper than iSCSI and vice versa, for example.

Regardless of the choice between these two protocols, one element remains the same: SCSI. Nearly every enterprise block storage protocol is based on SCSI, and it is one of the seminal technologies that enabled the development of enterprise storage as an industry. Every enterprise block storage protocol, including FCoE, iSCSI, SAS, and plain old Fibre Channel, is really a transport for SCSI.

This makes the selection of protocol less relevant to operating systems and applications, since all will “see” storage the same way. There are major differences between the three SAN protocol choices, in terms of routability, availability of host and initiator hardware and software, maturity, and the availability and selection of management tools.

iSCSI has a more robust support matrix than Fibre Channel over Ethernet, with hardware and software drivers available for nearly every operating system. It is widely supported with mature storage systems available from nearly every vendor. Green field SAN designs with no existing Fibre Channel infrastructure should look no further: iSCSI is a great choice for new storage networks.

The selection of FCoE, on the other hand, is more about evolution from Fibre Channel in enterprise storage networks. There is a threefold path for Fibre Channel architects: They can continue with end-to-end Fibre Channel, and Ethernet and FCoE at the edge, or attempt to build out an end-to-end FCoE SAN. This last option is only recently possible, and is by far the least popular model for Fibre Channel architecture at the present time but will become dominant eventually.

Making the Choice

There are good reasons and bad to pick one protocol over the other, and none rises to the level of religious conviction one might see perusing blogs and tweets on the subject.

FCoE is an evolutionary transition for organizations that already have a large installed base of Fibre Channel equipment, tools, and skills. These environments can incrementally adopt Ethernet as an edge protocol while they continue to leverage the enterprise Fibre Channel storage arrays they already own. Strategically, FCoE makes perfect sense for users of “blocks” or “stacks” from vendors like Cisco, EMC, HP, and NetApp. But FCoE remains somewhat unproven, and some supporting protocols, like congestion notification and so-called Ethernet fabric technology, are immature at best when it comes to interoperability.

One common refrain when comparing FCoE and iSCSI is the efficiency of the protocols. Packaging SCSI in TCP and IP can’t be efficient, can it? But an analysis of the protocols reveals that absolute bit efficiency is very similar between Fibre Channel, FCoE, and iSCSI. Tests by Dell’s Tech Center and others show that iSCSI is fairly efficient in terms of data throughput and CPU utilization as well.

Stephen’s Stance

iSCSI is an excellent choice in situations where Fibre Channel investment is nonexistent or badly in need of wholesale upgrade, while FCoE is likely to take over in high-end enterprise shops

iSCSI is an excellent choice in situations where Fibre Channel investment is nonexistent or badly in need of wholesale upgrade. It will continue to grow based on ease of use, low cost and high performance, and widespread support, in the transition to 10 Gb Ethernet could not be simpler. FCoE, on the other hand, is likely to take over in high-end enterprise shops. It is relentlessly promoted by major vendors, and it seems that they will force the upgrade eventually. But some areas are still not ready for prime time, and buyers should beware of grandiose promises at this point.

In counterpoint, one may ask the question of why we chose Ethernet at all. It required much work, and unnatural acts like DCB, to prepare Ethernet to become the dominant protocol for convergence. Why not use InfiniBand instead, since it already works, has widespread implementation, excellent performance and scalability, as well as interoperability and hardware availability? Price is one concern, but the major factor is far more basic: No one doubts that Ethernet will eventually ascend and overcome its obstacles. It is a foregone conclusion.

In retrospect, many alternative protocols might have been better suited to convergence, including ATM and even Token Ring. Although the topic of Fibre Channel over Token Ring (FCoTR) brings a smile to the faces of network and storage nerds everywhere, we all expect that fiber Channel over Ethernet (FCoE) and iSCSI will rule the day.

Photos by Peter Tsai

Watch the presentation from Interop (apologies for the poor camera angle and sound!)

© sfoskett for Stephen Foskett, Pack Rat, 2011. | FCoE vs. iSCSI – Making the Choice
This post was categorized as Computer History, Enterprise storage, Everything, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Servers, storage, and networks may be interconnected, but most large IT organizations keep the administrative teams from mixing. But the next-generation virtual data center might change that!

Lots of my IT infrastructure management clients are talking about how the advent of Ethernet/IP and virtualization is changing the roles of storage, server, and network administrators. The evolution of the storage role in particular in enterprise IT organizations has been a topic of particular interest to me for a while: I definitely remember thinking about this as iSCSI and Cisco came on the scene a few years back, but the question of integration of storage, server, network, and application management areas is as old as IT.

Back in 2005, I wrote a column in Storage magazine, asking Who Watches the SAN? Although there were (and still are) reasons one might consider handing SAN management over to the network team, such as the use of iSCSI, Cisco FC, or (soon) FCoE, I concluded that many network teams just aren’t ready to take on SAN management. Storage is different – extremely sensitive to latency and outages and burdened with interconnect concepts that are similar, but not identical, to their network cousins. So most people are better off leaving SAN management in the hands of storage people, regardless of whether their SAN is made up of FC or iSCSI.

The Virtual Data Center

Things are changing, however. The modern data center is evolving to virtualize all three major IT infrastructure components: Servers, storage, and networks. Over the last year or so, we’ve seen the first fully-virtual infrastructure built, with all three areas combined in a single box, soup-to-nuts. Consider a VMware ESX server with virtual servers talking over virtual networks to a virtual storage array from FalconStor or LeftHand – it’s an entire data center in a box!

Server admins are normally tasked with everything “in the box”, including VMFS and virtual network connectivity, just as they always had to manage volume managers and path management software back in the “physical server” world. I doubt this will change. So we could see not just storage but network folks excluded entirely from the next-generation virtual datacenter!

This doesn’t have to be how things end up, though. I strongly believe that network and storage pros have critical insight into their areas, and cutting them out would be a tragic loss akin to what happened when open systems folks decided not to pay attention to the lessons of the mainframe generation. We would effectively repeat a decade of experience and learning that could, with minor modifications, be brought right into the modern world.

There are three things to do:

1. Re-combine the stovepipe IT infrastructure organizations (server, storage, network) into a single management organization with specialists in these areas and others like virtualization and cloud computing.
2. Spend the time and money to cross-train everyone to re-apply their experience and skills in this new world. Storage folks, for example, must know a good bit about server virtualization or their skills will have much less value!
3. Bring the mainframe, security, and records management folks to the party, too! They all have essential insights, and a failure to give them a seat at the table would be a critical loss.

Database and Applications

Then there are the DBAs and IT application folks. These teams have always been held at arms-length in the open systems world, and much closer on the mainframe side (especially the database people!) I think there has been a feeling that there were enough interests at the IT infrastructure planning meetings already without mixing them in as well! The current response is a service-based approach, where IT infrastructure adds an analyst role to develop SLAs and standard service offerings and act as a liaison between ITI and IT Apps.

This is probably enough for a conventional system, but there are changes here as well. Virtual appliances can step right into the apps arena, and the database/storage hybrid devices from Oracle/HP, Netezza, and the rest tromp right through the DBA world. Then there are the “webby dubby” (Web 2.0) storage services/devices like Amazon S3, Nirvanix, and EMC Atmos to consider.

Stephen’s Stance

I think we’re seeing another serious change to the status quo: Right after the data center is virtualized and IT infrastructure is recombined, applications themselves will fundamentally transform, demanding a merger of the current IT infrastructure and IT applications groups. This could all come within five years, or it could be delayed or diverted by organizational infighting and intransigence. It will be very interesting to see how it plays out!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Changes in Technology Drive Changes in IT Organizations and Roles
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Tech Field Day 5 links
  • Official stuff:
    • Tech Field Day 5: The Links
    • Tech Field Day 5: The Videos
  • Rob Novak shared his Thoughts from Tech Field Day 5
  • A great post by Greg Stuart: Symantec NetBackup 7 for VMware vSphere 4
  • I loved Tom’s Things I Learned From Tech Field Day, Part 1
• My writing
  • I’ve wanted to write this for quite a while: ATA over Ethernet protocol overview for data storage managers
  • I talked to Intel this week and wrote it up: Where Is Intel’s FCoE Solution?
  • Another big area for my virtual storage seminar preparation: VMware VAAI Storage Array Support in Plain English
  • And finally: Aerobie AeroPress Review: The Hacker Coffee Maker
• Other great stuff I’ve been reading
  • Barry Burke responded to my VAAI post with some competitive digs: reality check vsp vaai support
  • From Robin Harris is a DEC obituary: Ken Olsen, 1926-2011
  • A clever piece by Tom Petrocelli: Public Clouds : An Unregulated Utility
  • vKernel has a good point to make: Dear CIO’s, Please Send Paul Otellini a Thank You Note
  • Chris Evans covers the arrays we saw at TFD: Data Robotics Releases Business-class Arrays
  • Ivan Pepelnjak (inspired by Greg Ferro): Open FCoE – Software implementation of the camel jetpack
  • I loved this infographic: The Most Common and Dangerous Passwords
  • And this one: A Decade of Storage
  • Another great piece by Chris Evans: Managing iSCSI on Windows 2008 Core with Hyper-V

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, February 11, 2011
This post was categorized as Computer History, Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Things I wrote
  • VMware ESX SATA and PATA Compatibility Cheat Sheet
  • VMware ESX FCoE CNA Compatibility in Plain English
  • Samsung SP-H03 Pico Projector Review
• Other great posts
  • Wikibon are stepping up their game lately with some great posts like this: HP and Intel Help Open the FCoE Market
  • Mini Microsoft is always worth reading: Microsoft FY11Q2 Results
  • Greg Ferro talks about The Importance of Provider Independent IPv6 Addressing
  • Scott Lowe does his own “back from the pile” and he has some great links in Technology Short Take #10
  • I’m so excited to have Chris Wells join us from Tokyo for TFD! vSamurai – Virtualization, Cloud Computing, Japan, 仮想化, クラウドコンピューティング, 日本 – EN – Gonna Have A Field Day!
  • Sean Clark will also come to TFD: Tech Field Day 5 – San Jose – SeanClark.us
  • It was way late, but the Freedom to Tinker Predictions for 2011 is really worth a read
  • I totally agree: 3D is a massive headache! Why 3D Doesn’t Work and Never Will
  • Jason Boche did some great work this week: Jumbo Frames Comparison Testing with IP Storage and vMotion
  • Derek Schauland has a great idea: Synchronization from Drobo to the cloud
  • Xsigo is taking on King Kong: The Inside Story: What Cisco says About Xsigo
  • Erin Collopy is back in her new blog with the importance of knowing your audience

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, January 28, 2011
This post was categorized as Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Converged Networking Adapters like this QLogic 8242 are all the rage, but which are supported in VMware ESX and which have the broadest coverage of DCB features?

VMware has one awesome hardware compatibility list. It’s not the breadth of it (in fact, ESX’s range of hardware support is astonishingly small) but the thorough, public way in which VMware shares this information. It’s really worth a look, and I regularly visit the list to see what’s changed.

But the thoroughness and detail of the VMware HCL can be daunting. It’s fairly easy to search for a specific piece of hardware, but it’s difficult to tell what’s supported in a general sense. This causes trouble for people who are shopping for hardware, since it’s hard to know what’s supported without an exhaustive search.

As part of the preparation for my Storage for Virtual Servers seminar series this year, I decided to do some research. I’ve boiled down certain key hardware categories into a general plain-english list of what’s in and out of the ESX HCL. In the spirit of openness, I’m presenting this data here for all to see, and I welcome corrections and updates. Indeed, I’ll try to keep this page up to date as new hardware is added!

Minimum Requirements

No one should run unsupported hardware in a production environment. So the very minimum requirement for every buyer should be a list in the VMware ESX HCL with the proper firmware, driver, and software.

FCoE also requires a flow control mechanism, preferably 802.1Qbb, and the DCBX protocol to enable communication between ports. I strongly suggest running 802.1Qaz bandwidth management as well, since it will allow you to get more performance from your CNAs than plain PFC.

Congestion management (802.1Qau) isn’t required, and I’m not sure about the state of affairs for interoperability. So that’s a “nice to have” at this point, though it will certainly become more desirable in the future.

You might also look for a CNA that supports other protocols over DCB, since iSCSI is an attractive alternative to FCoE in the data center, as is NFS. But that’s just my opinion!

FCoE CNAs for VMware ESX

Let’s kick things off with FCoE CNAs. I’m not a huge fan of FCoE in general, but I do see it as an increasingly-viable protocol for large-scale enterprise virtualization products.

It’s critical to have supported hardware, and I suggest researching specific models on the HCL before buying. But here’s the general state of affairs with regard to FCoE CNAs:

I was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such a spec!

It’s also puzzling to see that QLogic does not list 802.1Qaz Bandwidth Management (ETS) as supported for the (admittedly older) QLE8042 card. Perhaps that’s a error? But then again, Emulex doesn’t list 802.1Qbb Priority Flow Control (PFC) for the LP21000, and although FCoE doesn’t require Qbb, it’s a very good idea…

Update: Emulex tells me the LP21000 does indeed support PFC (Qbb).

Stephen’s Stance

If you’re shopping for networking gear and want to move to FCoE eventually, I hope this list will help you get started. Most OEMs supply the latest Emulex, QLogic, and Brocade CNAs with their own part number, and the long ESX HCL is testament to this fact. But, generally speaking, if you’re using a latest-generation CNA like the QLogic 8200 or Emulex OneConnect series, you ought to be fine. The Brocade CNAs look good, too, but are a little harder to find.

Emulex tells me they will add QCN support once it’s ratified, and I look forward to learning more about the Brocade and QLogic implementations. I’d also love some feedback on how well these features work, and if they’re all they’re cracked up to be. Is one vendor’s PFC or ETS as good as another’s? And does DCBX really work in practice? I believe most folks aren’t mixing HBAs widely and are instead sticking to one manufacturer. That’s probably a good idea out here on the cutting edge!

I was surprised that no Intel or Broadcom cards are listed as supporting FCoE in the HCL. The Intel X520 is impressive, and I’d taken their “open FCoE” claims to include VMware. But I guess they’re not there yet, and this seems to be a glaring omission for a major player like Intel! Similarly, Broadcom is only talking about iSCSI with VMware. Where is their CNA with VMware support?

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware ESX FCoE CNA Compatibility in Plain English
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Once something is in place, it's hard to get it to move again

In Philosophiæ Naturalis, Sir Isaac Newton defined inertia as follows:

The vis insita, or innate force of matter, is a power of resisting by which every body, as much as in it lies, endeavours to preserve its present state, whether it be of rest or of moving uniformly forward in a straight line.

Although Newton was referring to physical objects, the power of inertia affects companies, markets, and relationships in the same manner.  Humans are creatures of habit, and change is challenging.  When faced with a choice of continuing along the same road or branching off in a new direction, most will choose familiarity.

Inertia in IT Architecture

Consider the impact of inertia on IT architecture: once the solution is in place, it tends to remain there for a very long time.  This rule applies to practices, architectures, solutions, and hardware and software.  It explains the continued presence of Token Ring, MS-DOS, Mac OS 9, and Palm organizers in so many companies.  It also explains the curious devotion IT pros field toward solutions that are backward compatible: Ethernet, Windows, Intel x86 architecture, and so on.

Once, while visiting the data center of a midsize financial institution, I spotted a stack of old IBM PC desktop computers in the corner.  The company had purchased a company, which itself had purchased a bank many years ago.  The loans from that long ago and far off institution were still serviced by this archaic hardware and software.  The company’s IT staff had squirreled away half a dozen replacement computers so they could migrate the application to new old hardware in the event of a failure.  If this isn’t inertia, I don’t know what it is.

Overcoming Inertia

An external force is required to overcome inertia, and one must desire to initiate a change.  New products and solutions must not merely be attractive, it must also be compelling enough to overcome this inertia.  In my experience, there are three reasons that companies change direction when it comes to IT architecture:

1. A noticeable irrefutable return on investment (ROI)
2. A tangible and necessary performance benefit
3. A unique and desirable function

Many new technologies show promise in all three areas, including 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), server virtualization, and data deduplication.  But these potential benefits are not necessarily compelling in all IT environments.

A company with a substantial investment in Fibre Channel SAN hardware may find that upgrading to 8 Gb Fibre Channel is more compelling than a switch to converged networking and FCoE.  Many companies have found it hard to justify the additional cost of data compression or deduplication technology when compared with the decreasing cost of capacity or the benefits of improved utilization through better storage management.  The growth of server virtualization has been steady, but the hold-outs indicate that many companies find it hard to justify the technology.

Stephen’s Stance

Contrary to our nerd dreams, mere technical superiority does not guarantee the success of a new product or solution.  It must be better, faster, and cheaper to achieve widespread success.  In short, it must demonstrate a compelling case, or inertia will set in and derail its progress.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Three Requirements To Overcome Inertia
This post was categorized as Computer History, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.