"FCoE vs. iSCSI" isn't a battle or cage match. Your choice depends on many factors, and is more a reflection of convergence than a religious conviction. (photos by Peter Tsai, @SuperTsai)

My presentation at Interop in Las Vegas on May 11, 2011, focused on the protocols that will underpin converged storage networking in the future. My topic, assigned by network computing editor Mike Fratto, was “FCoE vs. iSCSI – Making the Choice.” Although this sounds like a grand competition between the two protocols, my take on the subject is very far from that idea. Rather than a battle, the rise of FCoE and iSCSI is part of the ascendance of convergence of storage and data networking on Ethernet.

“The notion that Fibre Channel is for data centers and iSCSI is for SMB’s and workgroups is outdated. Increases in LAN speeds and the coming of lossless Ethernet position iSCSI as a good fit for the data center. Whether your organization adopts FC or iSCSI depends on many factors like current product set, future application demands, organizational skill-set and budget. In this session we will discuss the different conditions where FC or IsCSI are the right fit, why you should use one and when to kick either to the curb.”

I began my session by pointing out that I am neither a vendor nor protocol cheerleader and don’t really have a horse in the race in terms of a transition to FCoE, iSCSI, InfiniBand, SAS, or any other protocol.  Frankly, I don’t see this as a race, and I don’t care who wins if it is one as long as IT infrastructure progresses to a more flexible state.

Converging on Convergence

The important aspect of any discussion of FCoE is not the protocol itself but the underlying shift away from specialized storage networks converging on Ethernet. ISCSI began this trend almost a decade ago, and the Ethernet roadmap leaves Fibre Channel in the dust.

I see three key elements converging to bring, if you pardon the pun, convergence of data and storage networking:

1. The wholesale adoption of Intel compatible processing architectures
2. A shift toward open systems (Windows and UNIX) for application processing
3. And the widespread adoption of IP as an internetworking protocol.

None of these “trends” is surprising or even questionable: Intel compatible open systems servers using IP dominate modern data centers.

Given this dominant processing architecture, Ethernet is a logical choice as an interconnect. No other network protocol even comes close to the market share, compatibility, and support for Ethernet.

Then we must consider the factors that drive convergence of networking protocols. After all, we have long seen a variety of different protocols in niches such as storage, voice, video, WAN, clustering, and other areas. But virtualization of servers, the need for consolidation to reduce port count and cabling, and a continuing thirst for better performance makes convergence on a single protocol a logical step for these and other areas of IT infrastructure.

If we converge on Ethernet, much will change both inside and outside the data center. Server managers will see greater flexibility and mobility of virtualized servers and blades, as well as increased performance overall. Storage managers will shift from managing esoteric networking protocols to a focus on data management and array performance. But network managers will bear the brunt of the shift, with a wider sphere of influence and new headaches from workloads that do not behave like conventional LAN applications.

The Performance Picture

Turning back to the face of storage networking, we see that one major driver for convergence is pure performance. Although Fibre Channel has an impressive roadmap, with performance doubling again and again, it can’t hold a candle to Ethernet. With historical leaps of an order of magnitude and performance, Ethernet will soon leave Fibre Channel well behind.

When iSCSI first appeared, it was hitched to fairly unimpressive Gigabit Ethernet even as Fibre Channel networks made a transition from 2 to 4 Gb. But iSCSI made a quantum leap in performance this year, transitioning to 10 Gb Ethernet even as Fibre Channel networks moved to 8 Gb. ISCSI FCoE will continue benefiting from Ethernet performance improvements in the coming years, transitioning to 40 Gb and 100 Gb. This will make 16 Gb and 32 Gb Fibre Channel look slow by comparison.

One area that is often overlooked in terms of performance is latency of I/O operations. Although iSCSI over 10 Gb Ethernet can carry 50% more data than 8 Gb Fibre Channel (thanks to more efficient encoding), it also benefits from drastically lower latency. It can handle 50% more packets than 8 Gb Fibre Channel or 10 times as many as Gigabit Ethernet. In other words, in a shared virtual environment, 10 Gb Ethernet allows more systems to get more work done in the same amount of time.

Ethernet Enhancing Data Centers

But performance is only half the story of converged Ethernet. It also supplies server connectivity, reducing the all too frequent situation where configuration and location of servers is dictated by cable availability rather than application need. This will change the face of the data center, encouraging the use of blade servers, virtualization, and flexible (dare I say “cloud”?) infrastructure. It will encourage mobility of machines, especially virtual ones, and demand new networking protocols like OpenFlow.

Ethernet required a serious upgrade to handle this workload, however. Although iSCSI works fine over just about any network, thanks to TCP/IP, FCoE and similar protocols require flow control and guaranteed lossless data delivery. This led to the development of data center bridging protocols (DCB), including priority flow control, bandwidth management, and congestion management. With the first two of these now widely available and the third following shortly, Ethernet is ready to take center stage.

FCoE vs. iSCSI

With discussion of convergence out of the way, we can finally talk about making the choice between iSCSI and FCoE. There are four main reasons to choose one protocol or the other:

1. Data center strategy
2. Performance needs
3. Desire for compatibility
4. Cost concerns

Each of these is a valid reason to pick FCoE or iSCSI in any given situation, and none is a drop–dead decision-maker. There are cases where FCoE will be cheaper than iSCSI and vice versa, for example.

Regardless of the choice between these two protocols, one element remains the same: SCSI. Nearly every enterprise block storage protocol is based on SCSI, and it is one of the seminal technologies that enabled the development of enterprise storage as an industry. Every enterprise block storage protocol, including FCoE, iSCSI, SAS, and plain old Fibre Channel, is really a transport for SCSI.

This makes the selection of protocol less relevant to operating systems and applications, since all will “see” storage the same way. There are major differences between the three SAN protocol choices, in terms of routability, availability of host and initiator hardware and software, maturity, and the availability and selection of management tools.

iSCSI has a more robust support matrix than Fibre Channel over Ethernet, with hardware and software drivers available for nearly every operating system. It is widely supported with mature storage systems available from nearly every vendor. Green field SAN designs with no existing Fibre Channel infrastructure should look no further: iSCSI is a great choice for new storage networks.

The selection of FCoE, on the other hand, is more about evolution from Fibre Channel in enterprise storage networks. There is a threefold path for Fibre Channel architects: They can continue with end-to-end Fibre Channel, and Ethernet and FCoE at the edge, or attempt to build out an end-to-end FCoE SAN. This last option is only recently possible, and is by far the least popular model for Fibre Channel architecture at the present time but will become dominant eventually.

Making the Choice

There are good reasons and bad to pick one protocol over the other, and none rises to the level of religious conviction one might see perusing blogs and tweets on the subject.

FCoE is an evolutionary transition for organizations that already have a large installed base of Fibre Channel equipment, tools, and skills. These environments can incrementally adopt Ethernet as an edge protocol while they continue to leverage the enterprise Fibre Channel storage arrays they already own. Strategically, FCoE makes perfect sense for users of “blocks” or “stacks” from vendors like Cisco, EMC, HP, and NetApp. But FCoE remains somewhat unproven, and some supporting protocols, like congestion notification and so-called Ethernet fabric technology, are immature at best when it comes to interoperability.

One common refrain when comparing FCoE and iSCSI is the efficiency of the protocols. Packaging SCSI in TCP and IP can’t be efficient, can it? But an analysis of the protocols reveals that absolute bit efficiency is very similar between Fibre Channel, FCoE, and iSCSI. Tests by Dell’s Tech Center and others show that iSCSI is fairly efficient in terms of data throughput and CPU utilization as well.

Stephen’s Stance

iSCSI is an excellent choice in situations where Fibre Channel investment is nonexistent or badly in need of wholesale upgrade, while FCoE is likely to take over in high-end enterprise shops

iSCSI is an excellent choice in situations where Fibre Channel investment is nonexistent or badly in need of wholesale upgrade. It will continue to grow based on ease of use, low cost and high performance, and widespread support, in the transition to 10 Gb Ethernet could not be simpler. FCoE, on the other hand, is likely to take over in high-end enterprise shops. It is relentlessly promoted by major vendors, and it seems that they will force the upgrade eventually. But some areas are still not ready for prime time, and buyers should beware of grandiose promises at this point.

In counterpoint, one may ask the question of why we chose Ethernet at all. It required much work, and unnatural acts like DCB, to prepare Ethernet to become the dominant protocol for convergence. Why not use InfiniBand instead, since it already works, has widespread implementation, excellent performance and scalability, as well as interoperability and hardware availability? Price is one concern, but the major factor is far more basic: No one doubts that Ethernet will eventually ascend and overcome its obstacles. It is a foregone conclusion.

In retrospect, many alternative protocols might have been better suited to convergence, including ATM and even Token Ring. Although the topic of Fibre Channel over Token Ring (FCoTR) brings a smile to the faces of network and storage nerds everywhere, we all expect that fiber Channel over Ethernet (FCoE) and iSCSI will rule the day.

Photos by Peter Tsai

Watch the presentation from Interop (apologies for the poor camera angle and sound!)

© sfoskett for Stephen Foskett, Pack Rat, 2011. | FCoE vs. iSCSI – Making the Choice
This post was categorized as Computer History, Enterprise storage, Everything, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Servers, storage, and networks may be interconnected, but most large IT organizations keep the administrative teams from mixing. But the next-generation virtual data center might change that!

Lots of my IT infrastructure management clients are talking about how the advent of Ethernet/IP and virtualization is changing the roles of storage, server, and network administrators. The evolution of the storage role in particular in enterprise IT organizations has been a topic of particular interest to me for a while: I definitely remember thinking about this as iSCSI and Cisco came on the scene a few years back, but the question of integration of storage, server, network, and application management areas is as old as IT.

Back in 2005, I wrote a column in Storage magazine, asking Who Watches the SAN? Although there were (and still are) reasons one might consider handing SAN management over to the network team, such as the use of iSCSI, Cisco FC, or (soon) FCoE, I concluded that many network teams just aren’t ready to take on SAN management. Storage is different – extremely sensitive to latency and outages and burdened with interconnect concepts that are similar, but not identical, to their network cousins. So most people are better off leaving SAN management in the hands of storage people, regardless of whether their SAN is made up of FC or iSCSI.

The Virtual Data Center

Things are changing, however. The modern data center is evolving to virtualize all three major IT infrastructure components: Servers, storage, and networks. Over the last year or so, we’ve seen the first fully-virtual infrastructure built, with all three areas combined in a single box, soup-to-nuts. Consider a VMware ESX server with virtual servers talking over virtual networks to a virtual storage array from FalconStor or LeftHand – it’s an entire data center in a box!

Server admins are normally tasked with everything “in the box”, including VMFS and virtual network connectivity, just as they always had to manage volume managers and path management software back in the “physical server” world. I doubt this will change. So we could see not just storage but network folks excluded entirely from the next-generation virtual datacenter!

This doesn’t have to be how things end up, though. I strongly believe that network and storage pros have critical insight into their areas, and cutting them out would be a tragic loss akin to what happened when open systems folks decided not to pay attention to the lessons of the mainframe generation. We would effectively repeat a decade of experience and learning that could, with minor modifications, be brought right into the modern world.

There are three things to do:

1. Re-combine the stovepipe IT infrastructure organizations (server, storage, network) into a single management organization with specialists in these areas and others like virtualization and cloud computing.
2. Spend the time and money to cross-train everyone to re-apply their experience and skills in this new world. Storage folks, for example, must know a good bit about server virtualization or their skills will have much less value!
3. Bring the mainframe, security, and records management folks to the party, too! They all have essential insights, and a failure to give them a seat at the table would be a critical loss.

Database and Applications

Then there are the DBAs and IT application folks. These teams have always been held at arms-length in the open systems world, and much closer on the mainframe side (especially the database people!) I think there has been a feeling that there were enough interests at the IT infrastructure planning meetings already without mixing them in as well! The current response is a service-based approach, where IT infrastructure adds an analyst role to develop SLAs and standard service offerings and act as a liaison between ITI and IT Apps.

This is probably enough for a conventional system, but there are changes here as well. Virtual appliances can step right into the apps arena, and the database/storage hybrid devices from Oracle/HP, Netezza, and the rest tromp right through the DBA world. Then there are the “webby dubby” (Web 2.0) storage services/devices like Amazon S3, Nirvanix, and EMC Atmos to consider.

Stephen’s Stance

I think we’re seeing another serious change to the status quo: Right after the data center is virtualized and IT infrastructure is recombined, applications themselves will fundamentally transform, demanding a merger of the current IT infrastructure and IT applications groups. This could all come within five years, or it could be delayed or diverted by organizational infighting and intransigence. It will be very interesting to see how it plays out!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Changes in Technology Drive Changes in IT Organizations and Roles
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing
  • Is NFS A Viable Protocol For Converged Networking?
  • EMC Taunts NetApp: Counting Coup or Poor Sportsmanship?
  • When Marketing Becomes Pointless
  • NetEx Joins the Roster of Tech Field Day Presenters
• Other great stuff
  • Bob Plankers takes his turn at bat for FCoTR with a Fibre Channel over Token Ring Presentation @ WI VMUG
  • This old post by Scott Drummonds about VMDirectPath is worth revisiting!
  • I couldn’t have said it any better, Louis: The Best Thing Apple Does Is Ignore Everyone Else
  • Chad Sakac and the gang were great hosts last week. Here’s more about their cool new tiny array: VNXe = incredible things.. in tiny, transformer-like packages!
  • Jon Langemak’s post on SAN Terminology is worth reading for newbies to the storage scene
  • Finally, Further clarification from Google about h.264

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, January 21, 2011
This post was categorized as Apple, Enterprise storage, Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• Things I wrote
  • I kicked off my new Network Computing column with this one: Ethernet Has A Goldilocks Problem (and here’s a response: Network Convergence: More than Just 10 GbE)
  • TSIA! Is PCIe SSD right for you? Deploying PCI Express SSD devices
  • I haven’t written for Silicon Angle in a while, but thought this was best there: From Scale-Out to Big Data to the Cloud
  • Will You Be At The ExecEvent? I will, and it’s just two weeks away!
  • Sometimes you have an idea: The Three Requirements To Overcome Inertia
  • I’m Open-Sourcing My 2011 Storage For Virtual Environments Seminar! So now I Need Numbers!
  • Apple Breaks ICS Calendar Auto-Subscription In iOS 4.2
• Other great links
  • Derek Schauland asks the eternal datacenter question: Where can I plug in?
  • Brandon Carroll documents that Solarwinds Install is Easy as Pie!
  • I love Greg Ferro’s Network ZEN, but this week’s was especially awesome: Network ZEN: A Switch
  • Nigel Poulton puts together some amazing storage content. Case in pout: VMAX vs VSP
  • Seriously, Google? Google stripping support for H.264 video out of Chrome
  • The most-scary WordPress article I’ve read in a long time: Why You Should Never Search For Free WordPress Themes in Google or Anywhere Else
  • Howard Marks takes on a topic we discussed last time I was in New York: More on Advanced Erasure Codes
  • Top Five Storage Technologies for 2011 <- Good guess!
  • The only insanely great thing to come out of CES: Breakthrough device of CES 2011: Motorola Atrix = Phone + PC
  • I love words 20 Awesomely Untranslatable Words from Around the World

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, January 14, 2011
This post was categorized as Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

2011 will look pretty much like 2010 apart from the differences...

Prognostication is a perilous business, but pundits are drawn to the topic in the month of December. The fact that most predictions fall on their faces demonstrates the intoxicating mix of hope, dreams, and irrationality that mark both geniuses and fools. I am neither, so I like to make predictions after the fact! But this year I’ve been asked to look to the future, so I’ll stick with the safe road and pick current trends rather than guessing what I hope will come.

Five Trends For Everyone

1 – Ubiquitous Connectivity

I named mobility as the mega-trend of the last decade, noting that it’s hard to spot a trend from the middle and harder still from the start. But I feel vindicated on that 2009 call, and will take it one further: 2011 will see ubiquitous connectivity become mainstream. With “MiFi” entering the vernacular, a proliferation of “4G” networks and integrated wireless data, and free WiFi having already become passe, I’d say “online everywhere” is here. Although incredibly challenging from both a technical and business perspective, I expect everyone and everything to be online-capable.

2 – The iPad and Foes

Steve Jobs’ bizarre claim that the overgrown iPhone tablet was the most important thing he ever did is starting to look prescient. A new generation is coming of age without windowing GUIs, mice, and keyboards thanks to gaming consoles, smartphones, iPods, and (finally) tablets. 2011 will finally see serious Android, WebOS, and Windows tablets, but the iPad (and forthcoming iPad 2) are the platform to beat, and Apple is firing on all cylinders. This war will be all-consuming next year.

3 – Wave “Hi” to Kinect!

Microsoft has a serious hit on their hands with the Kinect add-on to the Xbox 360. The gaming system is the best thing to come out of Redmond in a long time, and it continues the “no controller” concept of the iPad, finally kicking the Wii to the curb. The massive success of the Xbox will lead Microsoft shareholders to abandon their calls for Ballmer’s head, instead asking for a spin-out or IPO of the gaming division. Expect PCs to include Kinect-like features in the coming years as well.

4 – Facebook Is the Internet

Now boasting a quarter of all web pageviews. Facebook is looking increasingly like AOL for the rest of us. 2011 will see Facebook’s gravity pull in content from everywhere, and its satellites sprout all over the Internet. It will become the single sign-on, the central “like”, the address book, and the meeting place. But fear not, Facebook-phobes: Nothing is permanent, and this too shall pass.

5 – The Internet Changes and No One Notices

The IP address space is exhausted, BGP and DNS security lapses rise, and net neutrality falls by the wayside but Farmville still works so no one cares. The Internet is changing, and controversies over key components are coming coming to a head. I imagine the network engineers will be busy keeping ahead of catastrophe, but they’ll manage somehow. I’m not sure if IPv6 will finally take off or if the carrier gambit will succeed, but I’m confident we’ll still have an Internet at the heart of the technology world!

Five Trends For the Datacenter

6 – Clouds Gather Quietly

“Cloud” was the buzzword of the last two years, but now it’s getting down to work. Traditional IT staff still won’t see much of it outside of blogs and conferences, but non-traditional systems are all heading that way. Look for major uptake of cloud platforms and services from the home to enterprise applications and everywhere in between. Ironically, the “c-word” itself will soon be dropped from these successful services just as it gains acceptance in IT shops.

7 – Virtual Everything

The impact of server virtualization hasn’t been as great as supporters claim, but widespread acceptance of hypervisor-centric data centers is here. There’s really no reason not to deploy every datacenter server as a virtual machine and lots of resulting benefits. Expect to see mission-critical apps finally move to VMware ESX and Microsoft Hyper-V in 2011. And expect to see the resulting infrastructure called “cloud”!

8 – Farewell, Fast Hard Drives

Storage I/O performance is now the realm of solid state, not spinning disk. SSDs have reached the level of performance, capacity, availability, and sophistication that we no longer need 15k rpm enterprise hard disk drives. You’ll use SSD if you want IOPS, but you’ll still need spinning platters for capacity and maximum throughput for a long while. Don’t expect hard disk drives to disappear, but the fastest will exit at the end of the year.

9 – Not-So-Converged I/O (Yet)

DCB ain’t ready, folks. Neither is FCoE. Although Ethernet will eventually sideline InfiniBand and Fibre Channel, that’s not a 2011 topic. I expect to hear a lot of noise about converged network and storage I/O, including high-profile customer adoption stories, but we’re still a few years short of actual impact and serious market share movement. Practical application starts in 2011, though, and it’ll get major coverage and big-money action in the vendor space.

10 – RAID is (Finally) Dead!

There won’t be much ink spilled in memoriam outside storage blogs like this one, but conventional mirroring and parity has finally met its maker. Today’s hard disk drives are too big to rebuild singly, and alternatives like wide striping, erasure coding, and touch-me-not disk packs are taking over.

And One More For Me

11 – The Internet is Shiva

Every business will be permanently changed as Internet-enabled platforms destroy profitable monopolies and build new opportunities. Google conquered advertising and destroyed traditional publishing but enabled a flowering of democratic dialog. PayPal and Square will do the same to banking in 2011, but their own come-uppance might come sooner than they like. No matter your business, someone has their sights set on you and the Internet is their tool. The trick is to keep dancing, keep innovating, and love the paradox.
The text to appear after expiration date.
Image Credit: Eleven by Wetsun

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Eleven Tech Trends To Watch In 2011
This post was categorized as Apple, Computer History, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Today, Brocade announced that it has completed its acquisition of Foundry Networks.  This is just the latest move in the strategic game to control the next generation of Ethernet, and possibly all local connectivity, including storage. Although 1 Gb Ethernet, 4 and 8 Gb Fibre Channel, and InfiniBand are all still going strong, the attention of the industry, the pundits, and the prognosticators (myself included) is firmly fixed on enhanced 10 Gb Ethernet. So Brocade’s move seems especially relevant to the core question of which companies will thrive and which will fail in a 10 Gb world.

Cisco is, as always, the big player everyone is gunning for. They created a subsidiary, Nuova, to develop a next-generation architecture, then absorbed the company in April. The result of Nuova’s work is Cisco’s Nexus line of data center switching products as well as much contribution to enhancements to Ethernet, known as Data Center Bridging (DCB) to the IEEE, Data Center Ethernet (DCE) to Cisco, and Converged Enhanced Ethernet (CEE) to others. The whole Nuova episode worked out quite well for Cisco, and they are well-positioned in the next-generation Ethernet game.

Although the fight to control the next generation of datacenter I/O can sometimes seem like a two-way battle between Cisco and Brocade, there are many players involved:

• Woven Systems launched its second-generation product in October. Aimed squarely at the largest data centers, the Woven line sells at bargain prices and expands incrementally, maintaining performance, according to the company.
• Force10 has also found success selling a complete line of 10 Gb Ethernet products, from the core data center to the edge.
• HP’s ProCurve line is mostly focused on smaller businesses, but the number-two supplier of networking hardware shouldn’t be discounted in this horse race!
• Arista Networks is another dark horse. With ex-Cisco “Data Center 3.0 commander” Jayshree Ullal and ex-Sun Andy Bechtolsheim, Arista has so far played its 10 Gb song to a cloud computing refrain. But the company reportedly has strong technology, and could be a contender in the converged I/O space as well.

This playing field is wide open. I expect Brocade and Cisco to be major players over the next few years, and that HP will acquire one of the other players at some point to add to their ProCurve line. But no matter who wins, the dominance of 10 Gb Ethernet is a safe bet.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Who Will Capture the 10 Gigabit Ethernet Crown?
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.