Virtualization decouples physical infrastructure from logical resources

Virtualization of IT systems decouples physical infrastructure from logical resources, hiding complexity and enabling new capabilities. However, not all potential benefits of virtualization have meaningful value outside IT circles: Too many of our discussions revolve around the very complexity that virtualization technology should be hiding! True business value is derived from transformed virtual resources in the next-generation data center, not the incremental capacity gains of virtual servers. But how will we get there, and what will this future look like?

The Problem with Virtual Servers

Implementation of virtualization technology to date has merely delivered condensation of physical resources: 250 physical servers are condensed onto 20 physical servers, but 250 virtual server images remain. True, this does result in the reduction of data center footprint, from rack space to power and cooling, enabling moderate cost savings. But these are not examples of real consolidation, let alone business transformation.

Many have lamented this “virtual server sprawl” and suggested alternative methods of consolidating low-utilization applications into larger, more flexible “resource servers”. For example, numerous SQL servers can be combined on a single central server with more focused management. But these larger resource servers are not normally virtualized since their concentrated I/O demands can overtax current server virtualization platforms. Therefore, consolidation and virtualization remain separate.

This is the problem with conventional server virtualization. It enables us to condense data center demands for some systems, but delivers very little else apart from new backup and management headaches. Certainly we can provision servers more quickly, and we might be able to recover from a disaster more easily, but these are IT-facing benefits that other business entities care little about.

Storage and Network Virtualization

Virtualization of storage and network resources face even higher barriers. Where server virtualization has quickly delivered incremental “green” savings, these benefits are harder to come by in other areas.

Storage virtualization primarily delivers flexibility. SAN or NAS systems can be combined into larger pools, allowing existing resources to be better utilized or provisioned more quickly. But there is only a little cost avoidance to be gleaned from more efficient use of storage capacity. Real cost savings would require reduction of infrastructure, and constant data growth makes this extremely difficult to achieve. Other benefits, like enhanced data migration or heterogeneous replication, ought to be invisible to the business anyway.

Network virtualization lags even further behind. Only a few shops have attempted to use technology like InfiniBand to enable flexible virtual connectivity, though the future Converged Enhanced Ethernet concept is beginning to spark some interest. Here again, financial benefits from network virtualization technology are limited to a moderate reduction in future equipment cost.

Transforming the Data Center

In all three instances (server, storage, and network), the financial benefits are merely the sideshow. The underlying benefit from virtualization of IT infrastructure comes from the extension of IT systems outside the data center, a change on the order of the advent of minicomputers or the spread of open systems.

VMware recently laid out a serious and compelling vision of this future Virtual Data Center as VDC-OS. Their concept is evolutionary and radical at once, with the simple virtual server infrastructure of today augmented with increasingly uniform and flexible storage and network layers. This culminates in a truly virtual data center, where running server images can move from device to device, location to location, and even out to the cloud.

VMware’s brilliance is in leveraging what works today (virtual server images on ESX) to build a foundation for complete virtualization of physical resources. But virtual servers running on VDC-OS remain tied to the present: They run the same operating systems and will likely remain bound to the same “one (virtual) server per application” world view that pervades open systems today. This leads to exactly the same situation of server sprawl that has proven a management nightmare.

Others are extending the web hosting concept to enable custom applications to be run on the scalable, flexible, multi-homed servers that run the world’s biggest Internet applications. Google and Amazon’s visions are decidedly post-data center, with applications, rather than server images, being the primary unit, and database-style storage replacing conventional blocks and files. Use of these web-oriented application platforms has so far been limited to entirely new systems built from scratch to take advantage of them, limiting their appeal to current IT environments.

Where Is the Business Value?

Yet, most discussions of these virtualization strategies (mine included) fails when it comes to demonstrating real business value. We must move away from quickly-forgotten cost savings and focus instead on profoundly transforming how IT serves business goals. Virtualized infrastructure allows flexibility and scalability, changing how everything in IT works.

Whether it uses conventional operating systems and applications or re-engineered web-enabled solutions, virtualized infrastructure fundamentally changed our world. Organizations would be free to physically move their systems, even outsourcing or offshoring the infrastructure component entirely. They could move to an on-demand purchasing model for logical capacity, not just bits and bytes.

In the process, they would render current server platforms, operating systems, and storage devices irrelevant. Undoubtedly, attaining this future remains a while off, but IT professionals should consider its implications. Much of what we do is focused on making the “plumbing” work efficiently rather than serving the needs of the business. Where do we stand once the perennial issues of performance, availability, and scalability are solved?

photo by Roy Smith

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Where Will Virtualization of Data Center Infrastructure Take Us?
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Great! But I thought the new era for IT was a cloud with choice, mobility, and application support, as trumpeted by EMC’s VMware subsidiary! Wasn’t Cloud vServices from VDC-OS supposed to be the prototype cloud strategy for the datacenter?

What we have here is a simple clash of marketing amusingly taking place at (nearly) the same company. VMware figured out how to extend their server virtualization products outside the confines of the data center, and laid that technology out as a strategy with the trendy “cloud” name. Meanwhile, mother EMC is working on next-generation content storage software and decides to roll that out as a strategy and also jumps on the “cloud” meme. What’s an IT manager to do?

Defining Atmos

As predicted, EMC’s Atmos (code-name Maui) is a distributed software layer to handle the storage and management of data objects across geographically-dispersed storage devices. EMC’s Chuck Hollis demonstrates Atmos with a simple, practical example, perhaps making it sound too much like Akamai but generally getting the point across. You have a data object, write it to Atmos through REST/SOAP or CIFS/NFS, assign some metadata, and the software takes care of data placement for you. It’ll add local copies, replicate for availability and performance, compress or deduplicate, manage versions, and all sorts of goodies (if you ask it to).

But EMC already has a capable object storage platform, the Centera. We’ve just got used to the content-addressable storage (CAS) label for object storage (even though this name misses the point of object storage, in my opinion) and now EMC wants us to learn a new label for a somewhat-similar device? Steve Todd, EMC’s object guy extraordinaire, lays it out:

SAN Value = Centralized, secure multi-tenancy for blocks.

NAS Value = Centralized, secure multi-tenancy for files.

CAS Value = Centralized, secure multi-tenancy for objects (content + metadata).

COS Value = Globalized, secure multi-tenancy for content with rich policies.

Ok, so the defining capabilities of Atmos are its global scale and rich policies. And the fact that “objects” has become “content”, presumably since Atmos can handle traditional NAS (CIFS/NFS) chores as well.

Prayers Answered?

It sounds like EMC is answering my prayers for a storage revolution, delivering a highly-capable object storage platform that transcends the old limits of blocks, directories, and files. Steve Todd points out that Atmos handles five policy categories out of the box:

• Replication
• Compression
• Spin-down
• Object de-dup
• Versioning

So we write some data to Atmos, using either traditional NAS or webby dubby protocols like SOAP, and can then apply policies in any of these five categories to that data. One can also extend the Atmos to accept other policies, but the absence (out of the box) of concepts like encryption, secure deletion, retention, and access control are surprising.

I am quite puzzled about how practical these policy capabilities will be in the real world. How exactly would an application say “I want you to compress that file I wrote over NFS just now?” Hitachi’s HCAP platform, for example, also has policy capabilities and a NAS front end, and although archiving applications can communicate their policy needs, I don’t see lots of current general-purpose applications using it.

Strategic Storage?

This brings me to my puzzlement: The default Atmos policies are all general-purpose, production computing ideas, not the special-purpose, archiving and retention needs served by Centera, HCAP, and the rest. So the Atmos is clearly intended to be a production data storage system, not an archiving system to compete with Centera.

Since mainstream business applications currently don’t have any capability to specify policies like these when writing files, and since NAS protocols lack any means to communicate them even if the apps want to, we can conclude that EMC expects that Atmos users will write special applications to take advantage of it.

EMC certainly doesn’t expect that the NAS-capable Atmos will simply replace today’s distributed NAS solutions. NAS is a sideshow for Atmos. The real action will be in the REST/SOAP webby dubby applications that will be written with the platform in mind and will take full advantage of these capabilities.

If this is true, and I and others suspect that it is, then Atmos really isn’t a game-changing platform unless you change your game. If you write new applications to store data with SOAP, Atmos is a nice in-house alternative to Amazon S3 or Nirvanix, and offers a very compelling set of data management capabilities. And if you want to set up shop to compete with those service providers, Atmos is a dream come true with built-in multi-tenancy.

Datacenter Strategy

So EMC alone has two seemingly competitive datacenter strategies. And then there’s Microsoft, which announced its Azure cloud platform recently, and Amazon and the other cloud providers.

So let’s say you’re a CIO for a large corporation. Which of the following strategies is more compelling:

1. Use VMware VDC-OS to add capabilities and Cloud vServices extend your current virtual infrastructure geographically
2. Recompile and tweak your Windows applications to leverage Microsoft Azure
3. Develop new applications to take advantage of the impressive storage capabilities of an in-house EMC Atmos system
4. Point your new applications at a third-party cloud provider like Amazon or Nirvanix

IT people are practical. Although we love new technology, we tend to be cautious. We also hate massive software development efforts, and only sanction them when they’re absolutely necessary. Given these personality traits, I’d say VDC-OS and perhaps Cloud vServices still stands out as the most likely and practical scenario for the majority of applications and businesses.

This is not to say that EMC Atmos will be a flop. I’m impressed by the technology, and expect that Atmos will find buyers, just as Centera did. And Atmos might even replace Centera once EMC adds retention policies to it and scales it down as well as up and out. But Atmos will not redefine the datacenter. We’re stuck with blocks and files, and VMware’s practical strategy is a winner in that world.

Update: Marc Farley compares Atmos to WAFS, with ominous implications, and echos my recent question on what is and is not innovative.

Update 2: Chuck Hollis, Storagezilla, and Len Devanna have all come right out and said that this is only intended for certain customers with massive distributed storage needs, and is not intended as a new datacenter strategy. Even the “cloudfella” says “ciao”:

Update 3: More great information, including a reply regarding VDC-OS and Atmos from the one and only Chad Sakac, more great detail about the inner workings of Atmos from Steve Todd, and even more info from Dave Graham. Finally, although I think that Cloudfellas video is cute, I wouldn’t categorize it as viral. But those Mozy ads are awesome!

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | EMC Atmos Versus VMware VDC-OS: Will The Real Cloud Strategy Please Stand Up?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although my email archiving session always attracts a smaller crowd, they are all a dedicated bunch. One pertinent suggestion from an attendee was to ingest PST files into a special separate archive in order to ensure that messages recovered from it are treated with the proper skepticism. Questions after the session focused on the trick of engaging legal and business people in the decisions around email policy, truly a challenge. I suggested that an on-site mini-seminar for the relevant folks might help to break the logjam and illustrate the issues, something that I would be happy to arrange!

My storage virtualization session was once again placed in the main room, and a much larger group attended it. I was interested to hear just how great the impact of VMware’s VDC-OS had been. In just a week, a dozen or more folks in the audience had heard, comprehended, and strategized about the concept. It’s really that big! Others were very interested in the topic of green metrics for data center usage. How does one monitor and report the real “green” savings (power, carbon, cooling, space) for a virtualized environment? Although storage greenness is debatable, the savings from a virtualized server environment are real, and these often bundle in some of the storage numbers, too.

These topics are top of mind to me as well, and I will continue to investigate (and speculate) about them in the coming year. If you missed the show (or the handouts), I will be posting them here soon! Get my email address or head to LinkedIn by clicking the links in the sidebar (at top left).

Watch this space, and consider coming to my virtualization seminar in Charlotte on October 21 or to the Storage Decisions show in San Francisco, held November 17 to 19.

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Storage Decisions New York 2008 Feedback
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Hopefully, VMware's three pillars (vCloud, vClient, and VDC-OS) are a little more solid (not to mention closer) than the Three Pillars of Creation in the Eagle Nebula

VMware packed a lot into their 2008 VMworld conference, including an outline of their “three pillars” strategy, focused on vClient, vCloud, and something they are calling the Virtual Datacenter Operating System. While it is debatable if this last item really is an operating system, it’s certainly a major strategic change in messaging.

VDC-OS is divided into four “vServices” (Management, Cloud, Application, and Infrastructure), and one core Infrastructure vService is vStorage. Since my focus is enterprise storage, I thought I would take a moment to examine the current and future status of vStorage.

EMC’s Chad Sakac has taken up the challenge of communicating vStorage to the world, in a post to his blog, a pair of YouTube videos (multipathing, I/O dedupe), and (apparently) a session or two at VMworld. But I’m not sure it’s really entirely clear just what vStorage is and what this means to existing and future VMware storage developments. So let’s dive in and take a look.

vStorage = Storage vService for VMware Infrastructure

Let’s get one thing out of the way immediately: VDC-OS is not a new product – it’s a framework to organize VMware’s existing and future datacenter-targeted products like ESX. The company is sweeping away the current hodgepodge of server product elements and placing them all into a more-organized structure called VDC-OS.

As such, the whole thing is not fully baked at present – there are lots of missing elements, and these point to future products from VMware and others. I applaud the organization, and feel that this is the first really viable next-generation datacenter vision that I have seen. It accurately reflects the current reality of the corporate data center and includes a migration path to take it to a compelling new level.

VMware has a realistic approach to bringing current datacenter elements into the future

As we can see in VMware’s diagram (above), the Virtual Datacenter OS is made up of VMware Infrastructure, with application vServices resting on infrastructure vServices or the cloud. This last bit is the fundamental breakthrough that makes VDC-OS more than just slideware – if it works, applications (virtual appliances) can transparently move from local infrastructure to cloud infrastructure owned and operated by others. It’s transformative, and if they pull it off, this could usher in a new datacenter world.

vStorage is one of the core infrastructure services, along with vCompute and vNetwork. Although it’s not entirely clear at this point, vStorage presumably includes all existing VMware storage technologies as well as the new APIs for multipathing and I/O deduplication highlighted at the show. I envision a future in which vStorage is a set of APIs for everything from I/O to device management, and VMware even includes virtual storage appliances in this umbrella.

vStorage: What’s In It?

At the very least, vStorage includes the following:

• VMFS for shared storage
• Thin Provisioning support in VMware Infrastructure is apparently coming in 2009, but is also apparently incompatible with fault tolerance.
• Linked Clones is another desktop technology coming to VI 2009.
• The Site Recovery Manager API has also become part of vStorage, though this isn’t getting much press.
• VMware Consolidated Backup (VCB) will also apparently be a vStorage API.
• The existing snapshot API will also presumably included.
• The new API for Multipathing covers the old generic native multipathing (NMP) from ESX as well as EMC’s most excellent PowerPath. The latter is presumably an extra-cost option that enables lots of niftiness, as demonstrated by Chad in this video.
• Another new API claims to deduplicate I/O, reducing traffic for certain supported disk operations, as demonstrated in Chad’s second video.

What is in and what is out is still kind of up in the air. But you can see that vStorage includes most of our old favorites (VMFS, SRM, VCB), some desktop elements (thin volumes, linked clones), and some new surprises (PowerPath, I/O dedupe). There is enough meat here to separate this from traditional slideware: VDC-OS has legs!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | What is VMware VDC-OS vStorage?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It could be big. There is no denying the effect VMware has had on the modern data center – nearly everyone I talk to in the IT industry considers server virtualization a fundamental element of modern infrastructure strategy. And server virtualization has done more for the deployment of enterprise storage and other high availability technologies, than any other movement, from green computing to services-based infrastructure. And it has encompassed these movements, becoming the way, not just a technology.

But is VDC really an OS? And will it conquer the data center? And would this be a good thing? There’s the rub.

First, the obligatory description. Virtual Data Center effectively re-badges lots of things VMware (and the server virtualization industry in general) have been working on as “vServices”. They divide these up into Application vServices, Infrastructure vServices, Cloud vServices, and Management vServices.

These four elements, in fact, do sound like a post-modern definition of an operating system, much more so than Google Chrome. VMware includes the ability to share resources, execute applications, and store data in a managed way. And the cloud component is reminiscent of how the old client/server architecture has evolved into our modern connected world. In this way, VDC really is an operating system for the enterprise data center, and extends it into a cloud beyond those doors.

This is the most compelling and realistic post-datacenter world I have heard of, thoroughly trouncing shared infrastructure, the (Amazon/Google) cloud, SaaS, Java or Linux everywhere, Sun’s containers, and Microsoft’s world of Windows. For the first time, we are talking about an infrastructure that could actually be built, wouldn’t require a forklift (or shipping container) or the migration to an entirely new software environment, and reflects the diversity of modern IT systems.

Certainly, VMware has heavyweights in their corner. Cisco provides the connectivity, EMC provides the storage, Intel provides the CPU, Dell provides the servers, and so on. But it’s not that simple. Like Microsoft, VMware will have to manage the “input” from every networking, storage, CPU, and server provider, not to mention the vast ecosystem of software components. It’s much more like Windows than Macintosh in this respect, with VDC being a loosely-federated OS rather than a closed monoculture.

I predict that how well VMware handles the divergent parties trying to play in their OS will determine the future not just of VDC, but of VMware itself.

Oh, and VMware also introduced View, perhaps the future of the desktop.

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | VMware Virtual Datacenter Operating System: Heavyweight or Hot Air?
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.