I’ve posted before about the spot where most successful enterprise storage arrays sit: The “fat middle” between performance and capacity. But software is driving a wedge into the SAN, and I see radical architectural change on the horizon. There are so many storage virtualization technologies sneaking into server architecture that it’s impossible to think one won’t take hold. And virtualized storage doesn’t want a “fat middle” array!
Here’s the Rack Endgame series:
No Demand For Storage Virtualization
Storage virtualization has been around longer than me in this industry, but it has never taken hold. As far back as I can remember, companies have been developing and trying to sell a universal platform that would allow storage to be more dynamic, moving from place to place according to application needs. Companies have also been working on distributed storage platforms for decades, applying the principle of data gravity which says that storage should be close to compute.
Yet, as I discussed in my post, “The Fat Middle: Today’s Enterprise Storage Array“, today’s storage architecture is remarkably simple: Centralized storage systems form the core of the datacenter, a “Jack of all trades” approach with no consistent virtualization outside the array. Why has storage virtualization or distribution never taken hold?
Perhaps the main reason for today’s lack of storage virtualization is the heterogeneity of data center operating systems and applications. Simply put, there is no single platform that can be targeted by a storage virtualization layer, and there has been no demand for an in-network virtualization platform either. The safest approach has been to let the storage array itself provide what little virtualization there is, along with high availability and data services.
VMware Provides an Opening
But VMware challenges this paradigm. Today’s most widely-used “storage array” is actually not an array at all: It’s VMFS, the storage layer of VMware vSphere. It provides all of the functions of a traditional storage array within the hypervisor itself. Yet VMware is in the process of radically transforming this storage layer, and the implications are very exciting! VMware Virtual SAN (colloquially called “VSAN”) brings storage distribution and virtualization to mainstream data centers, reducing the pressure on centralized storage arrays to provide advanced data services.
With VSAN, a vSphere data center no longer needs centralized storage! VSAN supports all of the best features of vSphere (HA, DRS) without SAN or NAS and brings data distribution to the party as well, localizing data close to compute. It even supports automated tiering (caching really, but that’s a semantic battle for a different day). As demonstrated by EVO:Rail, VSAN gives customers a mainstream SAN-less alternative.
But it’s not just VMware that’s challenging the status quo. Microsoft is promoting an all-Windows Server data center. Nutanix has made waves with their converged data center solution, and it too boasts a “No SAN” approach. SimpliVity, Atlantis Computing, and Scale Computing are also charging in. Then there’s the dark horses of storage virtualization: Avere, PernixData, and Infinio are virtualizing storage today, though they’ve been pretty quiet about the implications of this. It’s a short jump from a caching solution to a real storage hypervisor!
Endgame: Distributed Storage
The goal of all this is accommodation of data physics, specifically the “gravity” of data proposed by Dave McCrory. Because data should not be centralized, storage shouldn’t follow a central/shared architecture either. Storage should be moved to the compute nodes to reduce latency and bring about real scalability of the entire system.
Architecturally, this is what Nutanix, VSAN, and the rest are all about. They move data closer to compute, eliminating the need for an artificial, external shared storage infrastructure yet preserving the enterprise-class data services we’ve all come to rely on. They identify and move data by placing storage intelligence closer to applications (on the same physical hardware if not in the same operating system).
The benefits are too compelling to ignore and have profound implications for the shape of enterprise storage in the future. If we live in a “no SAN” world, we no longer need to build “Jack of all trades” storage systems or rely on “do it all” interconnects like Fibre Channel. The distributed storage world wants specialist storage devices: Ultra-performance flash on PCIe or memory channel and ultra-scale storage on a slower and cheaper bus.
Stephen’s Stance
We were never able to achieve storage virtualization in mainstream enterprise IT because we lacked the ability to identify and move data non-disruptively. This has been solved by caching and distributed storage solutions, and it’s only a matter of time before the legacy need for centralized storage falls away. As the wise man said, maybe not today and maybe not tomorrow, but soon we’ll live in a different world.
Note: These are topics I discuss in my public speaking engagements, including my Truth in Storage seminar with Truth in IT. Check out the schedule for Truth in Storage and come listen to the whole story!
Disclaimer: I work with VMware, PernixData, Infinio, Avere, Nutanix, Atlantis Computing, Scale Computing, SimpliVity, and most other companies in enterprise storage with Foskett Services and Tech Field Day. I don’t think I’m biased, but you can draw your own conclusions.
“It provides all of the functions of a traditional storage array within the hypervisor itself.” Well, if you dont include data integrity protection, hardware/firmware management, background scans, …
That aside, consider that most of the time in hyperconverged solutions, the data that is being accessed by a VM isnt on the same physical server as the VM. Most IO will ultimately go over the interconnect. There is no relationship with which physical server a VM runs on, and the server that has all (or even some) of the data within the VSAN datastore. The SSD read cache is a L2 cache, and is on the disk group “under” the interconnect layer. Not that this is a bad thing – but technically speaking, once the cluster grows large enough the compute is still not on the same physical node as the storage. The only way to resolve that is to either fully mirror data objects (and not allow a mirrored copy to span more than one node), and then to limit the compute threads that access that data to running on those nodes.
I have nothing against hyperconvergence. Massive scale-out needs highly scalable ways to disperse computation across many nodes, even allowing for elasticity that stretches into the cloud. But the driver for hyperconvergence should be realistic about what it is and is not – and adopt the architecture when it makes sense.
Yes, I’m with NetApp, And my views and comments are my own and not represent those of my employer.
I thought that what you are describing here is exactly how Nutanix works. It uses meta data around the I/O to determine if the read/write copy of the vmdk should reside on the server where that VM is running. Intelligently reducing the interconnect traffic.
I just confirmed this. VSAN does read/write caching but Nutanix will move the writable copy to the local server automatically localizing all I/O for each VM.