How granular is your storage? This question is just as relevant today as when I first published this article in 2008!

Many storage challenges focus on correlating high-level uses of data (such as applications) with the nuts and bolts of storage infrastructure. These discussions often revolve around the conflict between data management, which demands an ever-smaller unit of management, and storage management, which benefits most from consolidation. Developing data management capability that is both granular enough for applications and scalable enough for storage is one key to the future of storage.

Storage Management: Scaling Up

As I discussed in a previous Sunday Series piece, Turning the Page on RAID, the data storage industry has traditionally focused on reducing granularity. Disk capacity has expanded, and RAID technology has multiplied this by combining multiple physical drive mechanisms into a single virtual one. Storage virtualization technologies, from the SAN to the server, have also often been touted primarily as a mechanism to reduce heterogeneity. From a technical perspective, therefore, granularity has been an obstacle to overcome.

The core organizational best practice for storage management is the reduction of complexity and the enforcement of standardization. Consolidation of storage arrays and file servers is a common goal, as IT seeks to benefit from economies of scale. The goal of both initiatives is the creation of a storage utility or managed storage service. This mirrors efforts on the server and network sides to consolidate and virtualize hardware.

Although both technological and organizational factors have traditionally driven granularity out of storage, this does not have to be the case. Virtual pools of storage are ideal for providing storage on demand, as disk-focused RAID groups give way to more flexible sub-disk storage arrangements. And an operational focus on standardized storage service offerings has the potential to enable scalable management of these smaller units.

Filing Service

File-based protocols would seem to have more potential for granular storage management, but they have been undermined by the hierarchical nature of modern file storage. Whether the connection to a file server uses NFS, CIFS, or AFP, the key unit of management is actually the shared directory, not the file. All files in the share \\firefly\backups would be located on the same server and would be managed as a unit.

NAS virtualization can change this somewhat, as can more specialized NAS servers. Although Microsoft DFS enables consolidation and virtualization of NAS shares, it does not allow subdivision of shares below the directory level – all files in a directory must be placed on the same server. Tricks like stubbing and links allow for some movement, but these do not solve the core issue. Specialized virtual NAS devices from F5 (the ARX, nee Acopia), NetApp, BlueArc, Symantec, and others have the ability to move files individually, providing as much a virtualized storage environment as any block-focused enterprise array. Avere is also beginning to talk about granular file management.

But even an ideal virtualized file server lacks the kind of granularity demanded by users. They care about data, not files, and most applications consolidate their data storage into a few files. Consider a database, for example, where users want each record treated uniquely but storage devices see just a few much larger files. We need a storage revolution, where someone creates an ideal storage platform in which each individual record or object includes custom metadata and is managed independently. This would truly be a massive change, however, and it is not clear that all applications will follow the object storage model of Google and Amazon.

Small is Beautiful

Barring a revolution in data management, our best hope is to allow greater granularity in storage management. As mentioned above, virtualization technology has the potential to enable management and protection of any unit of storage, right down to the individual block or record. But the reality of storage virtualization has not matched its promise.

What is needed is greater integration. Each layer of virtualization (file system, volume manager, hypervisor, network, array, and RAID) also hides necessary details from lower layers. Consider the case of a virtual server snapshot: The application and filesystem must be in a quiesced state to allow a snapshot to be taken at the storage level, but the storage array has no intrinsic information about how its capacity is used. A given LUN might contain dozens of servers on a shared VMFS volume, so all must be snapped together.

Integration can be enabled by sharing more information through APIs. VMware leverages Microsoft Volume Shadow Copy Service (VSS) integration for shared storage so a VMFS snapshot can call the operating system and even applications (Windows Server 2003 only, for now) to prepare the data. Similarly, VSS can communicate directly with supported iSCSI and Fibre Channel arrays, calling a snapshot at the right moment. And Microsoft is, no doubt, enhancing VSS as we speak.

As virtualization technology matures, expect this type of integration to improve. We hope to see more APIs exposed by VMware and Microsoft, allowing communication up and down the stack to break through the information barrier. Imagine a future where a standard API like VSS can pass a message through VMware, Xen, and Hyper-V to the underlying storage array to initiate a snap. I predict that this kind of integration-enabled granularity is not too far off.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Granularity: The Hidden Challenge of Storage Management
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

CommVault introduces an updated version of their all-encompassing Simpana suite

CommVault is one of those enterprise IT companies that likes to go their own way. A spin-out of AT&T’s famed Bell Labs, CommVault’s Simpana software integrates many aspects of data management, from backup to e-discovery, under one umbrella. Last year, the company impressed me by adding cloud storage as a backup target equal in status to disk and traditional tape. Now the company is doing the same for storage-based snapshots, accelerating data protection for virtual machines.

Simpana 9: A Broad Update

Nearly every aspect of CommVault’s Simpana 9 is refreshed, though development and unveiling has been an ongoing process throughout the year. Following Simpana 8 by two years, the new software integrates work done integrating cloud storage and virtual machine backup over the last year or so. The “data management” portion of the product (including backup, archiving, and replication) was detailed this week, and one expects more information about the other half (“information management”) to be forthcoming.

Simpana 9 is available as of October 5 from OEMs, resellers, also direct from CommVault. Pricing is simplified this time around with CommVault using a capacity licensing model. Cost is based on the largest backup or archiving job, though the old licensing model still still available. A “circuit breaker reset” allows the system to go over the licensed level temporarily in a pinch.

Pick Your Target: Tape, Disk, Cloud, or Snapshot

There has been a long-standing back-and-forth discussion in the enterprise backup community regarding snapshots as backups. My stance has been that, while snapshots are certainly a more-effective mechanism for copying data than dump-to-tape, it takes more than data to have a useful backup. Any system worth using must schedule, manage, organize, and recover data. Therefore, although snapshot-based backup is a good idea, it wasn’t ready for prime time until it was integrated with a solid backup management system.

Elevating snapshots as a first-class backup target is exactly what CommVault is doing. Just as their earlier releases added cloud as a backup target, Simpana 9 makes snapshots just another option. Commvault recognizes that needs change over the lifecycle of a backup, and the location of data should change, too. A snapshot makes an excellent “operational recovery” target, allowing frequent copying and quick restore, but is sub-optimal for long-term archiving. Conversely, tape is a poor frequent target due to its linear nature but has proven itself for long-term offline archive.

Simpana 8 supported HDS and Dell for snapshots, now Simpana 9 adds volume leaders EMC and NetApp, along with HP, IBM, LSI, and Sun. Data is indexed and cataloged, and these are stored with the data on disk, tape, or cloud for later access. Speaking of cloud, Simpana 9 supports Amazon, Azure, Nirvanix, Rackspace, Iron Mountain, EMC Atmos, Dell DX, and HDS HCP.

You might also want to read CommVault Gives Cloud Storage A Seat At The Adult Table

VM Backup Acceleration

CommVault was demonstrating their abilities to back up machines using VMware VADP at VMworld. There, they took just 17 minutes to back up 500 machines using snapshots on an IBM XIV storage array. I imagine most of the supported platforms could do just as well.

Simpana 9 automatically discovers running guests using VMware vCenter. This allows for daily movement of machines since they are located each time and the backups are coordinated for universal, consistent coverage. No machines will be missed, either. In fact, given that many snapshots work on a block level, it would be impossible to not backup up clients sharing the same LUN.

On the back side, Simpana uses an ESXi proxy to grab snap content and catalog the virtual machines contained there. It then moves the data off to disk, tape, or cloud just like any other backup content.

CommVault is also working on Hyper-V support, using VSS rather than VADP.

Universal Dedupe

The third element of Simpana 9 is integrated source and target deduplication. Simpana’s client software deduplicates local data on its own before sending it to the server. These deduped blocks are then checked against the universal set on the server before being stored, potentially bringing a “best of both worlds” solution to the source-versus-target argument.

Migration to Simpana 9 From NetBackup and TSM

One of the long-standing criticisms of Simpana is that it was just too far-reaching for easy testing, migration, and uptake. It has been seen as an all-or-nothing proposition, requiring a commitment that is hard to win and harder to back away from in the event of trouble.

Simpana 9 addresses part of this concern with a so-called “Fast Pass” migration system that allows configuration to be pulled in from more-traditional backup products for a seamless transition. The Simpana data connector talks to NBU 6.0, 6.5, 7.0 and TSM 6.1 master server, discovers backup policies, and imports these (clients, schedule, history, etc) into the new configuration. Then automatic install software pushes CommVault agents into place and takes over operations, leaving the old system running for historical recoveries only.

Stephen’s Stance

I love the idea of promoting data from operational snapshots to tape or cloud for archiving. And I love the idea that end-users can choose the combination that fits their needs within a unified package. I’ve long been a fan of CommVault’s products, and this upgrade makes them that much more compelling. I’ll leave it to wiser minds to debate their implementation and effectiveness, but it sure looks good for me. The simplified licensing and purchasing model is welcome, too – backup is a thorny combination of pricing models.

The idea of migrating seamlessly from old to new is interesting, but I wonder how well it will work in practice. My time as a backup consultant taught me that these environments tend to be complicated, finicky, and outdated. It’s nice that CommVault can read in a TSM 6.1 configuration, but what about older versions? The same goes for NetBackup. And will it correctly handle the weird rule sets found there? But a greater issue is whether it should import the old settings at all: Perhaps migrating to a new backup system is a good time to revisit the assumptions behind your configuration!

All in all, Simpana 9 looks like a compelling update for CommVault lovers or those disaffected by their current platform. Organizations having difficulty meeting backup windows should also give it a look, since the snapshot technology should be a big help there. And anyone looking at cloud storage as a backup and archiving target should start with CommVault. Others will certainly be intrigued, but I am not sure that they will be compelled to upgrade by these features alone.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | CommVault Simpana 9 Takes Backup To A New Level
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Update: @SSauer points out that this is vShield, and posted some information here: A New Generation of vShield Security Products

vSwitch, vNetwork, vShield?

Most hypervisor products include an internal virtual network switch, but VMware’s ESX has multiple choices. The original “dumb” virtual Ethernet switch was augmented by vSwitch back in the ESX 3 days, bringing more-advanced configuration options.VMware improved and renamed the vSwitch in vSphere 4, creating the vNetwork Standard Switch (vSS).

But it was the introduction of vNetwork Distributed Switch (vDS) in vSphere 4 that really set VMware’s network capabilities apart. The champion of this field is Cisco, whose Nexus 1000V virtual switch extends their NX-OS datacenter networking OS right into the ESX world.

The Cisco Nexus 1000V runs both the supervisor and Ethernet modules inside the virtual ESX environment

As illustrated above, the Nexus 1000v consists of two key components:

1. The Virtual Supervisor Module is an implementation of NX-OS on an ESX virtual machine, and provides the interface and configuration of the virtual network
2. The Virtual Ethernet Module runs at a lower level in ESX, replacing the vSwitch for networking between VMs

One can think of the Cisco Nexus 1000v as a specialized replacement for the more-generic vNetwork Distributed Switch. Both include plug-in vSwitch replacements and centralized management, and both implement more-advanced network protocols like private VLANS and receive-rate limiting as well as supporting vMotion. But Cisco’s 1000v goes much further, adding PortChannel, LACP, security and QoS, and advanced management features.

See this comparison of vSwitch, vSS, vDS, and 1000v as well as Rich Brambley’s vSS/vDS post

From a technology standpoint, the key to both vDS and 1000v is the ability to replace the core ESX vSwitch with a more-capable alternative. Now let’s turn to what VMware might be introducing next.

What Do We Know?

Howie Xu, VMware Director of R&D, released a video discussing his sessions at VMworld, entitled ”The Future Direction of Networking Virtualization” (TA8361). This video begins with a quick pan past Xu’s whiteboard (pictured below) and includes a discussion of the state of the art, vision, and product and technology roadmap for VMware’s networking-related efforts.

The part that really piqued my interest was later in the video, when Xu talks about creating a “networking virtual chassis or hypervisor” to allow third-parties to develop and roll-out advanced networking devices within vSphere. VMware has already steamrolled through the heart of server-based applications, making VMware-based virtual appliances as common an installation format as the DVD. Now the company is turning its attention to the network. This is huge!

Xu speaks of both a platform and a service to support this ”open extensible networking virtual chassis platform,” and goes on to suggest that it could be used by “networking security, load balance, application acceleration, IP address management, and performance management” products. The virtual appliance marketplace is already populated by the familiar names in networking, from F5 to Bluecoat to Checkpoint. Therefore, VMware must be talking about something much deeper and more advanced than merely encouraging the creation of more virtual appliances!

The core question is whether VMware is opening up the “green box” in my diagram above to run third-party applications and what level of system access they will get.

Consider this a vFabric pre-intro!

Then there is the name. The whiteboard prominently includes the words, “vFabric Intro” in the corner. Judging by the rest of the readable content, this indicates that this new technology will indeed be called “vFabric” as Stu speculated.

Update: It appears that vFabric is not the name of this virtual chassis (thanks, Stu and Howie!). Good thing, too, since the name, “vFabric” is a registered trademark of QLogic Corporation, for “computer software for managing computer hardware, namely switches used in networks.”

What’s Coming?

I’m more of a storage guy, so I rang up my friend Greg “Etherealmind” Ferro and ran some ideas past him. Greg and I talked about the needs of network-based devices, and how they differ from traditional server-based applications.

Greg suggests that vFabric will really assist vMotion

Networkers have been conditioned with the belief that custom silicon is the best way to achieve low latency and high performance for network devices. The same could be said of the storage world, where companies like HDS, 3PAR, and BlueArc pride themselves on their custom ASICs. But EMC, HP, and others are proving that Intel’s server-class CPUs and peripheral busses now have the guts to go head-to-head with custom silicon. The networking world is no different, with many newer companies basing their products around industry-standard hardware.

But deploying these systems in a virtual environment is more challenging. Can a virtual machine hypervisor prioritize threads for network devices? Can it handle the overhead related to networking operations in real-time? What happens in the event of a DDoS or network flood? Most network devices run real-time operating systems like VxWorks or QNX to ensure packet throughput, but virtual environments are notorious for “overflow” of I/O or CPU load between guest machines.

The whiteboard provides some hints as to how VMware will tackle these issues. First, we spot the term, “latency-aware queueing,” which suggests that a mechanism will monitor the hypervisor and alter the queues for virtual network devices as the load changes. As latency rises, the hypervisor can move workloads to different processor cores or even alternate hardware using vMotion. We also spot a reference to “non-blocking”, suggesting an asynchronous I/O mechanism will reduce the likelihood that one of these virtual network devices will have to wait for data.

Both of these technologies are hallmarks of real-time operating systems (RTOS), and are critical to the design of scalable hypervisors like VMware’s ESX. It is likely that the company is developing an advanced hypervisor environment for these specialized devices, an evolution of vDS and the API that allows the Cisco Nexus 1000v to run its Virtual Ethernet Module.

Stephen’s Stance

If our assumptions are true, then this is an exciting development indeed. If VMware exposes the “green box” in our diagram above to third-party developers, we could see an entirely new and more-powerful ecosystem evolve around VMware vSphere. Running virtual network devices in a quasi-real-time environment will enable even-greater integration and flexibility.

The Nexus 1000v has not eliminated purchasing of Cisco hardware, and vFabric will not destroy the larger network device market. But we expect wide vendor support for the concept, especially those involved in lower-end and remote-office applications. We would love to see Palo Alto Networks, Infoblox, SolarWinds, and Vyatta, to name a few, developing next-generation applications for vFabric. Virtualization-aware integrated networking shouldn’t be the sole domain of Cisco.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | A VMware Hypervisor For Networkers?
This post was categorized as Computer History, Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

For more information on earlier updates, see my articles:

• Storage Fixes in VMware ESX Server 3.5 Update 2
• Storage Changes in VMware ESX 3.5 Update 3

Expanded Support for Enhanced vmxnet Adapter

Not specifically a storage change, but the enhanced vmxnet adapter introduced back in the original release of ESX 3.5 now works with most versions of Windows Server 2003 and XP Pro. Look for improved performance when using guest-side SMB and NFS as well as the guest iSCSI initiator. Note that you cannot select this driver when configuring non-Enterprise Edition machines; you have to select Windows Server 2003 Enterprise Edition (64-bit) regardless of which version of Server 2003 you are using.

Expanded SAS and SATA Controller Support

If you’d like to install ESX on a server equipped with a PMC 8011, Intel ICH9 or ICH10, CERC 6/I SATA/SAS Integrated RAID Controller, or HP Smart Array P700m Controller, you’ll find happiness in Update 4.

The Intel controllers are especially important, as we’re seeing them used more and more and this driver is more full-featured than the earlier Broadcom HT 1000 and Intel ICH7 drivers. The Intel ICH9/ICH10 is a dual-mode (IDE/ATA and AHCI/SATA) driver, supports SATA hard drives, SSDs, and optical drives, and now enables VMFS support when in AHCI/SATA mode. It’s not clear whether VMware actually supports VMFS datastores on ICH9/10 SATA, but it says it works. Anyone want to try it out? One thing is certain: You can’t use SATA drives in a shared/clustered environment because SATA does not include reservations. See this tech note and especially this question:

Earlier, it was mentioned that we can create VMFS if we use AHCI/SATA mode. If so, why did VMware not claim VMFS support when using SATA controller running in AHCI/SATA mode?

VMware might decide to add support in the near future. There is no strong need to have VMFS support on a SATA drive, because native SATA protocol does not support reserve/release. Reserve/release is needed if VMFS is used as clustered file system in a shared disk environment.

PXE Boot Support

Rich at VM/ETC points out that Update 4 includes experimental PXE boot support for ESX and ESXi. As he notes, this has major implications for cloud computing platforms, since it means that ESX servers can boot guests without local storage at all. Very interesting! Let’s bet that Update 5 (expected in June or July) will include this as a supported option.

Updated QLogic, Emulex, and LSI Drivers

Like most ESX updates, this one included updated Fibre Channel drivers.

• The QLogic Fibre Channel Adapter driver and firmware (versions 7.08-vm66 and 4.04.06, respectively) include bug fixes and enhanced NPIV support.
• On the Emulex side, driver version 7.4.0.40 supports the company’s HBAnyware 4.0 management software.
• Users of SAS and SCSI LSI MegaRAIDs will find driver version 3.19vmw (megaraid_sas) and 2.6.48.18 vmw (mptscsi) which improves performance and enhances event handling capabilities.

Expanded Sun Storage Array Support

All you StorageTek loyalists out there will be happy to see support for Sun’s low-end StorageTek 2530 SAS array as well as the modular 6580 and 6780 Fibre Channel arrays. It looks like just about every model in Sun’s current storage lineup is now supported in ESX.

Expanded Network Card Support

Support for Gigabit cards is greatly expanded, including HP’s quad-port NC375i and dual-port NC362i and NC360m, Intel’s Gigabit CT and 82574L, and NetXtreme’s BCM5722, BCM5755, BCM5755M, and BCM5756. Intel’s widely-used 10-gig 82598EB cards are now supported as well.

Tweaks and Fixes

Looking through the release notes, a few storage-related tweaks and fixes stand out:

1. WMware can optionally automatically throttle back the queue depth when congestion is encountered. See Controlling LUN queue depth throttling in VMware ESX for 3PAR Storage Arrays for more information.
2. VMklinux module heap size can now be adjusted as LUN queue-depth values are increased. Since tuning LUN queue depths is one common trick of the storage trade to improve performance, especially in queue-stingy systems like ESX, this is welcome news. But call VMware support before you monkey with it!
3. An RDM-related issue where SCSI inquiry data over 36 bytes was truncated or corrupted (for example when using Microsoft VSS and NetApp SnapDrive) has been resolved.

Well, that’s all folks. I suggest you all read the release notes for yourself, and please leave a comment if you see an error in what I wrote here or have something to add!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Storage Changes in VMware ESX 3.5 Update 4
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Microsoft delivered a solid MVP Summit

Note: Every one of these items has previously been discussed by Microsoft. I can’t yet talk about any other storage features I might have seen…

1. iSCSI Quick Connect – No more fumbling through tabs and popups, just enter an IP address and you’re connected! Yeah, baby!
2. Wire-speed 10 Gb iSCSI – The software iSCSI initiator can saturate a 10 Gb Ethernet link to a NetApp FAS3070; 1,174 megabytes per second
3. BranchCache – Accelerated access for workers in remote offices; it can use peer-to-peer sharing of hashed chunks if a Windows server isn’t available to cache data
4. VSS LUN Resync – The awesome VSS API is extended with the ability to resync a LUN from a snapshot; supports both software and native array capability
5. VSS Express Writer – Now there’s no excuse for applications to not be VSS backup-compatible
6. PowerShell – The very-sweet command line and scripting environment works with many server areas, including some parts of the storage world. Please implement it everywhere, Microsoft!
7. Transparent Caching – Administrators can allow Windows 7 clients to transparently cache file content from SMB servers when links are slow
8. Background file sync – Windows 7 will synchronize offline files in the background
9. Native VHD Support – VDS now supports a virtual drive just like any other – create, expand, compact, etc… And Server Backup stores its data in VHDs too!
10. Windows Home Server Drive Extender – WHS has file-level data protection and flexible disk management – it reminded me of DROBO! Not new, but I hadn’t ever heard about it… Thanks to Donavon and Alex for the info!

“Hate Microsoft” is a popular meme, but it’s misplaced. The Windows Server team, and the storage folks in particular, keep moving the ball forward, and the whole world of IT infrastructure benefits. My laptop may have a fruit on the lid, and my focus may be on three-letter storage and server companies, but Microsoft is a key part of our world, too.

The MVP Summit wasn’t just a great learning opportunity – it was a blast! We’ll see how much of that leaks out, but there might be an incriminating YouTube video of yours truly on stage at the Experience Music Project, Donavon tagging Steve Ballmer for WHS, and a lot of MVPs letting their PC love fly for the camera. There was also a Twitter tribute to Frank McCallister an MVP who passed away shortly before the Summit.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | 10 Cool Storage Features From the 2009 Microsoft MVP Summit
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This is the second entry in my Top-Ten in Storage series.

Not every innovative product can succeed in the market, and no matter how good some ideas seem, they can fail to make much of an impact. The truth is, people buy solutions, not technologies.

This list includes products so cool, so ahead of their time, that they just couldn’t fail. But they did.

1988 EMC Orion/Allegro

EMC’s RAM-based storage array predated and presaged its Symmetrix, offering lightning-quick I/O for very small workloads. But the Orion had a few serious drawbacks: It was astonishingly expensive, offered tiny capacity, and was never available for open systems. EMC’s initial offerings in the open systems market, Centriplex and Harmonix, also deserve mention though the company proved its worth with its “best” Symmetrix a few years later.

1985 3COM 3Server

I’ve written about the 3Server before, but let’s sum up: Before RAID, SAN, and NAS, and even before NetWare and NFS became entrenched, 3COM introduced a flexible multi-disk network storage and print server called the 3Server. It was an innovative combination of hardware and software, but it lost out to NetWare on open PC hardware in the market. Although the 3Server lasted just a few years, it made a lasting contribution to the field: Its 3+Share software became 3+Open, then LAN Manager, and was finally absorbed by Microsoft to become SMB/CIFS.

HP AutoRAID (High Availability Disk Array Model 12)

The first post-RAID virtual sprang fully-formed from the big brains at HP to challenge the industry-leading modular offerings from Data General’s CLARiiON and DEC/Compaq’s StorageWorks. It one-upped everyone with automatic relocation of LUNs between different RAID levels based on workload about a decade before Compellent, EqualLogic, and 3PAR had anything of the sort. But the AutoRAID had two key drawbacks: Like mama bear, it wasn’t big enough for big companies but was too big and expensive for little ones, and it was SCSI-only (though a Fibre Channel bridge was a common accessory). Some of the concepts lived on in the HP EVA line, but AutoRAID didn’t last long.

Microsoft Object File System (OFS)/Relational File System (RFS)/WinFS

Microsoft shoulda-coulda-woulda revolutionized file storage with nearly every major operating system release. It looks the same every time: A database/filesystem hybrid that would add structure to the vast unstructured file world. Despite the hype, we haven’t seen it yet. 

MangoSoft Medley

Consider the typical LAN: Lots of clients with lots of hard drive space connected to a server with yet more disk capacity. Wouldn’t it be nice if the space on these clients could be used as a virtual network server? Well that’s just what MangoSoft announced in 1997, gaining much attention in the PC press. Sadly, the idea never really caught on, although MangoSoft continued. But distributed network storage has become the industry’s Brigadoon, trotted out as a new idea year after year.

StorageNetworks

This is my flop – I worked for StorageNetworks from 1999 through 2001.

StorageNetworks was launched in 1998 to provide off-premise storage and backup as an on-demand service for enterprise customers. They sought to take advantage of the emergence of high-speed Fibre Channel connectivity over metro distances to commoditize storage capacity. But only Houston (where I worked) had an acceptable infrastructure for the service, so the company changed focus to hosting centers. Then off-site backup. Then software. Nothing worked apart from vendor-independent services, and the (now public) company was shy to base its revenues on that. StorageNetworks was gone in 2003, but does the service remind anyone of Amazon S3 or Nirvanix? Maybe it was just ahead of its time.

Compaq VersaStor

Announced in 1999, VersaStor would have been a revolution in Fibre Channel SANs, a full out of band virtualization solution leveraging specialized HBAs directing traffic. It was continually pushed back, finally being “merged” (after Compaq) with HP’s StorageApps in-band SANLink to become CASA. But then EMC sued HP over virtualization patents, derailing CASA, and the whole mess was permanently shelved at the end of 2003.

IBM Ice Cube/Collective Intelligent Bricks Hardware

IBM rocked the storage press in 2003 with their announcement of Almaden Research’s Ice Cube concept. Instantly dubbed “Lego brick storage” (trademarks be damned!), the storage units could be stacked in two dimensions, scaling without limit. Coolest of all (literally), the bricks were chilled with water! Although the concept progressed, we still haven’t seen it. But this didn’t stop Seagate from developing a similar concept, ISE, which is now Xiotech’s main offering. IBM’s Storage Tank and VSS could have been nominated, too, but I’m not a sadist.

Revivio CDP

Revivio was the pioneer of continuous data protection (CDP), with great technology and people. But no one (other than Symantec, eventually) bought it.

Pirus

Little Pirus was working on a small but scalable virtualized target when it was acquired by Sun in September of 2002. Their technology was launched as the StorEdge 6920, a mini alternative to the HDS USP in 2004, but (according to insiders) it didn’t exactly light the world on fire. Sun finally pulled the plug on the 6920 in early 2007, with HDS taking over continuing support for anyone who bought the moribund product.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Top Ten Coolest Enterprise Storage Flops
This post was categorized as Computer History, Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

So let me sum up:

1. Server Message Block (SMB) 2.0 is probably the biggest news.  Microsoft re-worked this protocol for Vista and 2008 to reduce chattiness, combine multiple commands in a single packet, and allow larger packets.  This should improve performance for just about everyone, but is only in those two operating systems right now.
2. The excellent multi-path I/O (MPIO) driver from Microsoft is also included right out of the box, including device-specific modules for both iSCSI and Fibre Channel.  This is the first version of Windows to come with MPIO, though it’s available with the iSCSI initiator and OEM drivers for other versions.
3. As in Vista, Server 2008 allows you to shrink NTFS filesystems on the fly.  It’s not quite as flexible or forgiving as some third-party tools, but it’s certainly easy to use!  Just go into the disk administrator snap-in and try it for yourself.
4. Windows finally has the option to leave new LUNs alone instead of trying to mount them.  This was one of those things that gave Windows a rep as a bad neighbor in SAN environments.
5. NTBackup is gone, replaced by a new Server Backup MMC snap-in.  I haven’t tried it, but I hope it’s better than the one that comes in Vista!  I ended up keeping NTBackup around on my laptop…
6. A new MMC snap-in called Storage Explorer lets you manage WMI-compliant SAN devices.
7. The Volume Shadow Copy Service (VSS) and DFS interfaces are improved and are now scriptable.
8. Server 2008 now automatically aligns filesystem boundaries with storage, which was one of those dark and secret skills us storage guys used to share amongst ourselves.  This can increase performance in high-I/O environments.
9. NTFS (in both Server 2008 and Vista) now has symbolic link support, just like UNIX and Mac OS X.  Note that this got mangled in the TechTarget article.
10. NTFS was also tuned and tweaked a bit for better stability and crash recovery.

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Windows Server 2008 Changes Storage
This post was categorized as Enterprise storage, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

1. Support for Volume Shadow Copy Service (VSS) for filesystems Windows Server 2003 and Windows Server 2008 guests.  This means VMware services like VCB and SRM can now signal Windows Server to quiesce filesystems before creating a clone or snapshot and is a major addition!
2. Windows Server 2003 guests also get application quiescing, where supported.
3. You can now extend a live, running VMFS volume as storage is added, just like Windows Vista and 2008 guests could already do with raw device mode (RDM).  Note that this only works for flat disks with no persistent snapshots open.

A few items are now officially supported, too:

1. Support for Storage VMotion between and among Fibre Channel and iSCSI datastores is here, and in Update 1.
2. iSCSI over 10 gigabit Ethernet (10 GbE) is now supported allegedly, though the I/O Compatibility Guide still says “Gigabit only”.  Of course, it already worked, but it’s nice to have official support just the same…  Bring on the datacenter Ethernet!  They’ll support this in Update 1 too.
3. And NFS over 10 GbE is supported, too!

1. They supposedly added support for 8 gigabit Fibre Channel (8 Gb FC) HBAs, but I don’t see it in the I/O Compatibility Guide. The Emulex LP12000/LP12002 are noted as 4 Gb only, the QLogic QLE2560/QLE2562 aren’t listed, the HP 81/82 series isn’t there either.  Update:  QLogic and Emulex HBAs are supported after all!
2. Support for more Serial-Attached SCSI (SAS), including Dell SAS 5/E, LSI 1078, Sun SG‐XPCIESAS‐R‐EXT‐Z, and HP SC08GE is listed in the Stotage Compatibility Guide.
3. CIM system health monitoring of some QLogic and Emulex HBAs is new too.

[ad#banner]

See also:

• Scott Lowe’s (as always) excellent overview
• VM /ETC’s colorful post
• Excellent coverage at Yellow Bricks

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Storage Fixes in VMware ESX Server 3.5 Update 2
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

It wasn’t always like this – be thankful if you don’t remember FTEDIT! But ever since Windows 2000, Redmond has made improvement after improvement, remaking “bad (SAN) citizen” Windows into Martha Stewart. But unlike the latter, Microsoft hasn’t called much attention to its skills, and this is a shame…

Take iSCSI – Microsoft was an early supporter of the protocol, releasing an excellent software iSCSI driver as a free download. They also bundle a limited (but continuously-improving) volume manager with all modern versions of Windows. Then there’s VSS, which is the first hardware-independent (even hardware-free!) snapshot API I know of. I’ve written articles (1, 2) on Windows storage technologies which go into these in more detail.

But one question that came up in my virtualization seminar made me realize that I forgot one key piece of Gates-tech: MPIO. See, Microsoft has also been bundling a free hardware-independent multipath I/O driver in server editions of Windows since 2003, but lots of folks haven’t gotten the memo. It’s good stuff: A generic driver with device-specific modules (DSMs) for different storage array and network types. MPIO handles transparent multi-path failover (for availability) and load balancing (for performance).

If you have iSCSI, you simply must try MPIO since Microsoft’s own free DSM supports about everything you need, and compatibility is required for logo support. And if you’re on Fibre Channel, you’re probably in luck, too, since most major vendors provide DSMs for their arrays (but some might not be free, I’m told).

So there you have it. Another excellent (and free!) Microsoft product that you (probably) never heard of, cutting out proprietary solutions (at least for Windows Server…)

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Microsoft: The A-Rod of Storage
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Although most folks haven’t heard about it, Simple SAN is Redmond’s way to force vendors to improve interoperability and ease of installation for networked SAN storage, whether it is Fibre Channel or iSCSI.

How does this help us? Simply, Microsoft forced participating vendors to pair up qualified hardware and an integrated driver set with easy documentation, including a one-page poster for installation. Think TiVo easy.

And it works. I went up to EqualLogic on Monday and set up an iSCSI SAN myself. It was easy to get the thing started without cracking the books. Sadly, though, their poster ended right where I needed help – configuring Microsoft’s iSCSI initiator software, which proved to be a frustratingly multi-click, multi-tab, search-everywhere-and-click-advanced operation. Claude Lorenson promises to make it easier in the next rev… If I had cracked the books, I would have had all the answers, though.

All in all was so easy I got silly and started trying to do stupid stuff with it. Which I did! Since EqualLogic lets you write to snapshots, I made one and added it to a dynamic disk group of regular volumes. This is highly dumb, but it was fun to see it work. It was also fun to watch Victory by Design stream from one array while migrating the volume to another… But now I’m digressing!

Go see for yourself. Check out a Simple SAN product and be amazed what you can do. Despite some frustration, I bet any Windows admin could have a SAN up and running in under an hour. Which was the whole point…

© sfoskett for Stephen Foskett, Pack Rat, 2007. | Microsoft’s best-kept secret
This post was categorized as Enterprise storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.