Note: This information was based on the “VMware Storage/SAN Compatibility Guide” and is being regularly updated. Please add comments here and I will add products and change and update listings as soon as they appear in the guide!

Where, Why, and What is VAAI?

I’ve previously discussed the fact that VMware’s excellent ESX hardware compatibility list (HCL) is so comprehensive than obscures basic facts about supported products. This is especially true for VAAI, since compatibility is only noted as a footnote in individual storage array listings. It does not help matters that not all VAAI plugins support all three capabilities.

Like my previous posts regarding FCoE CNA’s, SATA and PATA chipsets, and home/lab network cards, I’ll attempt to boil down the VMware ESX HCL into plain English. This data will also be part of my Storage for Virtual Servers seminar presentation, the first of which will be given on March 10 in Philadelphia.

I’m not going to spend too much time on “what is VAAI” in this post. Instead, I suggest you read the following blog posts and VMware’s excellent guide, “What’s New in VMware vSphereTM 4.1 — Storage“?

• What is VAAI, and how does it add spice to my life as a VMware admin?
• vStorage APIs for Array Integration aka VAAI
• If You Ever Needed Convincing About VAAI…
• VMware VAAI pros and cons and the hidden fourth primitive

The Three VAAI Primitives

You ought to read the updated Complete List of VMware VAAI Primitives since it’s much more thorough and informative!

In ESX 4.1, that vStorage API for Array Integration includes three basic capabilities or primitives:

1. Blocks zeroing is a communication method for thin provisioning capable storage arrays, allowing them to quickly zero out storage capacity for later reclamation.
2. Full copy commands the storage array to make a mirror or snapshot of data without any I/O on the server hardware.
3. Hardware assisted locking enables more granular control of shared storage resources in ESX clusters

In order to support VAAI, a storage array requires two things:

1. Hardware capable of supporting one or more of the three primitives listed above
2. A software plug in for ESX enabling communication and integration

VAAI Plug In Support

Creating a VAAI plug in is not a trivial task, and not all storage arrays are yet supported. I have heard grumbling from storage vendors that EMC (the storage vendor that owns VMware) has been given early access to VAAI information, allowing them to support this feature set before their competitors. However, this has not stopped a diverse set of other unrelated storage vendors from quickly producing and releasing effective and complete VAAI plugins.

As of this writing, there are 11 array-specific plugins and one general-purpose plug in available for ESX 4.1. EMC, NetApp, 3PAR (HP), HDS, FalconStor, Fujitsu, IBM, Dell (EqualLogic), and HP (LeftHand, P9000, P2000) have produced VAAI plugins supporting all three primitives. Additionally, a cloud in supporting the T10 blocks zeroing methods is available, enabling other arrays to support this one primitive. Note that the T10 primitive should support nearly any capable array, but not all have been tested and qualified for use with it.

VAAI Support Matrix

Note that similar OEM versions (for example, Fujitsu’s FibreCAT CLARiiONs, and the Gateway/Lenovo/Acer AMS line) are also supported the same as the manufacturer’s offerings. I’ve simplified and eliminated similar models (the Dell EqualLogic PS6000E, PS6000S, PS6000X, PS6000XV, and PS6000XVS are all listed simply as PS6000).

Updates:

• IBM recently added full VAAI for the XIV, SVC, and similar Storwize V7000. I’m sure they’re also working on complete VAAI plugins for the big DS8000 systems!
• EMC certified the new VNX line for VAAI (FC only for now) and the V-Max just gained iSCSI VAAI support.
• FalconStor added VAAI for NSS, enabling any storage array to be used.
• HP created VAAI plugins for the P9500 and MSA P2000 lines as well.

Stephen’s Stance

VAAI is an exciting new capability for VMware ESX, and demonstrates the enterprise readiness of vSphere 4.1. Although not all storage arrays are yet supported, the diverse assortment listed above should cover the majority of enterprise storage environments. I fully expect that the obvious holes will be filled in soon, and I look forward to updating this list when I hear news of those product releases. I also look forward to learning of additional capabilities added as VAAI primitives in the future!

The Exhaustive List

I am attempting to keep this list up to date. My authoritative source of information is the VMware Storage Compatibility Guide. This is the only source of information I will use, since only official and supported implementations belong in production. But I welcome pointers, suggestions, and referrals for updates!

This list is complete as of February 21, 2011

Full VAAI (All 3 primitives)

• Dell
  • EqualLogic
    • iSCSI (vmw_vaaip_eql)
      • Dell EqualLogic PS4000E (Dell EqualLogic PS4000X; Dell EqualLogic PS4000XV)
      • Dell EqualLogic PS5000E (Dell EqualLogic PS5000X; Dell EqualLogic PS5000XV)
      • Dell EqualLogic PS5500E
      • Dell EqualLogic PS6000E (Dell EqualLogic PS6000S; Dell EqualLogic PS6000V; Dell EqualLogic PS6000XV; Dell EqualLogic PS6000XVS)
      • Dell EqualLogic PS6010E (Dell EqualLogic PS6010S; Dell EqualLogic PS6010X; Dell EqualLogic PS6010XV; Dell EqualLogic PS6010XVS)
      • Dell EqualLogic PS6500E (Dell EqualLogic PS6500X)
      • Dell EqualLogic PS6510E (Dell EqualLogic PS6510X)
      • EqualLogic PS100E
      • EqualLogic PS200E
      • EqualLogic PS300E
      • EqualLogic PS3600X
      • EqualLogic PS3700X
      • EqualLogic PS3800XV
        • EqualLogic PS3900XV
      • EqualLogic PS400E
      • EqualLogic PS50E
      • EqualLogic PS70E
• EMC
  • FC (vmw_vaaip_cx)
    • EMC CLARiiON CX4-120 (EMC CLARiiON CX4-120 C8/C8X; Dell/EMC CLARiiON CX4-120; Fujitsu FibreCAT CX4-120)
    • EMC CLARiiON CX4-240 C8/C8X
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8/C8X; Dell/EMC CLARiiON CX4-480)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8/C8X; Dell/EMC CLARiiON CX4-960)
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC VNX5100
    • EMC VNX5300
    • EMC VNX5500
    • EMC VNX5700
    • EMC VNX7500
  • FC/iSCSI/FCoE (VMW_VAAI_SYMM)
    • EMC Symmetrix VMAX
  • iSCSI (vmw_vaaip_cx)
    • EMC Celerra CNS
    • EMC Celerra NS-120
    • EMC Celerra NS-480
    • EMC Celerra NS-960
    • EMC CLARiiON CX4-120 C8
    • EMC CLARiiON CX4-240 C8
    • EMC CLARiiON CX4-480 (EMC CLARiiON CX4-480 C8)
    • EMC CLARiiON CX4-960 (EMC CLARiiON CX4-960 C8)
• FalconStor
• FC (unknown)
• CDP Gateway
• CDP SA
• CDP VS
• CDPx Gateway
• IPStor Enterprise
• NSS Gateway
• NSS SA
• NSS VS
• Fujitsu
  • FC (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
  • iSCSI (fjt_vaaip_module)
    • Fujitsu Eternus DX410
    • Fujitsu Eternus DX440
    • Fujitsu Eternus DX8100
    • Fujitsu Eternus DX8400
    • Fujitsu Eternus DX8700
• Hitachi
  • FC (vmw_vaaip_hds)
    • Hitachi AMS 2100 (Acer AMS2100; Gateway AMS2100; HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (Acer AMS2300; Gateway AMS2300; HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Acer AS2040; Gateway GS2040
    • Hitachi AMS 2010
    • Hitachi BR1600/BR1600E/BR1600S
    • Hitachi Virtual Storage Platform (Hitachi VP9500; HP StorageWorks P9500)
    • Nihon-Unisys Sanarena 1910
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
  • iSCSI (vmw_vaaip_hds)
    • Hitachi AMS 2010
    • Hitachi AMS 2100 (HDS AMS 2100; Lenovo-HDS AMS2100)
    • Hitachi AMS 2300 (HDS AMS 2300; Lenovo-HDS AMS2300)
    • Hitachi AMS 2500 (HDS AMS 2500)
    • Hitachi BR1600E
    • Nihon-Unisys Sanarena 1930
    • Nihon-Unisys Sanarena 1970
    • Nihon-Unisys Sanarena 1990
    • Hitachi Virtual Storage Platform
• HP
  • 3PAR
    • FC (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
    • iSCSI (3PAR_vaaip_InServ)
      • InServ E200
      • InServ F-Class
      • InServ S400
      • InServ S800
      • T-Class
  • P9500
  • FC (hp-vaaip-p9000)
  • HP P9500
  • P2000
  • FC/iSCSI (hp-vaaip-p2000)
  • HP MSA P2000
  • LeftHand
    • iSCSI (vmw_vaaip_lhn)
      • HP LeftHand P4500
      • HP LeftHand P4000 VSA
      • HP LeftHand P4000sb
      • HP LeftHand P4300 (HP LeftHand P4300 G2)
      • HP LeftHand P4500 (HP LeftHand P4500 G2)
      • HP LeftHand P4800
      • HP ProLiant DL380
      • Dell 2950
      • IBM x3650
      • LeftHand NSM 160
      • LeftHand NSM 185
      • LeftHand NSM 2060 (LeftHand NSM 2060 G2)
      • LeftHand NSM 2120 (LeftHand NSM 2120 G2)
      • LeftHand NSM 260
      • LeftHand NSM 320
      • LeftHand NSM 326
      • LeftHand NSM 3650
      • LeftHand NSM 380
      • LeftHand NSM 4150
      • LeftHand VSA
• IBM
  • FC/iSCSI (IBM_VAAIP_MODULE)
    • IBM XIV
    • IBM SVC
    • IBM V7000
    • Fujitsu VS850
    • Actifio
• NetApp
  • FC (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • FCoE (VMW_VAAIP_NETAPP)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series
  • iSCSI (VMW_VAAIP_NETAPP)
    • NetApp N3000 Series
    • NetApp N5000 Series
    • NetApp N6000 Series
    • NetApp N7000 Series
    • NetApp FAS2000 Series (Fujitsu Eternus NR1000F Series Model F2040)
    • NetApp FAS3000 Series
    • NetApp FAS3100 Series (Fujitsu Eternus NR1000F Series Model F3160)
    • NetApp FAS3200 Series
    • NetApp FAS6000 Series
    • NetApp FAS6200 Series

T10 Block Zero Only

• Bull
  • FC (vmw_vaaip_t10)
    • Bull Optima2000
  • iSCSI (vmw_vaaip_t10)
    • Bull Optima2000c
    • Bull Optima2000i
• NEC
  • FC (vmw_vaaip_t10)
    • NEC iStorage D3-30
    • NEC iStorage D4-30
  • iSCSI (vmw_vaaip_t10)
    • NEC iStorage D3-30/D3-30i
    • NEC iStorage D4-30/D4-30i
• Fujitsu
  • iSCSI (vmw_vaaip_t10)
    • Fujitsu Eternus VS850
• IBM
  • iSCSI (vmw_vaaip_t10)
    • IBM Storwize V7000
    • IBM SVC

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware VAAI Storage Array Support in Plain English
This post was categorized as Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Update: If you want to do this using a keyboard shortcut instead of speech recognition, see my followup piece, How To Assign Keyboard Shortcuts To AppleScript and Automator Actions

Separate Clipboards

Each Mac maintains its own universal clipboard, and each application interacts with this in its own way. Most applications use similar commands (command-c, command-v, etc) and I use these many times an hour, moving text, addresses, pictures and the like between applications.

But this all breaks down when using two Macs at once. The clipboards are not shared across the network (though some third-party applications allow this), setting up a roadblock to productivity. Since I usually have a Screen Sharing session open, I usually use that application’s “Get Clipboard” and “Send Clipboard” function to move clips between Macs. But these are buried in the “Edit” menu of that one application, forcing me to switch to it, move the pointer to select “Edit”, move it again to select the correct item (“Get” or “Send”), reactivate the original application, and finally do what I want with the clip.

After performing this dance literally hundreds of times a day for months, I decided to try to automate it. First, I thought to bind a keyboard command shortcut to it in System Keyboard Preferences. But this failed miserably – keyboard shortcuts only work with the active application, and Screen Sharing sends all keystrokes through to the remote machine.

Then I thought of the Mac OS X Speech function. It allows an AppleScript script to be run from any application. If I could script Screen Sharing’s Edit menu, I’d have solved about 90% of my issues: I still couldn’t run the script from within a Screen Sharing session, but I could run it from anywhere else.

AppleScript and Screen Sharing

AppleScript is a clever scripting language, allowing most Mac OS X applications to be automated using readable commands. It reminds me somewhat of Microsoft Visual Basic, but is integrated with the operating system. Some applications allow extensive scripting, while others (like Skype) offer a simple command for API access. Then there are those like Screen Sharing that have no AppleScript documentation at all.

Mac OS X comes to the rescue here, allowing AppleScript access to all GUI functions in any application through the “Universal Access” Preferences pane, but these aren’t well-documented. I downloaded the trial version of UI Browser to discover the proper GUI string to access the “Send Clipboard” and “Edit Clipboard” functions in Screen Sharing. Amazingly, it works! I now have an AppleScript to trigger these commands.

But the script includes a statement activating Screen Sharing, so it leaves me there when I’m done rather than the application I was in to begin with. I located a method of determining the current running application (a surprisingly-complex one – where’s the simple command, Apple?) and added a statement reactivating it at the end. Voila! My script was complete!

I saved two scripts to Speech’s Speakable Items folder, one for “get” and another for “send”, and am now able to speak a command to copy items in and out of my remote computer’s clipboard. Excellent!

The Recipe

Here’s what you have to do to duplicate my success:

1. Select “Enable access for assistive devices” in the Universal Access pane of System Preferences
2. In the “Speech Recognition” section of the Speech pane of System Preferences, turn on “Speakable Items”
3. While you’re there, make sure “Global speakable items” is selected under “Commands”
4. Open AppleScript Editor and paste in the following script. Save it as “~/Library/Speech/Speakable Items/Screen get clip”

   set front_app to (path to frontmost application as Unicode text)
   set AppleScript's text item delimiters to ":"
   set front_app to front_app's text items
   set AppleScript's text item delimiters to {""} --> restore delimiters to default value
   set item_num to (count of front_app) - 1
   set app_name to item item_num of front_app
   set AppleScript's text item delimiters to "."
   set app_name to app_name's text items
   set AppleScript's text item delimiters to {""} --> restore delimiters to default value
   set MyApp to item 1 of app_name
   
   activate application "Screen Sharing"
   tell application "System Events"
   	tell application process "Screen Sharing"
   		click menu item "Get Clipboard" of menu 1 of menu bar item "Edit" of menu bar 1
   	end tell
   end tell
   
   activate application MyApp

5. Create another AppleScript containing the following content and save it as “~/Library/Speech/Speakable Items/Screen send clip”

   set front_app to (path to frontmost application as Unicode text)
   set AppleScript's text item delimiters to ":"
   set front_app to front_app's text items
   set AppleScript's text item delimiters to {""} --> restore delimiters to default value
   set item_num to (count of front_app) - 1
   set app_name to item item_num of front_app
   set AppleScript's text item delimiters to "."
   set app_name to app_name's text items
   set AppleScript's text item delimiters to {""} --> restore delimiters to default value
   set MyApp to item 1 of app_name
   
   activate application "Screen Sharing"
   tell application "System Events"
   	tell application process "Screen Sharing"
   		click menu item "Send Clipboard" of menu 1 of menu bar item "Edit" of menu bar 1
   	end tell
   end tell
   
   activate application MyApp

6. Enjoy!

This may give you ideas for other AppleScript items. I know it sparked my creativity! Here’s a bonus AppleScript to set my Skype status to “Do Not Disturb”:

tell application "Skype"
	send command "SET USERSTATUS DND" script name "Set Skype Status Do Not Disturb"
end tell

Since Skype has an AppleScript bridge (“send command”) to their API, there’s no need to mess with application switching! I wish more applications exposed their innards like this!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How To Automate “Get/Send Clipboard” in Mac OS X Screen Sharing
This post was categorized as Apple, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

This new Simple Cloud API is a godsend to PHP developers looking for a simple object-oriented way to access cloud file, document, and queueing services. I’m not much of a programmer, but I use PHP all the time and am looking forward to adding simple calls to my code to store and access data wherever it lives, in S3, Azure, Cloud Files, and the Nirvanix SDN. Most of my PHP work goes into the WordPress blogging platform, and I’m already imagining cool new plugins that this simple interface makes possible.

But the most telling aspect of this announcement is not what it is but what it is not: Rather than meet with a committee or ask for approval, the fine folks at Zend and others in the PHP community just whipped up the whole thing and launched it. The fact that Amazon, Rackspace, Microsoft, and Nirvanix offer fully-documented APIs and leverage open protocols like IP and HTTP made the creation of this interface a straightforward matter of coding. This may not meet the definitions of open and free put forth by the open source and free software community in the last two decades, but it certainly meets the spirit of openness and freedom demanded by today’s web developers and end users. And isn’t that what it’s all about?

Of course, the Zend Simple Cloud API is by no means unique. Indeed, CloudLoop, which launched earlier this month, offers a similar interface to public cloud storage for Java programmers and even included a UNIX-like command line interface. Like Zend, CloudLoop was able to dispense with the politics and implement their open interface using programming skills, API specifications, and technical support forums. It is gratifying to see cloud vendors actively foster and support this kind of creativity, even creating and publishing their own frameworks for languages like Ruby and Python.

These practical interfaces really deliver the goods:

• Developers and programmers get simple, instant access to all of the major public cloud storage service providers.
• End users of these applications will get a choice of cloud services as well as easy portability if they decide to switch.

This is true freedom, available today, and enabled without contentious political battles and long waits for standardization. Another telling aspect: No one asked for permission from the service providers involved, they just went ahead and did it. Now that’s the Internet I love! Who needs standards?

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Zend Simple Cloud API = Freedom!
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

But cloud services are different. Seriously! Cloud services don’t need standards because:

1. Cloud services are still rapidly evolving – No one knows how they will look in a year, let alone a decade, and a premature standard will be worthless. Similarly, it’s not at all clear what use cases will eventually win out, and usage should drive interfaces, not the other way around.
2. Cloud services are many and varied – It’s incredibly hard to come up with a reasonably-complete standard programming API or management platform when each vendor’s offering is radically different. Standards must follow the 80/20 rule, but today’s cloud offerings are only about 20% similar.
3. (Real) cloud systems are open already – The whole point of the public cloud is to leverage existing open standards for access (IP/HTTP) and any worthwhile service already has a freely-usable REST-like API. Cloud services are engineered to be programmable and open, so the only lock-in is in how you use the cloud.

We can’t even agree on terminology at this point. Is data storage as a service DaaS (as SNIA says) or STaaS (as NetApp says)? How do you define public, private, and hybrid cloud? And what is cloud anyway? Cloud computing is not a war, it’s a fantastically exciting race to deliver value!

Open for Business

I’d like to return for a moment to that last point: The key element I’ve seen in most interesting cloudy products is programmability. Service providers publish API documents outlining the inputs, processing, and outputs for their systems and developers and end users create applications that leverage these. The best of these APIs use the concept of REST, delivering services through extremely simple and self-contained HTTP calls. This barely even rises to the level of software coding (and thus isn’t a true API) and is the hallmark of the cloud.

These systems are wide open: You can explore their interfaces, discovering new ways to use the them that were never intended. The same process accompanies all Internet systems, from RSS and Atom to Yahoo Finance. Just as one can rapidly migrate from Yahoo to Google by substituting a few URLs and parameters, so too can one move between cloud platforms.

Note that certain cloud systems lend themselves more to this kind of mobility. Once cannot move virtual machines from Amazon EC2 to Rackspace or Terremark because the underlying hypervisor technology is different. But even here companies like RightScale are stepping in to enable mobility.

When it comes to cloud storage services, the major players’ interfaces are open enough that migrating data in and out is simply a matter of performance: Read from this one, write to that one, and wait until the process is done. I am not a programmer and yet I was able to port an application from S3 to Nirvanix in just a few hours using only the respective API documentation. Interfaces like CloudLoop can also be leveraged to ease the movement of data.

Standards When?

Cloud services will eventually settle down and be standardized. I expect a workable cross-platform API for RESTful cloud storage within 24 months, for example. And one expects that the management of cloud compute instances will pass through a consistent and stable interface in that same timeframe. But these will develop as a natural part of the evolution of the systems themselves, not through some artificial “build it and they will come” standardization process.

There is nothing wrong with big companies sending their representatives to SNIA and DMTF meetings to talk about standardization. In fact, this is a great way to discuss ideas and begin to orient the industry. But the time for standards has not yet come, and users of cloud services have no need to wait for them. In fact, waiting for a standard will just prolong the maturation of cloud services, since real-world applications are the external pressure that forces evolutionary selection. Amazon would never have created their virtual private cloud (VPC) capability without customer input, and they will never perfect this capability if they rely only on pundits, bloggers, and product marketers.

Even when standards do appear, they will not eliminate per-solution APIs. Cloud service providers will continue to explore new concepts, and these will appear first in “proprietary” interfaces. Perhaps they will use entirely unique calls, or perhaps they will leverage reserved or unassigned sections of the standard, but innovation will continue. Witness the radical changes in HTML versions to date, the additions to CSS, and the wide world of browser plugins.

So we don’t need cloud standards yet. They will come, whether artificially pushed by committees or evolving through use, but only useful standards will survive. Isn’t that just how it should be?

Note: I am employed by Nirvanix, a cloud managed storage service provider, providing independent cloud strategy advice as Director of Consulting. Although this article was not created for my employer and is not intended to reflect their views, my perceptions are obviously colored by my daily work.

© sfoskett for Stephen Foskett, Pack Rat, 2009. | We Don’t Need Cloud Standards (Yet)
This post was categorized as Computer History, Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Check out these detailed posts on cloud storage I wrote for my other blog, Enterprise Storage Strategies:

• Cloud Storage, Storage in the Cloud, and Cloudy Storage Systems
• What Makes Cloud Storage Different from Traditional SAN and NAS?

Eventually, it dawned on me: There’s a big difference between real cloudstuff and plain old stuff in the cloud!

Lots of cloud computing offerings are startlingly conventional. They’re plain-jane IT infrastructure just like we all have today: Virtual servers, storage, and databases. Strip away the management API and self-service model and Amazon EC2 and EBS looks an awful lot like the Xen-based virtual server infrastructure you might find at any old IT shop. The same goes for Rackspace’s Mosso Cloud Servers: They’re extremely similar to Rackspace’s Slicehost virtual private servers!

This isn’t all bad, of course. As I discussed with EMC’s Barry Burke last week, you can theoretically run your ERP application on EC2 without major gyrations. Try that with Microsoft Azure or Google App Engine! And the management layers, especially those from companies like RightScale, turn these run-of-the-mill parts into something really extraordinary! VMware’s vCloud concept really hammers home this evolution-not-revolution mindset.

Real cloudstuff is completely different. Comparing a Xen instance running Linux on some disk (a-la EC2) to a programmable platform like Azure is problematic. Just about the only thing they have in common (apart from the cloud name) is the fact that they’re hosted on multi-tenant servers and offered to the public on a pay-per-usage model. Cloudstuff is the IT revolution that application developers have dreamed of!

Of course, the problem with whole-cloth reinvention is that it’s slow to take hold. Although net-new apps can be built to take advantage of full-on cloud infrastructure today, it will literally be a decade before the corporate IT applications we all rely on will run there. The early adopters will be companies like Microsoft and Google, who have a vested interest in seeing the concept succeed and the development muscle to make it happen.

Then there are the bridges between today’s world and this cloudstuff future. Consider applications like Nirvanix CloudNAS and Jungle Disk: They hide the complexity of API-driven cloud storage behind the familiar face of file server or backup application. Once the data is loaded, cloud-aware applications can access it. This is where the magic happens!

© sfoskett for Stephen Foskett, Pack Rat, 2009. | CloudStuff Versus Stuff in the Cloud
This post was categorized as Computer History, Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Sun is getting into the cloud services business, but which market is their target?

While the bulk of Sun-related news this week relates to reported talks of a buyout by IBM, the company took a break from negotiations to introduce their own cloud computing and storage infrastructure, challenging Amazon, Google, Rackspace, and perhaps VMware, Microsoft, and Nirvanix.

Sun is leveraging the assets they acquired from Q-layer earlier this year on top of OpenSolaris, MySQL, ZFS, and just about everything else in their arsenal to offer their own virtual data center (VDC) strategy. The Sun Cloud will be a private (inside the firewall) environment offering mobility of virtual machines. Q-layer had partnerships with both VMware and Microsoft and functioned with Windows, OpenSolaris, and Linux, suggesting that this will be quite a full-featured offering. Suddenly Sun’s free OpenSolaris offering on Amazon EC2 makes a lot more sense – it provides a gateway to take virtual computing business from the Bezos team!

One very nifty angle Sun is taking is enabling VirtualBox system images to be saved to (and presumably run in) their cloud. I wonder about monetization, since VirtualBox is more of a desktop virtualization system than VMware ESX and Microsoft Hyper-V, but the prospect of clicking “Upload to Cloud” is intriguing! OpenOffice will also allow cloud storage, a foil to Google Apps.

The Sun Cloud also includes a managed storage service. Sun apparently has three storage protocols: A proprietary Sun Cloud Storage API, WebDAV, and an object API likely leveraging Amazon’s AWS. The company claims that they are API-compatible with AWS, allowing applications written with Amazon in mind to be easily ported to their cloud storage service.

One key point to consider with all of this cloud talk, however, is how prepared each company is to support enterprise computing needs. Long-term viability depends on paying customers, and only the largest systems can attract enough end-user nickels and dimes to survive. Enterprise solutions are where the real money is, and questions remain about how prepared companies like Amazon, Google, and Rackspace are to support the needs of corporate users.

There are really three cloud markets: Shared clouds for small developers and enterprise customers and private cloud systems. Amazon was strong from the start with the little guys, Web 2.0 startups and end-user services like Jungle Disk, and Rackspace/Mosso and Google are challenging them in this space. Sun’s focus on AWS compatibility and VirtualBox suggests that they plan to play in this sandbox.

But the enterprise cloud is another matter entirely. Nirvanix stands strong in shared managed storage services, racking up win after win with big customers. EMC, VMware, Microsoft and others are positioning themselves as private alternatives in this space. Will Sun try to compete here, too? They are certainly talking about private clouds and the virtual data center, but there is a serious risk that they will lose focus trying to take on too many roles, and enterprise users won’t tolerate poor pre- and post-sales support! 

© sfoskett for Stephen Foskett, Pack Rat, 2009. | Sun Launches Their Own Cloud, But For Which Market?
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

First, let’s clarify just which products and capabilities Symantec is offering here:

• Veritas Storage Foundation version 5.0MP3 for Unix/Linux includes SmartMove and the Thin Reclamation API
• Veritas Storage Foundation for Windows 5.0 only includes SmartMove at this point, but it will be updated to include Thin Reclamation at some point in the coming year

Although there is no real information on Symantec’s web site about all this yet, Symantec’s director of Storage Management and High Availability, Sean Derrington, assures me that their software is available now. Although no compatible arrays are in end-user hands, 3PAR will update their T-Class firmware to support the API shortly, and HDS and HP are on the way as well.

Thin Aware Software

Next, contrary to what I inferred from the announcement, there is no native thin provisioning capability in the file system or volume manager. So the first item in my list is right out. However, the volume manager is now “thin aware”, which means that it will communicate up to the file system and down to the array to coordinate more effective use of space.

When the volume manager is used with Veritas File System (VxFS) on UNIX or NTFS on Windows Server 2003 or 2008, it will automatically keep track of deleted files and will pass this information down the stack to the array. This is a major piece of functionality to add, especially to NTFS, “hole punching” (like NetApp) to maximize thin provisioning.

The Storage Foundation tools have also been updated to properly report on thin provisioned volumes. For example, the following screenshot shows three disk devices where encl1 supports thin reclamation and encl0 does not.

# vxdisk list
DEVICE        TYPE   DISK          GROUP         STATUS
encl0_0       auto   encl0_0       mydg online   thin
encl1_0       auto   encl1_0       mydg online   thinrclm
encl1_1       auto   ecnl1_1       mydg online   thinrclm

Thin Reclamation API

The Veritas Thin Reclamation API allows the Storage Foundation volume manager and file systems to communicate with thin-capable arrays when data is deleted on thin-ified LUNs, maintaining their thin-ness as you go. When a file is deleted, the file system will communicate to the volume manager that that space is no longer needed. When the server administrator runs the “vxdisk reclaim” or “fsadm –R” commands, the volume manager will communicate this information to the array (using SCSI commands) that any vacated disk blocks can now be reclaimed. Symantec expects folks to set up a cron job to reclaim space, or perhaps just run it when they see the need.

This is brilliant stuff, and ought to make thin provisioning shine in terms of array utilization. In an environment of thin-enabled Veritas volumes and supported storage arrays, the amount of space used on an array will be awfully close to the amount of space used in the file systems. This is a massive win - a capacity gain of on the order of 50%-70% in an average environment!

For more on this topic, see my recent post on storage utilization

If the storage array fully supports Symantec’s API, the tools will also report physically allocated storage behind thin and thin_reclaim devices.

# vxdisk –o thin list
DANAME        DISK SIZE(Mb)        PHYS_ALLOC(Mb)       DISK GROUP TYPE
encl0_0       2000                 50 mydg              thin
encl1_0       200                  50 mydg              thinrclm
encl1_1       500                  500 mydg             thinrclm

SmartMove

SmartMove is Symantec’s new capability for online migration from “thick” to thin LUNs. It is included in Storage Foundation for Unix/Linux and Windows and works with any thin storage array, not just those that support the API. This is basically a tweak to the old storage migration support we have all known and relied on in Veritas Storage Foundation for over a decade, except that it’s smart enough to not request blocks that it won’t use. One could theoretically “SmartMove” a volume regularly to reclaim space without using the API at all, but those commands are sure a lot simpler.

Note that SmartMove speeds up migration too, even for thick volumes! When you use a SmartMove-enabled version of Storage Foundation to move a volume, it will only send the blocks that have changed over the wire. This reminds me a little of VMware’s new I/O deduplication capability talked about at VMworld, but it’s focused only on migrations, not other I/O situations.

For more on this topic, see my recent post on VMware vStorage

The Plot Thickens

So I was wrong about one item, but the other two remain true. Is Symantec’s new capability a winner? I give it a silver medal – it’s good stuff, but some issues remain.

1. My primary concern remains – thin provisioning does nothing to address the lack of storage management that is so prevalent. It enables greater utilization of capacity, but does nothing to control how that capacity is used. This isn’t a beef with Symantec’s Veritas Storage Foundation or 3PAR or HDS or EMC or anyone in the thin industry, really. Instead, it is a wake-up call to all of the storage organizations out there who have filesystems full of uncontrolled junk!
2. My second concern is the lack of capacity management. Thin provisioning is a lie, promising more capacity than is available. This might be acceptable in certain controlled circumstances like operating system or application volumes, but telling end users that they have plenty of available space is a recipe for disaster. Storage use is like air – it expands to fill all available volume. Without capacity management, your thin volumes will be “overdrawn” and your storage “account” will be bankrupt.
3. Then there is the issue of proprietary APIs versus standards. Let me say right away that I always support standards over proprietary technology. But, at the same time, given the choice between nothing and something, I’ll take the proprietary API. Thin provisioning is a good idea with poor implementation. This API helps to make it useful in the real world, and having a market leader like Symantec behind it makes it all the more relevant. I certainly hope the entire storage industry will come up with a standard thin API, and when that happens I hope Symantec will support it. Until then, at least we have something.

I will be writing more about thin provisioning in the coming weeks. Until then, I continue to applaud Symantec, 3PAR, HDS, and HP for their work in making this technology somewhat more practical. Now how about VMware, Microsoft, Sun, and the Linux guys get some thin technology going, too?

See my posts on Gestalt IT for similar enterprise IT infrastructure commentary

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Symantec’s Thin API: The Plot Thickens
This post was categorized as Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.