Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
This post was categorized as Enterprise storage, Everything, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Learn to drive a Ford Model T and you will be amazed how automobile controls have progressed! It looks similar but it's totally different from today's cars.

“The nice thing about standards is that you have so many to choose from.” Andrew Tanenbaum’s statement may be a cynical and dismissive, but it’s not far from the mark. Indeed, there are a great many standards, perhaps as many as there are efforts to standardize the world around us! But what good are standards, really?

The 3 Standard Types

In technical fields, standardization is a process of establishing a specification, definition, or procedure that is generally applicable. In other words, a standard is the exact opposite of a one-off or proprietary item.

End-users and vendors often clamor for standardization, though not usually for the same reasons:

• End users like standards because they promote options and tend to drive down costs
• Incumbent vendors like standards that give them control over the market or competitors, while challengers prefer “open standards” that allow them entry

Not all standards are created equal, however. Some are designed to be open and free to use, while others simply fall into widespread use. Some are designed by committee, while others are driven by a dominant player in the market. Generally, standards fall into one or more of the following categories:

• De facto standards emerge “accidentally” as they become used more and more broadly. In many cases, companies are hesitant for their developments to become de facto standards, since they may lose control of the market and usage of their products.
• De jure standards are legally binding requirements from contracts, laws, or regulations. These are quite rare, and often adopted only when absolutely required to ensure safety or avoid major market upheaval.
• Other standards are made available on a voluntary basis, in hopes that they will be used. Whether designed by a committee or a single entity, voluntary standards usually serve to encourage market development.

Whose Standard Is It?

When considering one standard or another, it’s important to keep in mind it’s origin. Sony’s Memory Stick, the Blu-Ray disc, the Apple dock connector, the Microsoft Windows API, and so many more were all developed to lock in licensing and product revenue. Although it is beneficial to consumers to have standard camera media, multimedia discs, and such, these were not developed solely with the interests of consumers in mind.

Automobiles present an interesting case in standardization. It may come as a surprise to the uninitiated, but nearly every part of the car is proprietary, right down to the control mechanisms we take for granted while driving. Certain elements (seatbelts, windshield wipers, and the gasoline fill valve) are indeed de jure standards, but most everything else is subject to the whims of each manufacturer. Think of how difficult it is to operate the air conditioning or set the cruise control in a rental car. Then go to the auto parts store and see just how many different air filters they stock!

Consumers generally benefit when broadly accepted standards emerge, regardless of the origin. The Edison screw base on a lightbulb, for example, was developed to encourage a market for electric light fixtures but also to secure licensing revenue for the Edison company. In contrast, the “flash shoe” found the top most high-end cameras developed accidentally and incrementally over the last century. Both are now de facto standards out of control of their originators, but despite major shortcomings the value of interchangeability has made them commonplace.

Stephen’s Stance

I certainly benefit from standardization of the world around me, and I welcome interoperability and interchangeability as well as the price and product selection advantages. But I am not blithely focused on standardization above all else. I will happily use a proprietary solution if the alternative is inelegant, ineffective, or insufficient.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | The Myths of Standardization
This post was categorized as Apple, Computer History, Features, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

The great battle of enterprise storage is on!

The time has come to take sides on the core question of storage for virtual servers: Do you want storage intelligence to live in the hypervisor or the array? Most administrators are already lining up on one side or the other, unintentionally casting their vote while the rest flounder. But the storage industry must wake up and embrace the divide.

Hypervisor Huggers Unite!

Do you "heart" virtualization?

VMware’s vSphere dominates the world of enterprise server virtualization and has become the core element of the modern open systems datacenter. Microsoft recognizes this but has been unable to field a competitive hypervisor ecosystem for the virtual datacenter. Today, vSphere is the state of the art and nowhere is this more apparent than in storage.

In just a few years, VMware has delivered and updated a host of advanced storage functionality, from provisioning to migration and load balancing to backup and data protection. vSphere 5 includes an advanced and scalable storage virtualization layer, delivering everything a datacenter needs. VMFS sculpts basic block storage into a shared resource for virtual machines, with snapshots, policy-based layout and movement, and flexible allocation and thin provisioning.

Most VMware administrators are “server guys” and relish these features. They have never experienced an automated “storage service” like this, and the enterprise storage world has never been able to construct anything remotely as flexible, user-friendly, and functional. And Hypervisor Huggers don’t need complex enterprise storage arrays to do it: They can use basic iSCSI or Fibre Channel devices to provide performance and capacity and let VMware do the rest!

Storage DRS is exemplary of the new virtual datacenter world. Introduced in vSphere 5 (and restricted to the pricey Enterprise Plus license), Storage DRS uses the core technology of Storage vMotion to dynamically balance I/O and capacity across a diverse pool of storage. Storage DRS even uses Policy-Driven Storage and VASA to enforce tiered storage and data placement strategy. This kind of virtualization has been a “holy grail” quest for the enterprise storage industry, but they’ve never delivered on their promises.

Cheers for Storage Stalwarts!

Do you wish all this virtualization nonsense would just go away?

But not every IT environment wants be 100% vSphere focused, and many aren’t convinced that dumb storage is the smartest place for their data. These Storage Stalwarts want smarter and better-integrated storage arrays, and VMware is innovating here as well.

VMware’s Storage APIs for Array Integration (VAAI) technology is squarely aimed at this market. VAAI allows vSphere to hand off heavy storage operations to the high-end storage arrays from the major players. It works transparently, too, coordinating cloning without the kind of scripting and hair-pulling that used to require. VAAI in vSphere 4.1 also includes block zeroing support and something called “atomic test and set” which we’ll get to in a moment. Microsoft announced their own cloning integration, ODX, but it won’t ship until Windows Server 8 appears sometime next year.

But cloning is just the tip of the iceberg when it comes to Hypervisor-to-array integration. The rising army of NAS users have seen the glory of more-complete array integration for a while, and they’re not quiet about it. They love that VMware’s NFS protocol support makes storage “disappear” in vCenter, becoming just another resource with integrated thin provisioning and flexible allocation and movement.

VMware is moving aggressively to please their Storage Stalwarts, adding more VAAI support for block and file in vSphere 5. But, as the company laid out at VMworld 2011, neither access method is ideal for virtual servers. So VMware has been pushing the enterprise array vendors for ever-greater integration. They see a future where a VAAI-based protocol enables arrays to de-multiplex I/O streams from the hypervisor and intelligently handle per-VM data.

Stephen’s Stance

You can spot a Hypervisor Hugger by their big LUNs: They would rather treat storage as a bulk commodity, and array vendors should be lining up to get their business. Storage Stalwarts will jump on each new VMware innovation, finally making use of the capabilities they have spent over a decade paying for but not utilizing. The only untenable stance is trying to keep a foot in both worlds: It’s foolish to buy an enterprise array and use it as bulk storage!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Are You a Hypervisor Hugger or a Storage Stalwart?
This post was categorized as Computer History, Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Microsoft MVP

Microsoft began recognizing “the best and brightest from technology communities around the world” with their Most Valuable Professional (MVP) award way back in the days of CompuServe. This program serves two purposes: It recognizes those who have contributed to the Microsoft community over the previous year, and bring them in contact with the company in the coming year.

Many outside the program believe that Microsoft MVPs get special access, privileges, and information about upcoming products, but this is not true. At least in my area of File System Storage, very little is shared about upcoming products or technology that is still in the lab.

Instead, the MVP allows Microsoft product managers to directly interact with a group of people who are very interested in their particular area of the company. Probably the highest profile interaction occurs during the annual MVP Summit in Redmond, WA, where MVPs prowl the campus and attend meeting after meeting discussing various products and technologies with Microsoft employees.

I first received the Microsoft MVP award in 2008, after a series of articles I wrote focusing on various aspects of Microsoft’s storage technology development. In the years previous, as a writer for Storage Magazine, a speaker at Storage Decisions, and other venues, I had extolled the virtues of the Microsoft iSCSI initiator, the VSS system, the MPIO driver, and the volume manager for Windows. I also suggested a number of improvements to the systems, and the Microsoft product group reached out with the MVP award to continue this dialogue.

In the years since, I have had many wonderful interactions with Microsoft product managers. But I have remained critical of the company in many areas, especially when they fall short of the goal. I am disappointed to see little development of Fibre Channel and FCoE support in Microsoft Windows, and the cancellation of Windows home server with its clever post-RAID storage layer. I would like to see Microsoft match VMware’s incredible pace of development, and feel that Hyper-V is at risk of losing the hypervisor war. And I am skeptical about many of Microsoft’s other facets, from Windows Phone to Bing.

Happily, none of this criticism has caused strife in my relationship with Redmond. To their credit, Microsoft employees seem genuinely interested in improving their products and doing the right thing in terms of product and technology decisions. They didn’t even mind (too much) when I showed up on campus with my iPhone and MacBook Pro!

VMware vExpert

The VMware vExpert award is obviously patterned on the Microsoft MVP program, but VMware deserves credit for going in their own direction. vExperts are not expected to be supreme vessels of product and technical knowledge, but are instead recognized for contributing to the greater good of the virtualization community.

VMware vExperts are occasionally briefed on forthcoming product announcements or given access to beta software for review and feedback. VMware encourages active dialogue between vExperts, the virtualization community, and VMware employees. The atmosphere is one of open communication and assistance, and there is no clubby “we’re better than them” feeling. This is a good thing, since many non-vExperts communicate and contribute at a high-level!

It seems that VMware is still figuring out how to deal with, and benefit from, the vExpert community. The program is developing nicely, and I expect it to continue to improve in the coming year. Although it has been around much longer, the Microsoft MVP program is similarly in a constant state of flux, with malcontents and fan boys both inside and outside the company and program. I guess this is just how programs like this operate!

Community Support

I began my career as a systems administrator, before switching to consulting and now mainly writing and speaking on technical topics for living. Although both the VMware and Microsoft award recognize my contribution to the community, the truth is that I could not make any contribution without the community supporting me in return. In fact, none of my vaunted contributions come from me alone.

One reason that I’m willing to speak at events like Storage Decisions and my Storage for Virtual Environments seminar series is the incredible amount of information that comes back to me from the audience. I like to joke that I will repackage any feedback as my own thoughts in the future, but this isn’t all that far from the truth. Every time an active discussion starts at these events, I am learning just as much as the audience members. And I do, in fact, incorporate this sort of feedback into future events and publications.

If I have a talent worth recognizing, it is my compulsion to collect and share information about technical topics. When I discover a new area of technology, I love to go out and collect information and try to translate that into educational materials for others. This constantly happens when I am preparing seminar presentations, technical articles, and even blog posts. And I suppose this is the real reason that Microsoft and VMware are recognizing me.

Stephen’s Stance

I sincerely appreciate receiving these awards, and thank Microsoft and VMware for their faith in me. I also sincerely appreciate the contribution and feedback I get from the technical community around me, and thank you just as much for these awards. I would not have received them without the readers of my blog, the attendees at my seminars, and the vast community that I interact with on a daily basis. Thank you!

© sfoskett for Stephen Foskett, Pack Rat, 2011. | I’m a Microsoft MVP and VMware vExpert for 2011!
This post was categorized as Enterprise storage, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

VMware's PSA is awash in abbreviations and options

I am often questioned during my Storage for Virtual Environments seminar presentations about VMware’s Pluggable Storage Architecture (PSA). This system is fairly straightforward and concept: VMware provides native multipathing support for a variety of storage arrays, and allows third parties to substitute their own plug-ins at various points in the stack. But the profusion of acronyms and third-party options makes it difficult for end-users to figure out what is going on. In an effort to help, I present here another entry in my “VMware storage features in plain English” series.

Note: I am more of a storage guy than a virtualization expert. I consider myself one of those end-users who have had trouble figuring out what’s going on with PSA specifically, in VMware storage features in general. I welcome comments and suggestions for corrections or improvements to this and all of my articles. Thanks for your help!

Introducing Pluggable Storage Architecture (PSA)

Pluggable storage architecture was one of the major enhancements introduced in vSphere 4. Functionally similar to Microsoft’s MPIO stack for Windows, PSA includes native multipathing support and allows vendors to plug in their own advanced features.

I find the VMware diagram confusing. Is mine more or less accurate and readable?

The ESX kernel (VMkernel) walks down through three layers when communicating with storage:

1. In the top layer, VMware native NMP or third-party MPP software decides which SATP to use, or whether to use the native interface. MASK_PATH also operates at this layer.
2. The SATP layer includes native generic path selection (active/active, active/passive), standard ALUA, as well as allowing third-party plugins (SATP) to override its behavior. The SATP monitors these paths, reports changes, and initiates fail-over on the array as needed.
3. At the PSP layer, software decides which physical channel to use for I/O requests.

There are three types of PSA plugins for vSphere 4:

1. Storage Array Type Plug-In (SATP)
2. Path Selection Plug-in (PSP)
3. A complete third-party multipathing software stack (MPP)

As is the case with VAAI, VMware includes a number of third-party plug-ins in the ESXi install. Users can simply activate many of these according to their needs, though some require additional fees and licensing.

Storage Array Type Plug-in (SATP) List

Storage Array Type Plug-Ins (SATPs) to the VMware Pluggable Storage Architecture multipathing solution for the specific characteristics of the storage array. This is very important, since each storage array design differs substantially in detail and support, especially when it comes to load-balancing and failover between controllers, ports, and paths. So it is critical for VMware to have developed a standard interface to communicate with arrays.

SATPs allow load balancing across multiple paths, intelligent path selection, and over troubled conditions such as “chatter”, when passed rapidly fail back and forth between controllers.

The SATP has critical tasks to perform in the PSA stack:

1. Decide which method of communication to use with the storage (PSA or native)
2. Monitor the health of the physical I/O channels or paths
3. Report any changes in the state of the paths up the stack
4. Perform actions required to fail over storage between controllers on the array

VMware vSphere includes a variety of generic plugins for storage arrays. I’ve identified the following:

• VMW_SATP_LOCAL – Local SATP for direct-attached devices
• VMW_SATP_DEFAULT_AA – Generic for active/active arrays
• VMW_SATP_DEFAULT_AP – Generic for active/passive arrays
• VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arrays

Although I have sometimes seen other SATP plug-ins mentioned, the following plug-ins are all that are listed in the VMware ESX Hardware Compatibility List.

• VMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGI
• VMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)
• VMW_SATP_CX – EMC/Dell CLARiiON  and Celerra (also VMW_SATP_ALUA_CX)
• VMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, Invista
• VMW_SATP_INV – EMC Invista and VPLEX
• VMW_SATP_EQL – Dell EqualLogic systems

EMC PowerPath and HDS HDLM also support a variety of storage arrays, but I would classify these as full MPP replacements for PSA, rather than SATP plug-ins.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp satp list

Path selection plug-in (PSP) List

In contrast to the diversity of VAAI and SATP plug-ins, the universe of path selection plug-ins is fairly small. Most storage arrays are supported with either Most Recently Used (MRU) or Fixed path selection approaches. Many also support Round Robin (RR) path selection. The only vendor with a specific PSP that is not also part of a full MPP (like EMC PowerPath or HDS HDLM) is Dell, which offers a special routed path selection plug-in for the EqualLogic iSCSI arrays.

• VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arrays
• VMW_PSP_FIXED – Fixed – Supports hundreds of storage arrays
• VMW_PSP_RR – Round-Robin – Supports dozens of storage arrays
• DELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arrays

As mentioned, EMC PowerPath also offers path selection as a plug-in in addition to the full MPP stack. Many other vendors offer unique path selection plug-ins, over 100 in total, but these are not specifically called out in the VMware HCL apart from their existence. I would love to learn more about them, however.

You can see which SATP plug-ins are available using the following esxcli command:

esxcli nmp psp list

Tell Me More About PSA!

As mentioned above, I am by no means an expert in VMware Pluggable Storage Architecture. Rather, I am interested in learning more and passing on this knowledge to others. I welcome your comments and feedback, and especially your corrections to the information presented here. I will try to keep this page updated as new versions of vSphere are introduced and as I learn more about this technology. Thank you for your help and understanding!

Further Reading:

• What’s New in VMware vSphere 4: Storage
• Next-Generation ESX Storage: A Pluggable Core Storage Architecture
• Storage Changes in the VMware vSphere 4 Family
• Pluggable Storage Architecture, exploring the next version of ESX/vCenter
• Vmware vSphere 4.1 PSA (Pluggable Storage Architecture) Understanding
• Explain the Pluggable Storage Architecture (PSA) Layout

© sfoskett for Stephen Foskett, Pack Rat, 2011. | VMware PSP and SATP in Plain English
This post was categorized as Enterprise storage, Everything, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Dragon Dictate might be the best dictation option for Mac users, but it's not very good at that

For the last few months, I’ve been experimenting with dictation software on the Mac. Previously, I had used the built-in dictation software in Windows Vista and Windows 7, but it was annoying to launch a virtual machine every time I wanted to dictate something. On the Mac and purchase an (expensive) copy of Dragon Dictate for Mac. So far, my experience has been less than positive.

What I Was Looking For

I use dictation software for dictation, not control my computer. I was looking for a product that would allow me to speak naturally to my computer as a way to accelerate my writing. See, I am a professional writer and regularly churn out thousands of words per week. Although I can type very fast, I find that dictation allows me to sit back and collect my thoughts without being distracted by the keyboard.

It is critical for me that the software I select recognizes my words consistently and does not require a great deal of post-speech editing. In other words, I want to be able to carefully talk through an entire piece without going back and doing word by word checking after the fact.

Since I write on technical topics (I specialize in enterprise data storage), it is critical that the dictation software is flexible in allowing bizarre spelling and capitalization combinations. It seems that every enterprise product and company has some strange combination of capital letters and made-up words!

Dragon Dictates (Sometimes)

Dragon Dictate has proven frustrating in regular use, especially for a software package that cost nearly $200. Although it is quite adept at recognizing flowing speech, it utterly fails to be useful in regular applications and workflows.

Adding new words to the Dragon dictionary is frustratingly complicated, requiring nearly a dozen steps with keyboard, mouse, and the GUI. Why can’t an end-user dictate a new word? Alas, this seems to be impossible. Spelled (or even typed) words don’t appear to be added to the dictionary and will be mis-recognized even in the same document.

Dragon stubbornly refuses to learn certain words, as well. I can’t figure out how to tell it to spell the word “minutes”, for example, rather than the abbreviation “min.” though there is such an option in Dragon’s PC product. It is frustratingly inconsistent at recognizing common punctuation marks and commands, often typing “exhalation point” or “go to and” rather than “!” or moving the cursor to the end of the line. I can’t figure out how to tell it to spell my name without a “V” and it even mis-recognizes its own name as “DragonDictate” (no space).

But by far the most frustrating aspect of Dragon Dictate is its incompatibility with just about every native application I use. The only satisfying dictation experience relies on the ultra basic text editor included with Dictate. Although it is supposed to be compatible with Microsoft Word, actual usage is frustrating. And attempting to use dictate in other applications makes me want to throw my computer out the window, as illustrated in the video below.

Stephen’s Stance

I am utterly frustrated with Dragon Dictate for Mac. It shows sparks of brilliance but is so tarnished by the obnoxious experience of actually using it that I have half a mind to demand my money back. And it certainly did cost quite a bit of money, especially compared to the free and less frustrating speech recognition software bundled with Microsoft Windows!

I’d like to see Dragon improve this product in key areas:

1. Add a simple, streamlined dictated command to add a word to the dictionary – “add that” should allow you to spell and train in one go
2. Fix the obnoxious behavior outside the Dictate “notepad” window – or add an “insert only” setting to not “guess” where you are in a document and overwrite things
3. Fix the weird “add extra characters after the cursor” bug demonstrated in my video
4. Allow me to fix one or two words, not a whole phrase, and allow me to fix it outside the Recognition window’s (limited) choices
5. Add a “no abbreviations” setting to the Mac version so it stops correcting “minutes” to “min.”
6. Allow me to say “always recognize it this way” for words like “Fibre Channel” and “Stephen’s stance” (not “fiber channel” and “Steven’s stance”)
7. Improve recognition of common commands and terms (“explanation point”? Seriously? When has anyone ever dictated that?)

See Nuance Responds to My Dragon Dictate Concerns for detailed information on these concerns

Finally, I suggest that Dragon does something to lower the price. I got mine at a steep discount, but it was still $129, which is pretty stiff for a utility of limited utility. The included headset and USB adapter are of mediocre quality and aren’t worth more than $20. If Dragon brings Dictate to the Mac App Store at $49 and fixes the bugs it might be worthwhile. But for now, it earns a “don’t buy“.

One more thing: 90 days of support? For a product costing over $100? Seriously? I was so frustrated at my initial attempts to use Dictate that I stuck it on the shelf for months before giving it another shot. Now I have to pay for support. Or not.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Dragon Dictate for Mac: Utterly Frustrating
This post was categorized as Apple, Everything, Features, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Now this is a respectable support matrix! (Saint Vitus' Cathedral, Prague)

As a computer hobbyist and tinker, I love making things work, but as an IT professional I just can’t countenance this kind of hacking. The sad truth is, regardless of whether or not something works, product support is usually more important. That’s why I’m singing the support matrix blues: I don’t care if you say it works, just tell me if it’s supported.

“Supported” and “Functional” Are Not Synonyms

I hear it all the time from IT product vendors: our product works great and we support it with Oracle, VMware, Microsoft Windows, and every other application. But what about the opposite? Does VMware support running in this configuration? Will it run in a virtual machine? And can I call all the involved vendors when things fail, or will they just point fingers?

The truth is, interoperability in modern IT environments is tricky enough without the added pressure of production, downtime, and hostile users. Although many of us are clever hackers by nature, we have to set this aside and be as conservative as possible when designing production systems. We just can’t afford to use components that aren’t supported.

The time has come to draw a line for vendors: Stop encouraging buyers to use configurations that aren’t supported by everyone involved, and stop confusing the terms, “supported” and “functional” when discussing your products. If it’s not on the hardware compatibility list (HCL), if not supported regardless of what you say.

The HCL Bill of Rights

Buyers deserve better than empty promises:

• A hardware compatibility list (HCL) or support matrix is a one-way list of approved functionality. In other words, it’s a way for Vendor A to affirm that Vendor B’s product works with their own. Presence on the list is usually initiated by Vendor B, however, who often bears the financial and functional burden of securing a spot.
• A good support matrix should list revisions as well as products, preferably down to point releases of drivers, firmware, and hardware. Often, special versions are produced just to attain certification with some popular vendor, and it’s up to the buyer to ensure that he is using these magical revisions.
• Hardware compatibility lists should be open to the public for browsing, querying, and deep investigation. Take a look at the VMware ESX HCL for a good example. Updated continually, it can be queried interactively or downloaded on demand as a PDF file. Nice!

Real-World Objections

Vendor support matrices are fallible, of course. Just because the list says it works doesn’t mean it does, and just because it’s not on the list doesn’t mean it doesn’t. But presence on the support matrix lets the end-user (also known as “the buyer”, “the customer”, and “the whole reason for the existence of product vendors in the first place”) get everyone on the phone when something breaks.

Why isn’t everything on the HCL? There are lots of good reasons:

• Some things just don’t work. Not every implementation of iSCSI, NFS, or SMB is equal, and some just don’t interoperate. Obviously, this is a very good reason for product to be absent from the HCL, and one should always assume that things don’t work rather than assuming that they do.
• Some things work reasonably well but fall apart in production or under load. These may or may not appear on the HCL, the listing means that buyers have some recourse when things fail in production. These can be difficult to locate in the lab or proof of concept testing, though both testing and listing are valid considerations when making a purchase.
• Some vendors lack the resources or motivation to get their products listed on all of the various compatibility lists and support matrices. This is fine: if you don’t want to go through the motion of getting on the supported list, I don’t have to buy your product for that application!
• New products often take a while to get on the supported list, and vendors commonly advise customers to buy and implement them anyway. “It will be included next month,” they will claim, “go ahead and buy it now.” This is absolutely irresponsible behavior, and no customer should buy a brand-new product that’s not listed on the empty promise of their sales rep.

“But it’s just a paper certification,” I heard the vendors cry. “It doesn’t matter if things work, all you have to do is fill in the paperwork and pay the money.” This may or may not be true, but it is absolutely an irrelevant argument. Do you really think that VMware, Microsoft, Oracle, or anyone else will decide to support your broken application just because you derive their paper certification requirements?

Stephen’s Stance

As an IT professional, I subscribe to a simple rule: I won’t recommend a solution that’s not listed in the support matrix or HCL. “I don’t care if it works, only that it supported,” is really a cynical overstatement of this position. Of course I care whether or not a solution works! But merely claiming that something works is not sufficient for me to recommend it. I need to be able to demonstrate that it’s a supported configuration. Show me the listing, and I’ll start recommending it.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | I Don’t Care If You Claim It Works (The Support Matrix Blues)
This post was categorized as Enterprise storage, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

How granular is your storage? This question is just as relevant today as when I first published this article in 2008!

Many storage challenges focus on correlating high-level uses of data (such as applications) with the nuts and bolts of storage infrastructure. These discussions often revolve around the conflict between data management, which demands an ever-smaller unit of management, and storage management, which benefits most from consolidation. Developing data management capability that is both granular enough for applications and scalable enough for storage is one key to the future of storage.

Storage Management: Scaling Up

As I discussed in a previous Sunday Series piece, Turning the Page on RAID, the data storage industry has traditionally focused on reducing granularity. Disk capacity has expanded, and RAID technology has multiplied this by combining multiple physical drive mechanisms into a single virtual one. Storage virtualization technologies, from the SAN to the server, have also often been touted primarily as a mechanism to reduce heterogeneity. From a technical perspective, therefore, granularity has been an obstacle to overcome.

The core organizational best practice for storage management is the reduction of complexity and the enforcement of standardization. Consolidation of storage arrays and file servers is a common goal, as IT seeks to benefit from economies of scale. The goal of both initiatives is the creation of a storage utility or managed storage service. This mirrors efforts on the server and network sides to consolidate and virtualize hardware.

Although both technological and organizational factors have traditionally driven granularity out of storage, this does not have to be the case. Virtual pools of storage are ideal for providing storage on demand, as disk-focused RAID groups give way to more flexible sub-disk storage arrangements. And an operational focus on standardized storage service offerings has the potential to enable scalable management of these smaller units.

Filing Service

File-based protocols would seem to have more potential for granular storage management, but they have been undermined by the hierarchical nature of modern file storage. Whether the connection to a file server uses NFS, CIFS, or AFP, the key unit of management is actually the shared directory, not the file. All files in the share \\firefly\backups would be located on the same server and would be managed as a unit.

NAS virtualization can change this somewhat, as can more specialized NAS servers. Although Microsoft DFS enables consolidation and virtualization of NAS shares, it does not allow subdivision of shares below the directory level – all files in a directory must be placed on the same server. Tricks like stubbing and links allow for some movement, but these do not solve the core issue. Specialized virtual NAS devices from F5 (the ARX, nee Acopia), NetApp, BlueArc, Symantec, and others have the ability to move files individually, providing as much a virtualized storage environment as any block-focused enterprise array. Avere is also beginning to talk about granular file management.

But even an ideal virtualized file server lacks the kind of granularity demanded by users. They care about data, not files, and most applications consolidate their data storage into a few files. Consider a database, for example, where users want each record treated uniquely but storage devices see just a few much larger files. We need a storage revolution, where someone creates an ideal storage platform in which each individual record or object includes custom metadata and is managed independently. This would truly be a massive change, however, and it is not clear that all applications will follow the object storage model of Google and Amazon.

Small is Beautiful

Barring a revolution in data management, our best hope is to allow greater granularity in storage management. As mentioned above, virtualization technology has the potential to enable management and protection of any unit of storage, right down to the individual block or record. But the reality of storage virtualization has not matched its promise.

What is needed is greater integration. Each layer of virtualization (file system, volume manager, hypervisor, network, array, and RAID) also hides necessary details from lower layers. Consider the case of a virtual server snapshot: The application and filesystem must be in a quiesced state to allow a snapshot to be taken at the storage level, but the storage array has no intrinsic information about how its capacity is used. A given LUN might contain dozens of servers on a shared VMFS volume, so all must be snapped together.

Integration can be enabled by sharing more information through APIs. VMware leverages Microsoft Volume Shadow Copy Service (VSS) integration for shared storage so a VMFS snapshot can call the operating system and even applications (Windows Server 2003 only, for now) to prepare the data. Similarly, VSS can communicate directly with supported iSCSI and Fibre Channel arrays, calling a snapshot at the right moment. And Microsoft is, no doubt, enhancing VSS as we speak.

As virtualization technology matures, expect this type of integration to improve. We hope to see more APIs exposed by VMware and Microsoft, allowing communication up and down the stack to break through the information barrier. Imagine a future where a standard API like VSS can pass a message through VMware, Xen, and Hyper-V to the underlying storage array to initiate a snap. I predict that this kind of integration-enabled granularity is not too far off.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Granularity: The Hidden Challenge of Storage Management
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

• My writing:
  • My thoughts concerning HDS following their “blogger day” in London
  • My good friend W. Curtis Preston announced more Backup Central Live! dates; you really ought to go see him!
• Other great stuff:
  • Intel Announces New Xeon 4 Socket CPU (E7); Dell, HP and IBM Slated to Refresh Blades
  • Awesome! Free Microsoft iSCSI Target!
  • Anatomy of an Attack is a forthright and honest statement from an RSA blogger
  • The Impossible Project -new- PX 680 Film is amazing. Just amazing!
  • The Iomega StorCenter ix2-200 Network Storage, Cloud Edition is out
  • Jeff Fry shares his impression of Drobo

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, April 8, 2011
This post was categorized as Enterprise storage, Everything, Personal, Terabyte home. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

VMware is embracing FCoTR just as it rises to dominance in converged networking

VMware shocked the world today, pre-announcing an important addition to the vStorage family of APIs found in their marquee vSphere family of products. The vStorage API for Token Ring Integration (VATRI) promises to ease the transition to converged networks featuring the emerging Fibre Channel over Token Ring (FCoTR) storage protocol. According to a recent report by The D’Plata Group, FCoTR is set dominate the enterprise storage market, and this integration is a confident step in that direction.

The FCoTR Connection

VMware’s vStorage API for Array Integration (VAAI) in vSphere 4 was a major step for vSphere, enabling the offload of thin provisioning and data protection tasks to a storage array. But array integration was not enough to cement iSCSI and Fibre Channel (FC) in the virtualized data center. Indeed, Cisco’s oft-reported troubles with the myriad enhancements needed to make Ethernet a viable converged networking protocol indicated that a more-appropriate alternative was needed.

Fibre Channel over Token Ring (FCoTR) was created by a group of top storage, networking, and virtualization minds and launched to much fanfare in 2010. The Fibre Channel over Token Ring Alliance estimates an uncountable number of supporters for FCoTR, and such luminaries as Greg Ferrous and Joe Iamsick have thrown their weight behind the protocol. But it takes more than a few drunken bloggers to change the course of history, and VMware’s support is critical.

Storage vendors were quick to jump on the FCoTR bandwagon beginning with Compelling (since acquired by Dull), Numble Storage, and NES, which announced their support at Tech Field Day immediately after the unveiling of the spec. Network vendors have been slower to embrace the standard, though many individuals have been spotted proudly wearing FCoTR buttons. But perhaps the most ringing endorsers are IBM and HP, with the latter proudly including FCoTR on a banner at their recent Technology Summit.

VATRI: PFM!

While VAAI integrates high-level storage functions, VATRI functions at a deeper level. VMware vSphere 5 will include a standard virtual Token Ring adapter with full FCoTR support, enabling seamless integration of storage and servers. FCoTR adapters certified by DampStorage Labs will also get a paravirtualized Token Ring driver for enhanced performance, reliability, efficiency, cost savings, and wholesomeness. Finally, the revolutionary Token Cleanup API will take care of dropped tokens, a common datacenter occurrence.

“I guess the cat’s out of the bag now,” reports VMware spokesman, Jon Truer. “I might as well agree that FCoTR will take its rightful place at the center of the datacenter.” This ringing endorsement was echoed by Microsoft “Virtual PC Guy” Lance Armstrong, who added “even though VMware is first to market, our MSATRI implementation will be superior in every way.”

VATRI beta testing has already begun. As usual, VMware’s vExperts were given early access to the API and have been using it in secret for months. “It seems to work ok,” claims Jason Botch, while vShawn Cleric donned a sweater vest emblazoned with the FCoTR logo to show his enthusiasm. Storage wonks were equally enthusiastic, with Robin Hairs suggesting that Apple was behind the curve for ignoring FCoTR in his StrangeMojo blog.

Stephen’s Stance

VMware is really showing leadership here. Adding FCoTR support to vSphere 5 just cements their position as the dominant technology company for the next century. Since server virtualization is the computing paradigm that will rule the world for ever and ever, their embrace of Token Ring makes perfect sense. I look forward to adding FCoTR adapters and arrays to my home lab just as soon as they show up on eBay!

Update

Predictably, this story is getting major coverage! Check back throughout the day and I will update the page with news as it appears!

• Storage Vendors Announce FCoTR and VATRI Support

© sfoskett for Stephen Foskett, Pack Rat, 2011. | vSphere 5 To Include vStorage API for Token Ring Integration
This post was categorized as Computer History, Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.