Stepping out of a subway car is an entirely different matter when it's moving!

Mobility of virtual machines is a sticky wicket. As open systems infrastructure becomes increasingly virtualized, administrators and managers wish to use the technology to balance workload, ease migration, and provide better availability. Although technology is improving, actually moving virtual machines is not always a piece of cake. Let’s lay down a baseline of information so we may begin a discussion on the true nature of virtual machine mobility.

Mobility of What?

Let us consider first the question of what exactly is being moved. Systems administrators often focus on “the machine”, which encompasses the operating system and configured state of the virtual machine itself. But the true “mass” of the system is its stored data. Hypervisor vendors have come up with different techniques of moving these two essential elements, reflecting the unique characteristics of each.

• The virtual machine is an instance of an operating system along with its state and configuration. Mobility of virtual machines requires all of this to be preserved, along with any I/O channels. Live migration of virtual machine requires that any active network sessions be maintained, along with RAM content, registers and buffers, and so many other elements.
• The virtual machine image (commonly referred to as “storage”) is the static content addressed by a virtual machine. Typically a VMDK or similar virtual disc image, it must be accessible to the virtual machine at all times. Live migration of a virtual machine image is tricky, but perhaps not quite as complex as live migration of a running operating system.

VMware, Microsoft, and others recognize these 2 distinct elements to be migrated, and have come up with a variety of complementary technologies for each:

• vMotion is VMware’s virtual machine migration solution, and has continually evolved with each iteration of the hypervisor. DRS leverages vMotion to automate mobility. VMware has also created Storage vMotion and Storage DRS as complements to handle mobility of virtual machine images.
• Microsoft Hyper-V Live Migration is conceptually similar to vMotion, though newer and less full-featured. With Hyper-V 3.0, Microsoft will introduce Storage Live Migration as a complementary technology akin to Storage vMotion. Most other virtual machine managers also support some form of live migration, though live migration of storage is less common.

Mobility in What State?

One of the key benefits of virtual machine technology is the ability to “run anywhere” on dissimilar hardware. From the very beginning, hypervisors have provided the ability to create a universal virtual machine image that would run on a variety of supported platforms.

This leads to one of the key values of server virtualization in the data center: Disaster recovery. The ability to take a virtual machine image and system state and bring it online after a disaster is a true revolution for open systems IT. The benefits of the single usage of server virtualization technology easily justify the investment to many businesses.

But this sort of “cold” migration seems passé when compared to the live or “hot” migration possible with technologies like VMware vMotion. Live migration is much more difficult, since active client sessions must be preserved in activity must not be greatly interrupted.

This is the second great question that must be asked when considering virtual machine mobility: In what state will the virtual machine be moved? Will it be a cold, powered down image of the system? A suspended or paused operating system image? Or a full, running machine?

Mobility to Where?

Once we have decided whether we are discussing virtual machine migration or movement of storage resources, we must consider the scope of the movement. The ability to move a virtual machine from one member of the cluster to another has now become fairly common. But what about systems that are not related in a cluster? Or that are spread over great distances?

• The nice thing about clusters is that they share resources before and after a virtual machine is moved. It is practical to move the running virtual machine, its storage, or both independently and to expect that performance will not dramatically suffer as a result. The cluster can also preserve network connections, and even I/O state, without much impact on clients or other external elements.
• It is a bit more difficult to move systems within a data center, since one must maintain the I/O connections that might be interrupted. It is fairly trivial to configure an IP network and storage array to allow multiple machines to access the same iSCSI or NFS storage resources. It is a little more difficult to configure Fibre Channel (and, by extension, FCoE) SAN’s to handle this sort of dynamic movement, but it is not impossible. Although moving a running machine from one network port to another could cause client access to be interrupted, technologies like VXLAN allow these sessions to continue, and improved network switching technology should reduce performance impact.
• Moving the machine to a different data center is another matter entirely. Stretching a layer-2 Ethernet LAN or Fibre Channel SAN across a metro or greater distance, while possible, will always be problematic. IP routing is flexible, but it takes time for changes to propagate when live machines are moved. And it is difficult to keep storage in sync over long distances due to the amount of time it takes for information to transit. Again, all of these challenges are being addressed in various ways, but they’re still hard!

Stephen’s Stance

Moving cold virtual machine images from system to system, or even across great distances, is one of the main selling points of server virtualization. But it becomes much more difficult to manage movement of virtual machines that are still running, especially outside cluster or across WAN links. When talking about virtual machine mobility, it is important to consider what is being moved, the state it is in, and where it is going.

Note: This discussion is part of “Building Virtual Infrastructure”, my new seminar series with Truth in IT.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Virtual Machine Mobility: Of What, and to Where and in What State?
This post was categorized as Enterprise storage, Features, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Windows 8 server editions will include a filter driver for NTFS for data deduplication

The next version of Microsoft Windows Server includes integrated data deduplication technology. Microsoft is positioning this as a boon for server virtualization and claims it has very little performance impact. But how exactly does Microsoft’s de-duplication technology work?

Introducing Windows 8 Deduplication

Let’s make one thing clear right from the start: Microsoft started from a clean sheet and invented their own deduplication technology. This is not a licensed, cloned, or copied feature as far as I can tell. There are some clever aspects to it, along with a few head scratchers for folks like me who’ve seen lots of different deduplication approaches.

Microsoft’s deduplication is layered onto NTFS in Windows 8, and will be a feature add-on for Server users. It is implemented as a filter driver on a per volume basis, with each volume a complete, self describing unit. It is cluster aware, and fully crash consistent on all operations. This is a pretty neat trick: As is typical for Microsoft, deduplication will be a simple, transparent feature.

Now let’s talk for a moment about what Windows 8 deduplication is not.

• It is a server-only feature, like so many of Microsoft’s storage developments. But perhaps we might see it deployed in low-end or home servers in the future.
• It is not supported on boot or system volumes.
• Although it should work just fine on removable drives, deduplication requires NTFS so you can forget about FAT or exFAT. And of course the connected system must be running a server edition of Windows 8.
• Although deduplication does not work with clustered shared volumes, it is supported in Hyper-V configurations that do not use CSV.
• Finally, deduplication does not function on encrypted files, files with extended attributes, tiny (less than 64 kB) files, or re-parse points.

Some Technical Details on Deduplication in Windows 8

Microsoft Research spent 2 years experimenting with algorithms to find the “cheapest” in terms of overhead. They select a chunk size  for each data set. This is typically between 32 KB and 128 KB, but smaller chunks can be created as well. Microsoft claims that most real-world use cases are about 80 KB. The system processes all the data looking for “fingerprints” of split points and selects the “best” on the fly for each file.

After data is de-duplicated, Microsoft compresses the chunks and stores them in a special “chunk store” within NTFS. This is actually  part of the System Volume store in the root of the volume, so dedupe is volume-level. The entire setup is self describing, so a deduplication NTFS volume can be read by another server without any external data.

There is some redundancy in the system as well. Any chunk that is referenced more than x times (100 by default) will be kept in a second location. All data in the filesystem is checksummed and will be proactively repaired. The same is done for the metadata. The deduplication service includes a scrubbing job as well as a file system optimization task to keep everything running smoothly.

Windows 8 deduplication cooperates with other elements of the operating system. The Windows caching layer is dedupe-aware, and this will greatly accelerate overall performance. Windows 8 also includes a new “express” library that makes compression “20 times faster”. Compressed files are not re-compressed based on filetype, so zip files, Office 2007+ files, etc will be skipped and just deduped.

New writes are not deduped – this is a post-process technology. The data deduplication service can be scheduled or can run in “background mode” and wait for idle time. Therefore, I/O impact is between “none and 2x” depending on type. Opening a file is less than 3% greater I/O and can be faster if it’s cached. Copying a large file can make some difference (e.g. 10 GB VHD) since it adds additional disk seeks, but multiple concurrent copies that share data can actually improve performance.

Stephen’s Stance

Although I am intrigued by Microsoft’s new deduplication technology in Windows 8 server, I still have many questions about its usefulness and impact on performance. Concentrating duplicate data in the system volume makes sense from a technical perspective, but could lead to an I/O hotspot on the disk. This is especially true for external caching storage systems, since there is no integration between Microsoft deduplication and storage array features. I am particularly concerned about the use of deduplication with VHD files in Hyper-V, since it could eat up valuable system RAM and impact I/O performance.

If you would like to try Microsoft deduplication for yourself, I am happy to report that it is included in the developer preview of Windows 8 that is available on Dev Center. Here are a few commands to get you started, and read Rick Vanover’s post too!

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication
Enable-DedupVolume E:
get-dedupvolume

Note: I am a Microsoft MVP and Microsoft briefs me on upcoming technologies under NDA. This post is based on a Microsoft briefing from November which was said at the time not to be covered by any NDA. All of this information could be gleaned by experimenting with the Windows 8 developer preview, but it’s much easier to just go to the source.

© sfoskett for Stephen Foskett, Pack Rat, 2012. | Microsoft Adds Data Deduplication to NTFS in Windows 8
This post was categorized as Enterprise storage, Everything, Gestalt IT, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

VAAI in 4.1: Three For Block (Plus One More)

vSphere 4.1 officially includes three VAAI primitives: Full Copy, Hardware Assisted Locking, and Block Zeroing. But it was intended to include a fourth (Thin Provisioning Stun) and variations in nomenclature have confused administrators.

Here’s the complete list of VAAI primitives in vSphere 4.1, including a little more about what they really do.

Atomic Test & Set (ATS) (AKA Hardware Assisted Locking or HardwareAcceleratedLocking)

This is one of the most valuable storage technologies to come out of VMware, but it’s usually overlooked. It allows compare and write of SCSI blocks in one operation using proprietary array-specific op-codes. This enables granular locking of block storage devices beyond the basic full-LUN “reservations” included in SCSI from days of yore. This is huge for performance when LUNs are shared, with HP reporting at VMworld allowing 6 times more VMs per LUN once VAAI is turned on. NFS doesn’t really need this since locking is a non-issue and VM files aren’t shared the same way LUNs are.

To verify from the CLI:

esxcfg-advcfg -g /VMFS3/HardwareAcceleratedLocking

Cloning Blocks (AKA Full Copy, Extended Copy, or HardwareAcceleratedMove)

This is the signature VAAI command. Instead of reading data from the array then writing it back, the hypervisor can command the array to make a mirror of a range of data on its behalf. If supported and enabled, operations like Clone and VMotion can be lightning quick, since many high-end arrays are very capable of this sort of action. This leverages the SCSI XCOPY command (or other proprietary alternatives enabled with a VMware plug-in) and defaults to a 4 MB block size.

To verify from the CLI:

esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove

Zeroing File Blocks (AKA Block Zeroing or HardwareAcceleratedInit)

Thin provisioning is difficult to get right because storage arrays have no idea what’s going on in the hosts. VAAI includes a generic interface for communication of free space, allowing vast ranges of blocks to be zeroed out at once. Along with the generic T10 “WRITE_SAME” command, VAAI allows vendors to use “UNMAP” or their own mechanisms with a plug-in. VMware defaults to a 1 MB block size for zeroing operations. But zeroing only works for supported block arrays, and only for capacity inside a VMDK: Clones and vMotions leave the array guessing.

To verify from the CLI:

esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit

Out of Space Condition (AKA Thin Provisioning Stun)

Running out of capacity is a catastrophe, but it’s easy to ignore the alerts in vCenter until it’s too late. This command allows the array to notify vCenter to “stun” (suspend) all virtual machines on a LUN that is running out of space due to thin provisioning over-commit. This is the “secret” fourth primitive that wasn’t officially acknowledged until vSphere 5 but apparently existed before. In vSphere 5, this works for both block and NFS storage. Signals are sent using SCSI “Check Condition” operations.

VAAI Commands in vSphere 4.1

esxcli corestorage device list

esxcli vaai device list

esxcli corestorage plugin list

VAAI in 5.0: Lots More

vSphere 5 includes a new generation of VAAI primitives, including support for NFS storage. Once again, VMware includes a “secret” primitive not yet implemented, and puzzlingly lumps some of the official entries together in company documentation. But VMworld changed that, with one presentation including insight into the real list of VAAI primitives.

vSphere 5 includes and enhances all the VAAI primitives from vSphere 4.1 (including “Out of Space Condition”) as well as the following new capabilities.

vSphere 5 includes the above primitives, plus these

UNMAP (AKA Space Reclamation or EnableBlockDelete)

Zeroing File Blocks is great, but VMFS operations like cloning and vMotion didn’t include any hints to the array. This means that some of the biggest storage operations aren’t accelerated or thinned out! UNMAP is a SCSI command, and it was added to VAAI to allow thin-capable arrays to clear unused VMFS space as well. But vCenter couldn’t handle waiting for the array to return “UNMAP” command status, so the use of this primitive is officially suspended by VMware until this issue is resolved.

To verify from the CLI:

esxcfg-advcfg -g /VMFS3/EnableBlockDelete

For now, disable it:

esxcfg-advcfg –s 0 /VMFS3/EnableBlockDelete

Quota Exceeded Behavior (AKA TP Soft Threshold)

It would be nice if vSphere could react before an out-of-space condition is reached. This primitive allows the array to notify the hypervisor preemptively, possibly triggering a Storage DRS re-balance, for example. It also allows the hypervisor to query the array for its threshold values using SCSI “mode sense”.

TP LUN Reporting (AKA Report Thin Capacity)

A very simple command, this allows the hypervisor to query the array to determine if a LUN is thinly provisioned. On block arrays, the SCSI “TPE” bit is used, but this also works with NFS.

NFS Full File Clone (AKA Hardware Snap)

NFS already did a nice job with storage provisioning and presentation, but VMware couldn’t make use of hardware snapshot capability. This changes in vSphere 5, with a “Full Copy”-like mechanism appearing for NAS. But the NFS version doesn’t support Storage vMotion operations like its block cousin.

NFS Space Reservation (AKA Zeroedthick Files)

NFS is always thin, which is a good and bad thing. This new primitive adds two new file type for NFS-based VMDK files: “Eager” and “Lazy” Zeroed Thick. These are similar to the old default “thick” volume in block storage, but thick provisioning and space reservation is brand new territory for NFS admins. This primitive uses the NFS “reserve_space” command.

NFS Extended Stat (AKA Extended Status)

One-upping the SCSI block world, vSphere 5 can query NAS servers for all sorts of information, including file size. This should make NFS in vCenter even more friendly!

NFS Space Reclaim (AKA Space Reclamation)

Although the block-only UNMAP feature doesn’t work, the NFS equivalent Space Reclaim seems fine. This enables the hypervisor to unlink old files once they’re no longer used. This primitive uses the NFS “unlink” command.

NFS Fast File Clone (AKA Linked Clone Offload)

This “secret” VAAI primitive allows a future release of VMware View (and only View) to offload creation of linked clones to a NAS array. Hopefully we’ll see more of this in the future!

VAAI Commands in vSphere 5

esxcli storage core device list

esxcli storage core device vaai status get

esxcli storage core plugin list

© sfoskett for Stephen Foskett, Pack Rat, 2011. | A Complete List of VMware VAAI Primitives
This post was categorized as Enterprise storage, Features, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

How granular is your storage? This question is just as relevant today as when I first published this article in 2008!

Many storage challenges focus on correlating high-level uses of data (such as applications) with the nuts and bolts of storage infrastructure. These discussions often revolve around the conflict between data management, which demands an ever-smaller unit of management, and storage management, which benefits most from consolidation. Developing data management capability that is both granular enough for applications and scalable enough for storage is one key to the future of storage.

Storage Management: Scaling Up

As I discussed in a previous Sunday Series piece, Turning the Page on RAID, the data storage industry has traditionally focused on reducing granularity. Disk capacity has expanded, and RAID technology has multiplied this by combining multiple physical drive mechanisms into a single virtual one. Storage virtualization technologies, from the SAN to the server, have also often been touted primarily as a mechanism to reduce heterogeneity. From a technical perspective, therefore, granularity has been an obstacle to overcome.

The core organizational best practice for storage management is the reduction of complexity and the enforcement of standardization. Consolidation of storage arrays and file servers is a common goal, as IT seeks to benefit from economies of scale. The goal of both initiatives is the creation of a storage utility or managed storage service. This mirrors efforts on the server and network sides to consolidate and virtualize hardware.

Although both technological and organizational factors have traditionally driven granularity out of storage, this does not have to be the case. Virtual pools of storage are ideal for providing storage on demand, as disk-focused RAID groups give way to more flexible sub-disk storage arrangements. And an operational focus on standardized storage service offerings has the potential to enable scalable management of these smaller units.

Filing Service

File-based protocols would seem to have more potential for granular storage management, but they have been undermined by the hierarchical nature of modern file storage. Whether the connection to a file server uses NFS, CIFS, or AFP, the key unit of management is actually the shared directory, not the file. All files in the share \\firefly\backups would be located on the same server and would be managed as a unit.

NAS virtualization can change this somewhat, as can more specialized NAS servers. Although Microsoft DFS enables consolidation and virtualization of NAS shares, it does not allow subdivision of shares below the directory level – all files in a directory must be placed on the same server. Tricks like stubbing and links allow for some movement, but these do not solve the core issue. Specialized virtual NAS devices from F5 (the ARX, nee Acopia), NetApp, BlueArc, Symantec, and others have the ability to move files individually, providing as much a virtualized storage environment as any block-focused enterprise array. Avere is also beginning to talk about granular file management.

But even an ideal virtualized file server lacks the kind of granularity demanded by users. They care about data, not files, and most applications consolidate their data storage into a few files. Consider a database, for example, where users want each record treated uniquely but storage devices see just a few much larger files. We need a storage revolution, where someone creates an ideal storage platform in which each individual record or object includes custom metadata and is managed independently. This would truly be a massive change, however, and it is not clear that all applications will follow the object storage model of Google and Amazon.

Small is Beautiful

Barring a revolution in data management, our best hope is to allow greater granularity in storage management. As mentioned above, virtualization technology has the potential to enable management and protection of any unit of storage, right down to the individual block or record. But the reality of storage virtualization has not matched its promise.

What is needed is greater integration. Each layer of virtualization (file system, volume manager, hypervisor, network, array, and RAID) also hides necessary details from lower layers. Consider the case of a virtual server snapshot: The application and filesystem must be in a quiesced state to allow a snapshot to be taken at the storage level, but the storage array has no intrinsic information about how its capacity is used. A given LUN might contain dozens of servers on a shared VMFS volume, so all must be snapped together.

Integration can be enabled by sharing more information through APIs. VMware leverages Microsoft Volume Shadow Copy Service (VSS) integration for shared storage so a VMFS snapshot can call the operating system and even applications (Windows Server 2003 only, for now) to prepare the data. Similarly, VSS can communicate directly with supported iSCSI and Fibre Channel arrays, calling a snapshot at the right moment. And Microsoft is, no doubt, enhancing VSS as we speak.

As virtualization technology matures, expect this type of integration to improve. We hope to see more APIs exposed by VMware and Microsoft, allowing communication up and down the stack to break through the information barrier. Imagine a future where a standard API like VSS can pass a message through VMware, Xen, and Hyper-V to the underlying storage array to initiate a snap. I predict that this kind of integration-enabled granularity is not too far off.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Granularity: The Hidden Challenge of Storage Management
This post was categorized as Computer History, Enterprise storage, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Now I have been asked to write an article for TechTarget on the very same subject: How to select a virtual server backup product. I’d like to request input for this piece, and hope we can work together to produce a useful list of recommendations. Note that this isn’t a buyer’s guide like the DCIG effort: There will be no exhaustive lists of functions and features here. Instead, I’m writing about the options available in a more general sense.

I came away from the Tech Field Day presentations by Veeam, Symantec, and others impressed by the creativity of virtual server backup approaches. Clearly, these aren’t the same old backup products! W. Curtis Preston quipped that a hypervisor sometimes seems like a little datacenter in a box, and that changes everything. Every action taken impacts the operation of the whole system, but the virtualization layer also adds new flexibility.

Virtual Server Backup Features

What would you look for in a virtual server backup product? Here’s my initial concept:

• Supported environments - What dependencies do the products have?
  • Do they require integration features like VCP or VADP? We should definitely talk more about the integration features offered by the hypervisor!
  • Some products support only virtual environments, while others work on traditional machines as well. And what about non-VMware support?
• Advantages and disadvantages – Applications designed for virtual environments certainly offer many advantages, but what about the drawbacks?
  • How important is it to use a single unified product across both virtual and non-virtual environments, as many traditional backup vendors suggest?
  • Is the whole “full-machine vs. file-level” argument worth having?
• Application integration – To what extent to they extend above the OS and hypervisor?
  • I’d like to talk about integration with common virtualized applications, but which are most prevalent?
  • We should talk about crash-consistent backup, too, especially when it comes to virtual machines.
  • Then there’s management integration: What options are there for automated deployment and setup? And what metrics and reports are important?
• Advanced features – These new products have some pretty impressive features!
  • Is deduplication a critical need in the backup app?
  • We should also talk about non-scheduled approaches like continuous data protection (CDP)
  • There are also alternative backup targets, like snapshots, disk/VTL, and the cloud
  • I’m also curious about throttling/QoS features
• Other areas
  • What about supportability of virtual machine backups – are there issues with companies not allowing advanced backup features to be used?
  • Do the “new guard” of virtual server backup companies offer any unique support and integration options?

I’d like to identify a few key features to look for in particular. What are the most-important elements for a virtual server backup product, as opposed to a traditional offering?

Finally, what about non-VMware environments. I know a bit about Microsoft DPM, having been at the introduction of that product, but what offerings are there for Hyper-V or Xen environments? How do they differ from VMware-focused products?

Stephen’s Stance

I welcome the input of end-users, analysts and consultants, and the product vendors themselves. I want to come up with a good guide to selecting these products, and I think an open discussion is a good way to make that happen. Who knows, it might even be a worthwhile companion to the information in the DCIG guide!

I’m not sure how the final article will be released, but I believe it will be offered as a free download on BitPipe.com. Regardless, the discussion and commentary here will remain free, open, and available. Thank you!

Updates

I’ve already had some excellent feedback from Twitter. Thanks!

• @Stu suggested I read What’s Next in VMware Backup
• @ChrisDearden says “what about talking about how to get the data out of the backup system – ie to offsite / archive etc” and “I’m not sure if trap is the word I would use. no more than all the tape based providers use the same format (“
• @WGoderis says “what about scalability , what if you need to back up hundreds or thousands of vms”

© sfoskett for Stephen Foskett, Pack Rat, 2011. | How To Select a Virtual Server Backup Product?
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Now for some highlights from the week:

• My writing:
  • I wrote another post for Network Computing: Storage Capacity Still Drives Accessibility
  • I picked up a new 2011 MacBook Pro last Thursday, and took it along to Redmond. You might want to read my MacBook Pro review series:
    • 2011 MacBook Pro Review: Introduction
    • Decision Point: Comparing the 2011 MacBook Pro Models
    • Benchmarking the 2011 13″ Core i5 MacBook Pro
  • I had some time on the plane, so wrote up some things I’ve been meaning to get out:
    • How Google and Dropbox Revolutionized My Laptop Migration
    • Keep Multiple Macs in Sync with Dropbox
  • Some good deals popped up this week, too: Great Deals on iPads (for now) and Get a Cradlepoint PHS300 Portable 3G WiFi Router For Just $79!
  • This week marks my first Storage for Virtual Environments seminar, so I kicked it off with Introducing Storage for Virtual Environments (From My Seminar)
• Other great stuff:
  • Terry Slattery is perhaps the most-famous CCIE, and we were lucky to have him join us at Networking Field Day in September. But he wanted to talk about Infoblox & NetMRI at Tech Field Day from last month! He also wrote about IPv6 Economics – Why Move To IPv6?
  • Another great post from Tech Field Day comes from Matthew Norwood, who will also be joining us in two weeks at Wireless Field Day: HP Networking – The Vision(As I Understand It)
  • Last week saw the unveiling of the iPad 2: I don’t often share John Gruber’s writing, but I loved The Chair, and also really enjoyed MG Siegler’s piece, An iPad Lover’s (Initial) Thoughts On iPad 2
  • Some interesting tech notes out of Redmond:
    • Using Hyper-V with large sector drives on Windows Server 2008 and Windows Server 2008 R2
    • An update that improves the compatibility of Windows 7 and Windows Server 2008 R2 with Advanced Format Disks is available
    • File Server improvements from Windows Server 2003 to Windows Server 2008 R2. 8 items for 8 years…
  • A great Greg Schulz post on bit-level encoding: From bits to bytes: Decoding Encoding
  • Nigel Poulton is always worth reading, but his take on Storage Benchmarking and Formula 1 is especially rewarding
  • Nerd Vittles didn’t like the Motorola Xoom: A Disappointing Introduction to Android 3.0
  • Another great enterprise networking piece by Greg Ferro: Diverging Ethernet Switch Markets
  • Who’d have thought it? Exploring the Limits of Datacenter Temprature
  • Some great information on the next Mac OS: iOS on the Desktop: Hands-On With the OS X 10.7 Lion Preview
  • Bill Hill put together a great single-piece on Tech Field Day Posts as well as Fixed Block vs Variable Block Deduplication – A Quick Primer

Subscribe to my Google Reader feed or follow me on Twitter to see these in real-time.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Back From the Pile: Interesting Links, March 4, 2011
This post was categorized as Apple, Enterprise storage, Everything, Personal, Terabyte home, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

My Storage for Virtual Environments seminar is really packed with content!

Next Thursday will be the first time I present my new-for-2011 Storage for Virtual Environments seminar. I’ve worked hard on the content, and appreciate all the input I’ve had from my readers and industry contacts since I first asked for help here on my blog.

The Virtual Opportunity

So what am I going to cover in this seminar? I hit on an interesting twist while working on the content: Virtual environments pose a challenge but also create an opportunity for storage people. Virtual environments are very demanding of storage features and performance, but we have the necessary tools in our arsenal. We just have to use them!

Server virtualization is the first step to a new IT paradigm

The first of three seminar hours is spent talking about the “big picture” issues for server virtualization. We talk about the trend for IT to move in a new direction, and how server virtualization is just the first step. It’s a bridge to the past, enabling legacy applications and operating systems to me used in a more-dynamic and flexible fashion. But the real change comes next, as virtualization and containerization hits the applications and they move away from legacy infrastructure.

This InformationWeek Analytics survey shows a slowdown in the trend towards virtual servers (from their 2010 Virtualization Management Survey)

Bur the reality check for server virtualization is starting to hit us. Look at the chart above, from an InformationWeek Analytics study on Virtualization Management. Server virtualization pros are scaling back their projections for virtualization penetration this year. Although most still believe many servers will be virtualized before the end of the year, it’s not the clear majority who thought that in 2009.

The VIrtualgeek survey demonstrates VMware's amazing market share

Another great source of information is the virtualization survey conducted by Chad Sakac. Although any web poll is not as scientific as a professional survey, it does reveal some interesting facts. One stand-out fact is the overwhelming: VMware ESX really dominate market for server virtualization. Although Microsoft Hyper-V and Citrix XenServer do have users, VMware controls more than half of the market.

Next, I talk about some of the realities of server virtualization:  I contrast the simple virtual hardware environment that the guest believes it is running on with the complicated reality of the modern virtual data center. I also talk little bit about the cloud, and what this future IT infrastructure will look like.

Transitioning to Storage

Storage and network I/O have a massive impact on virtual machine performance

But this is all a prelude to the meat of the seminar content: The impact that these virtual environments will have on storage infrastructure. As one of the three pillars of server performance, storage I/O is critical. But we must also consider the integration opportunities for advanced storage technologies.

This leads me to the roadmap for the rest of the seminar. Storage folks hold the keys to the future, but they must unlock the doors. It’s time to start thinking about storage in a new way:

1. Storage presentation must adapt to the demands of the hypervisor
2. Converged I/O technologies (“Everything over Ethernet”) bring exciting new opportunities
3. Storage virtualization technology (volume management, thin provisioning, automated tearing, and so on) finally have a use in virtualized environments
4. New storage architectures, including solid-state disks, are also required

This is the real thrust of my presentation. Storage folks need to change both their assumptions and architectures to respond to the challenges of this new virtual environment.

Join In!

Want to get involved? Here’s what you can do:

1. If you’re not already, subscribe to my rss feed – I’ll use the “Virtual Storage” category for this topic
2. Check back over the next month or so as I post slides, commentary, and data regarding storage for virtualization environments
3. Consider joining me at the seminars – I have 9 dates, and I’m sure most readers would qualify to attend free of charge!
   1. Thurs. Mar. 10 – Philadelphia, PA
   2. Thurs. Jun. 16 – Phoenix, AZ
   3. Tue. Jun. 28 – London, UK
   4. Tue. Jul. 19 – Toronto, ON
   5. Thurs. Jul. 21 – Cleveland, OH
   6. Tue. Aug. 9 – Denver, CO
   7. Tue. Oct. 18 – Dallas, TX
   8. Thurs. Oct. 20 – St. Louis, MO
4. As the year progresses, I’ll be asking for clarification and posting updates to the material

I look forward to hearing from you, meeting you at the seminars, and putting some great content together!

A Note on Copyright

As promised, the seminar slides will be released using the Creative Commons license, specifically CC-BY-NC-SA 3.0. This means that anyone out there can get a copy, remix them, or reuse them as long as they attribute me, share their work with the world under the same license, and don’t try to make money off of it. As producers of this seminar series, TechTarget will have a special license allowing them to use the content.

© sfoskett for Stephen Foskett, Pack Rat, 2011. | Introducing Storage for Virtual Environments (From My Seminar)
This post was categorized as Enterprise storage, Everything, Personal, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

Iomega's new ix12-300d brings EMC's SOHO company into the data center

EMC’s Iomega unit today released the 12-drive rack-mount storage product we have all been waiting for. It was never a question of whether Iomega could produce such a beast: The EMC LifeLine software and Iomega hardware were definitely up for it. The question was always if EMC would direct Iomega to fill the gaping hole in their storage lineup between the 4-drive ix4-200r and the entry-business AX4 arrays. The new ix12-300r packs 12 drive bays, scaling from 4 TB all the way to 24 TB, and backs it with quad gigabit iSCSI, redundant power, and everything else the small data center needs.

Stepping Up

The ix12 is a big step up. Although they sold multi-drive RAID systems even before the EMC acquisition, this new device is unlike anything we’ve seen before from Iomega. This slim (2U) chassis is all drives up front, with 12 hot-swap slots full of 3.5″ SATA storage. Under the hood is a dual-core 3 GHz Intel Core2duo E8400 “Wolfdale” CPU, a major horsepower upgrade from the single-core Celeron in the ix4-200r. It also has double the memory (2 GB) compared to that 4-drive product.

It all makes sense now: EMC's storage spectrum, from home to enterprise

If you’re concerned about performance after trying out a desktop ix4-200d, you needn’t worry. The desktop unit has a lowly 1.2 GHz Marvell 6281 and just 512 MB of RAM. While this might be enough for a desktop user, it could never handle the pounding of servers in a shared networked configuration. The ix4-200r, with its 3.2 GHz Intel Celeron D 352 “Cedar Mill“, offered much better performance even though its name was just one letter different. And the ix12′s CPU is three times faster still, though it remains a single-controller system.

More significant changes lurk around the back of the ix12. Dual redundant power supplies, a frequent request in this class, and variable-speed fans, surround four Ethernet ports. Each sports gigabit speed and the set supports Microsoft Windows MPIO, can be aggregated with 802.3ad, or used in VLAN configurations with up to 4 VLAN tags per port. The ix12 speaks just about every language, from NFS and SMB to AFP and iSCSI, and now adds WebDAV and DFS support, too.

A few limitations separate this new ix12 device from its enterprise-grade brothers, however. As noted, a single controller manages all access, so redundancy and parallel processing are limited. Although the ix12 sports 12 drive bays, it only has four SATA channels internally; each bay shares a channel with two others using SATA expanders. Don’t expect to push wire speed over all four Ethernet ports at once, even with all this newfound CPU power.

A Wall of Drives

Base ix12s ship with 4 drives installed, but we were disappointed to learn that additional drives must be purchased in 4-disk packs from Iomega. Although this decision is understandable, the ix series remains a holdout amid growing legions of bring-your-own-drive competitors. At least the company supports mixing and matching drive sizes, including 1 TB and 2 TB at present. We suspect that the unit uses the same reliable 5900 rpm Seagate Barracuda LP drives as the ix4-200d.

Iomega added a few tricks to the LifeLine software to take advantage of a possible 12 drives installed. First up is the addition of dual-parity RAID-6 for improved data protection. The company (and this reviewer) suggest this over RAID-5 once more than 5 drives are combined in one set. Don’t worry, though, because RAID configuration can be changed online and any unused drive can be used regardless of its location in the array. The ix12 also adds drive spindown, saving power when the RAID set isn’t in use.

Like the ix4, any portion of a RAID set can be carved out into an iSCSI LUN for Ethernet-connected hosts. Iomega claims that LUN provisioning times have improved with the added horsepower and software tweaks, and we hope this is true. A maximum of 256 LUNs can be configured in this way, though even 12 drives are unlikely to drive much performance to that many storage users.

Where to Use It?

Although not listed yet, Iomega promises that the ix12 will have a place on the Exchange ESRP, Windows Server and Hyper-V logo list, and VMware Compatibility Guide this month. It’s already the first Iomega product to be “EMC E-Lab Tested”, meaning it is on the EMC Support Matrix; this fact alone speaks volumes of EMC’s expectations for the unit. My own experience shows that Iomega iSCSI is fine for smaller VMware and Hyper-V deployments.

Clearly, the ix12 is a new kind of Iomega array. If the 200d and 200r were a sign that the company wanted to move out of the house, the ix12 is a demonstration that they have graduated. Starting at US $5,000, the ix12 is all business and its resume ought to impress in interviews. It can’t quite boast the scalability and redundancy of established arrays (including its brothers from EMC), but it ought to be an easy acquisition for companies looking for a little more storage here or there.

One is left with questions, though: How big will EMC let Iomega get? If 12 drives are acceptable, what about 24? Is SAS off limits? What about 10 Gigabit Ethernet and even Fibre Channel over Ethernet (FCoE) eventually? Can we dream of dual controllers? At some point, the Iomega lineup could even threaten the CLARiiON!

Then there is the competitive landscape. Iomega leapfrogged the 8-drive Data Robotics lineup and landed squarely in competition with the likes of the revitalized Overland Storage but at a much lower price. We also have Netgear, HP, Dell, and Promise, and there is an attractive D-Link box packing 15 drives and 10 GbE. Iomega also has to worry about its own big brother, the Dell/EMC AX4, starting around $12k. It’s a competitive market, and Iomega is in for a fight as even more vendors wake up to the possibilities in this market.

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Iomega Graduates and Goes to Work with the ix12-300r
This post was categorized as Apple, Enterprise storage, Everything, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

What iSCSI Did

Let’s review the test for a moment. In March, Microsoft and Intel demonstrated that the combination of Windows Server 2008 R2 and the Xeon 5500 could saturate a 10 Gb Ethernet link, pushing iSCSI throughput to wire speed. That’s 1,174 MB/s, right around the theoretical maximum of a ten-gigabit link, given a tiny bit of overhead. The pair reunited in January to show that this same combination could deliver an astonishing million I/O operations per second, too.

Both of these results are astonishing. Sure, many high-end Fibre Channel SANs and storage systems blast out gigabytes of data and millions of I/O operations every second, but these tests are much more focused. Benchmarks are perilous, but the folks at Microsoft and Intel devised a fairly clever and focused set. Rather than a “mine’s bigger” contest, the pair only needs to prove that iSCSI can play with the pros.

The side effect is a demonstration of the capabilities of Microsoft and Intel components. Microsoft showed off the capabilities of Windows Server 2008 R2, Hyper-V, and their software iSCSI initiator, while Intel can brag about the Xeon 5500 server platform and X520-2 10 Gb Ethernet Server Adapter with their 82599EB controller. Your mileage may vary, but it is possible to construct a true storage monster on an average server budget.

Intel Inside

Let’s start by looking at the configuration of the local end of the tested configuration. I’m a storage guy so I think of it as the initiator, but you might say it’s the server, the client, or the host. Regardless, the system under test (SUT) is what was put under the microscope. The configuration was a common one: A high-end computer packing an Intel Xeon CPU and 82599-based 10 Gb Ethernet adapter. Most data centers have a machine or two just like this one.

Looking closely, we see that the test in question relied on the following key components:

• Intel’s “Shady Cove” S5520SC workstation-class motherboard
• The Intel Xeon W5580 CPU (4 cores, 8 MB cache, 3.20 GHz)
• 24 GB of DDR3 RAM
• Intel “Niantic” 82599EB 10 Gb Ethernet controller
• Microsoft Windows Server 2008 R2 x64

This combination would set you back about $7,500 – $450 for the motherboard, $1,500 for the CPU, 6 2 GB DDR3 SDRAM modules at $80 each, $1,200 for the Intel X520 NIC, and $4,000 for an Enterprise copy of Windows Server 2008 R2. Not cheap, but not an exotic server either.

Initiate and Optimize

The secret to push the tested system to perform like it did is in the optimizations in the server platform, the NIC, and Windows Server itself.

• The Xeon 5500 processor series includes many enhancements:
  • An integrated memory controller allows for faster RAM access
  • QuickPath interconnect (QPI) replaces the old front-side bus and enhances I/O off the core
  • A new I/O subsystem with PCIe integrated into the CPU
  • MSI-X expands the number of interrupts a PCI device can use
  • New instructions for on-board CRC-32C decoding, speeding up iSCSI digest processing
• The 82599 Ethernet controller also includes enhanced capabilities:
  • VMDq maps I/O queues to multiple cores and virtual machines, reducing I/O bottlenecks
  • Offload of TCP segmentation and receive-side coalescing
  • Interestingly, it does not appear that VMDc/SR-IOV was employed in the test
• Microsoft Windows Server 2008 R2 and Hyper-V are ready to use all of these features and more:
  • R2 uses multi-core CPUs more effectively in general
  • Receive-side scaling (RSS) spreads the I/O workload across all four Xeon cores
  • The iSCSI initiator now allows CRC digest offload (using the new Xeon command set)
  • Numerous “NUMA I/O” optimizations in the initiator
  • TCP/IP Nagle can be disabled in the registry
  • Hyper-V VMQ allows the network packets to be copied directly into the guest virtual machine’s memory

Whew! Put all of these optimizations in a blender and Hyper-V virtual machine iSCSI access will be twice as fast as before. No kidding!

Stay On Target

But we knew all of this back in January. We also saw that a Cisco Nexus 5020 switch was used to fan out to 10 software iSCSI targets. But until now there was no mention of what targets were used exactly.

The final footnotes in Intel’s whitepaper reveals that the storage backing the million IOPS test was none other than StarWind Software‘s iSCSI SAN! It is unclear what led Microsoft and Intel to use this particular iSCSI target (the earlier throughput tests ran on NetApp filers), but it does speak to the quality of this product.

It is not clear how many disk drives were used, but I would guess that SSDs or ramdisks might have been employed to pull a million IOPS. Network optimizations are also not mentioned, though jumbo frames would not be a benefit in an IOPS test.

StarWind’s software runs on Microsoft Windows and creates a full-featured iSCSI target, complete with data mirroring, automatic failover and failback, replication, snapshots, and thin provisioning. The company prices their iSCSI SAN at $6,000 for two nodes and competes with the likes of DataCore and Open-E. But the StarWind solution seems at a glance to be more full-featured than these other offerings.

Try It Yourself!

I imagine many folks like me might be tempted to try to reproduce these results. More valuable would be a set of best practice guidelines for the deployment of software iSCSI in Windows Server 2008 R2 and Hyper-V environments. Given the relatively modest hardware involved, there should be nothing stopping us!

These test results also prompted me to get in touch with StarWind to try their iSCSI target software. I was pleasantly surprised to learn that they are currently offering free non-production licenses to VMware vExperts, VCPs, and VCIs as well as Microsoft MVPs, MCPs, and MCT Professionals. Many of my readers fall into one (or more) of those buckets, and I applaud the company for this offer. If only more companies realized the value in giving away test licenses to influencers and thought leaders!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | How Did Microsoft and Intel Get 1 Million iSCSI IOPS?
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

How fast can iSCSI get?

“Maximizing Hyper-V iSCSI Performance with Microsoft and Intel” might sound like another “blah blah” marketing piece, but a little birdy tells me that this webcast will drop a bombshell about iSCSI performance.

Lots of storage and networking folks don’t give iSCSI and Microsoft the credit they deserve. “iSCSI is cheap and easy,” they say, “but real performance requires Fibre Channel.” Those of us who have an open mind about such things know that this is simply not the case. The fastest SAN I ever saw was based on iSCSI, and Microsoft demonstrated wire-speed iSCSI over 10 Gb Ethernet in March. I never saw a Fibre Channel SAN (even an 8 Gb/s one) push over a gigabyte per second over a single link!

Still, ask the average sysadmin and they will tell you that iSCSI isn’t for high performance applications. That’s why folks should tune in to this webcast, as Microsoft and Intel knock down another iSCSI performance myth. Note that even though Hyper-V is called out in the title and description, this discussion is really about Windows Server 2008 R2 and applies equally regardless of whether or not you use Microsoft’s hypervisor.

Watch this space for a summary of the news immediately following the announcement.

• What: Maximizing Hyper-V iSCSI Performance with Microsoft and Intel webcast
• When: Thursday, January 14, 2010 8:00 AM Pacific Time
• Where: MSEvents.Microsoft.com
• Who: Anyone interested in high-performance storage and server I/O

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Microsoft and Intel Pushing iSCSI Performance Limits
This post was categorized as Enterprise storage, Gestalt IT, Virtual Storage. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.